May 2019 - Libguestfs - libguestfs List Archives

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones 29 Nov '19

29 Nov '19

My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 18

[PATCH] v2v: rhv-upload-plugin - improve wait logic after finalize (RHBZ#1680361)
by Daniel Erez 09 Aug '19

09 Aug '19

After invoking transfer_service.finalize, check operation status by examining ImageTransferPhase and DiskStatus. This is done instead of failing after a predefined timeout regardless the status. * not verified * Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1680361 --- v2v/rhv-upload-plugin.py | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 2a950c5ed..873c11ce1 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -523,14 +523,30 @@ def close(h): # waiting for the transfer object to cease to exist, which # falls through to the exception case and then we can # continue. - endt = time.time() + timeout + start = time.time() try: while True: time.sleep(1) - tmp = transfer_service.get() - if time.time() > endt: - raise RuntimeError("timed out waiting for transfer " - "to finalize") + transfer = transfer_service.get() + + if transfer is None: + disk_service = h['disk_service'] + disk = disk_service.get() + if disk.status == types.DiskStatus.OK: + continue + + if transfer.phase == types.ImageTransferPhase.FINISHED_SUCCESS: + debug("finalized after %s seconds", time.time() - start) + break + + if transfer.phase == types.ImageTransferPhase.FINALIZING_SUCCESS: + if time.time() > start + timeout: + raise RuntimeError("timed out waiting for transfer " + "to finalize") + continue + + raise RuntimeError("Unexpected transfer phase while finalizing " + "upload %r" % transfer.phase) except sdk.NotFoundError: pass --

3 9

[libnbd] Simultaneous read and write
by Richard W.M. Jones 03 Jun '19

03 Jun '19

This is a continuation of a discussion we were having on IRC. The problems with IRC are it's not recorded and it's hard to have deep technical conversations. I hope this is a decent summary. Problem simply stated: Certain NBD servers (qemu-nbd in particular) are able to simultaneously read and write on a socket. ie. They can be simultaneously reading a request and writing the reply to a previous request. However libnbd is unable to do this trick. Although multiple requests can be in flight, libnbd is either writing on the socket or reading from the socket, but never both. Visualized it looks like this: write write write |===========|______________|=======|_______|===| --> rq to server _____________|============|_________|=====|_____ <-- rp from server read read whereas an ideal libnbd which could write and read simultaneously, coupled with a server which can do the same: write |===========||=======||===| --> requests to server |============||=====||===== <-- replies from server read Eric already observed through testing that the ideal client can be up to twice as fast as libnbd, as is obvious from the diagram. The reasons why libnbd can't do this are: Problem (a) : Only one thread of control can hold the libnbd handle lock (h->lock), and since there can only be one thread of control running in each handle at any time, that thread can only be reading or writing. Problem (b) : There is only one state machine per handle (h->state), whereas to handle the write and read sides separately requires two state machines. In the IRC discussion we gave these the preliminary names h->wstate and h->rstate. ---------------------------------------------------------------------- It's worth also saying how the current API works, although we might want to change it. You grab the underlying file descriptor using nbd_aio_get_fd, which is what you poll on. You also have to call nbd_aio_get_direction which returns READ, WRITE or BOTH (== READ|WRITE). You then set up some events mechanism (eg. poll, epoll, etc.), poll until the file descriptor is ready, and call one of nbd_aio_notify_read or nbd_aio_notify_write. The direction can change any time the handle state changes, which includes whenever you issue a command (eg. nbd_aio_pread), or whenever you call nbd_aio_notify_*. You therefore have to call nbd_aio_get_direction frequently. A typical loop using poll might look like: fd = nbd_aio_get_fd (nbd); for (;;) { /* <-- If you need to issue more commands, do that here. */ dir = nbd_aio_get_direction (nbd); pollfd[0].fd = fd; pollfd[0].events = 0; if (dir & LIBNBD_AIO_DIRECTION_READ) pollfd[0].events |= POLLIN; if (dir & LIBNBD_AIO_DIRECTION_WRITE) pollfd[0].events |= POLLOUT; poll (pollfd, 1, -1); if (pollfd[0].revents & LIBNBD_AIO_DIRECTION_READ) nbd_aio_notify_read (); else if (pollfd[0].revents & LIBNBD_AIO_DIRECTION_WRITE) nbd_aio_notify_write (); } ---------------------------------------------------------------------- The above code is of course assuming a single thread. But to simultaneously read and write we will need at least two threads. It's hard for me to see a nice way to evolve the API to support multiple threads, but I guess issues include: - If one thread is waiting in poll(2) and another thread issues a command, how do we get the first thread to return from poll and reevaluate direction? (Eric suggests a pipe-to-self for this) - If two threads are waiting in poll(2) and both are notified that the fd is ready, how do we get one of them to read and the other to write? I think this implies that we don't have two threads doing poll(2), but if not how do we farm out the read/write work to two or more threads? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

2 4

[PATCH] python: PYTHON_LIBS is not set in Python 3.8 (RHBZ#1705482).
by Richard W.M. Jones 31 May '19

31 May '19

Python 3.8 no longer links C extensions to -lpython, instead relying on the fact that the python binary itself already contains those symbols. This means $PYTHON_LIBS is empty and so the Python bindings are not built. Use a different test to see if the python module is available. --- m4/guestfs-python.m4 | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/m4/guestfs-python.m4 b/m4/guestfs-python.m4 index 7d4c991bb..befa9b102 100644 --- a/m4/guestfs-python.m4 +++ b/m4/guestfs-python.m4 @@ -33,14 +33,17 @@ AS_IF([test "x$enable_python" != "xno"],[ PYTHON_VERSION_MINOR=`$PYTHON -c "import sys; print (sys.version_info@<:@1@:>@)"` PYTHON_VERSION="$PYTHON_VERSION_MAJOR.$PYTHON_VERSION_MINOR" AC_MSG_RESULT([$PYTHON_VERSION]) + # Debian: python-2.7.pc, python-3.2.pc PKG_CHECK_MODULES([PYTHON], [python-"$PYTHON_VERSION"],[ + have_python_module=1 AC_SUBST([PYTHON_CFLAGS]) AC_SUBST([PYTHON_LIBS]) AC_SUBST([PYTHON_VERSION]) AC_DEFINE([HAVE_PYTHON],[1],[Python library found at compile time]) ],[ PKG_CHECK_MODULES([PYTHON], [python],[ + have_python_module=1 AC_SUBST([PYTHON_CFLAGS]) AC_SUBST([PYTHON_LIBS]) AC_SUBST([PYTHON_VERSION]) @@ -49,6 +52,7 @@ AS_IF([test "x$enable_python" != "xno"],[ AC_MSG_WARN([python $PYTHON_VERSION not found]) ]) ]) + AC_MSG_CHECKING([Python prefix]) PYTHON_PREFIX=`$PYTHON -c "import sys; print (sys.prefix)"` AC_MSG_RESULT([$PYTHON_PREFIX]) @@ -101,4 +105,4 @@ AS_IF([test "x$enable_python" != "xno"],[ AC_SUBST(PYTHON_EXT_SUFFIX) ]) AM_CONDITIONAL([HAVE_PYTHON], - [test "x$PYTHON" != "xno" && test "x$PYTHON_LIBS" != "x" ]) + [test "x$PYTHON" != "xno" && test "x$have_python_module" = "x1" ]) -- 2.21.0

1 0

[nbdkit PATCH 0/4] Play with libnbd for nbdkit-add
by Eric Blake 31 May '19

31 May '19

Patch 1 played with an early draft of Rich's Fedora 30 libnbd package: https://bugzilla.redhat.com/show_bug.cgi?id=1713767#c17 Note that comment 21 provides a newer package 0.1.1-1 with a different API; and that libnbd has more unreleased API changes in the pipeline (whether that will be called 0.2 or 0.1.2); so we'll have to tweak things based on what is actually available in distros. Patch 2 is mechanical, to try and make patch 3 a little easier to follow. Patch 3 shows that libnbd is currently feature-compatible with my hand-rolled nbd client, albeit with an order-of-magnitude slowdown that we'll have to investigate. And patch 4 is my driving reason for this series - it solves a TODO item about allowing the NBD plugin to be the symmetric solution for connecting a new server that requires encryption to an old client that can't speak it. (Although it's late enough at night that I have not actually yet added the tests to 'make check' to exercise it.) Also available at https://repo.or.cz/nbdkit/ericb.git under tag use-libnbd-v1 Eric Blake (4): nbd: Check for libnbd nbd: s/nbd_/nbdplug_/ nbd: Use libnbd 0.1 nbd: Add TLS client support plugins/nbd/nbdkit-nbd-plugin.pod | 77 +- configure.ac | 20 + plugins/nbd/nbd-standalone.c | 1364 +++++++++++++++++++++++++++++ plugins/nbd/nbd.c | 1308 +++++++++------------------ TODO | 13 +- plugins/nbd/Makefile.am | 25 +- 6 files changed, 1860 insertions(+), 947 deletions(-) create mode 100644 plugins/nbd/nbd-standalone.c -- 2.20.1

2 7

[PATCH nbdkit 1/2] nbd: Fix -Werror=maybe-uninitialized warning.
by Richard W.M. Jones 30 May '19

30 May '19

GCC is concerned that if we never go round the loop then fd will be uninitialized. By asserting that getaddrinfo set result != NULL we can avoid this. nbd.c: In function ‘nbd_open_handle’: nbd.c:974:5: error: ‘fd’ may be used uninitialized in this function [-Werror=maybe-uninitialized] 974 | close (fd); | ^~~~~~~~~~ nbd.c:954:7: note: ‘fd’ was declared here 954 | int fd; | ^~ cc1: all warnings being treated as errors --- plugins/nbd/nbd.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c index dba46f1..4d0a67c 100644 --- a/plugins/nbd/nbd.c +++ b/plugins/nbd/nbd.c @@ -960,6 +960,8 @@ nbd_connect_tcp (void) return -1; } + assert (result != NULL); + for (rp = result; rp; rp = rp->ai_next) { fd = socket (rp->ai_family, rp->ai_socktype, rp->ai_protocol); if (fd == -1) -- 2.21.0

2 2

[PATCH libnbd 0/2] Avoid lock and error overhead on some calls.
by Richard W.M. Jones 30 May '19

30 May '19

This works. I'm in the middle of testing whether there is any noticable benefit. Rich.

1 3

[PATCH 0/5] RFC: switch augeas APIs to OCaml
by Pino Toscano 30 May '19

30 May '19

This synchronizes the embedded ocaml-augeas copy, and reimplements the augeas APIs using it (dropping all the C code). The behaviour seems unchanged, although I may have not tested all the various corner cases. Pino Toscano (5): common/mlaugeas: Synchronize with latest ocaml-augeas daemon: fix/enhance error reporting of Augeas exceptions Revert "Revert "daemon: implement OptString for OCaml APIs"" daemon: move augeas APIs to OCaml daemon: drop usage of C augeas library .gitignore | 1 + common/mlaugeas/augeas-c.c | 190 ++++++++++++-- common/mlaugeas/augeas.ml | 18 +- common/mlaugeas/augeas.mli | 32 ++- daemon/Makefile.am | 7 +- daemon/aug.ml | 143 +++++++++++ daemon/aug_utils.ml | 42 +++ daemon/aug_utils.mli | 32 +++ daemon/augeas.c | 493 ------------------------------------ daemon/cleanups.c | 11 - daemon/daemon-c.c | 29 ++- daemon/daemon-c.h | 1 + daemon/daemon.h | 35 --- daemon/lvm-filter.c | 2 - daemon/mount.c | 3 +- daemon/mount_utils.ml | 2 +- docs/C_SOURCE_FILES | 1 - generator/actions_augeas.ml | 15 ++ generator/actions_core.ml | 2 + generator/daemon.ml | 18 +- po/POTFILES | 1 - 21 files changed, 498 insertions(+), 580 deletions(-) create mode 100644 daemon/aug.ml create mode 100644 daemon/aug_utils.ml create mode 100644 daemon/aug_utils.mli delete mode 100644 daemon/augeas.c -- 2.21.0

1 5

[PATCH] appliance: Remove /etc/dhcp/dhclient-enter-hooks.d/resolved.
by Richard W.M. Jones 29 May '19

29 May '19

Workaround for Ubuntu which uses this script to try to start a systemd service. That won't work because systemd is not used inside the appliance. See: https://bugs.launchpad.net/ubuntu/+source/supermin/+bug/1824236 Thanks: Ioanna Alifieraki --- appliance/init | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/appliance/init b/appliance/init index 4f2b55822..b1c4d09ea 100755 --- a/appliance/init +++ b/appliance/init @@ -122,7 +122,9 @@ ip link set dev lo up if test "$guestfs_network" = 1; then iface=$(ls -I all -I default -I lo /proc/sys/net/ipv4/conf) - touch /etc/fstab # Workaround for Ubuntu. + # Two workarounds for Ubuntu: + touch /etc/fstab + rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved if dhclient --version >/dev/null 2>&1; then dhclient $iface else -- 2.20.1

3 6