April 2019 - Libguestfs - libguestfs List Archives

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones 29 Nov '19

29 Nov '19

My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 18

[PATCH] v2v: rhv-upload-plugin - improve wait logic after finalize (RHBZ#1680361)
by Daniel Erez 09 Aug '19

09 Aug '19

After invoking transfer_service.finalize, check operation status by examining ImageTransferPhase and DiskStatus. This is done instead of failing after a predefined timeout regardless the status. * not verified * Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1680361 --- v2v/rhv-upload-plugin.py | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 2a950c5ed..873c11ce1 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -523,14 +523,30 @@ def close(h): # waiting for the transfer object to cease to exist, which # falls through to the exception case and then we can # continue. - endt = time.time() + timeout + start = time.time() try: while True: time.sleep(1) - tmp = transfer_service.get() - if time.time() > endt: - raise RuntimeError("timed out waiting for transfer " - "to finalize") + transfer = transfer_service.get() + + if transfer is None: + disk_service = h['disk_service'] + disk = disk_service.get() + if disk.status == types.DiskStatus.OK: + continue + + if transfer.phase == types.ImageTransferPhase.FINISHED_SUCCESS: + debug("finalized after %s seconds", time.time() - start) + break + + if transfer.phase == types.ImageTransferPhase.FINALIZING_SUCCESS: + if time.time() > start + timeout: + raise RuntimeError("timed out waiting for transfer " + "to finalize") + continue + + raise RuntimeError("Unexpected transfer phase while finalizing " + "upload %r" % transfer.phase) except sdk.NotFoundError: pass --

3 9

[PATCH] appliance: Remove /etc/dhcp/dhclient-enter-hooks.d/resolved.
by Richard W.M. Jones 29 May '19

29 May '19

Workaround for Ubuntu which uses this script to try to start a systemd service. That won't work because systemd is not used inside the appliance. See: https://bugs.launchpad.net/ubuntu/+source/supermin/+bug/1824236 Thanks: Ioanna Alifieraki --- appliance/init | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/appliance/init b/appliance/init index 4f2b55822..b1c4d09ea 100755 --- a/appliance/init +++ b/appliance/init @@ -122,7 +122,9 @@ ip link set dev lo up if test "$guestfs_network" = 1; then iface=$(ls -I all -I default -I lo /proc/sys/net/ipv4/conf) - touch /etc/fstab # Workaround for Ubuntu. + # Two workarounds for Ubuntu: + touch /etc/fstab + rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved if dhclient --version >/dev/null 2>&1; then dhclient $iface else -- 2.20.1

3 6

[PATCH nbdkit 0/7] server: Implement NBD_FLAG_CAN_MULTI_CONN.
by Richard W.M. Jones 15 May '19

15 May '19

First thing to say is that I need to do a *lot* more testing on this, so this is just an early peek. In particular, although it passed ‘make check && make check-valgrind’ I have *not* tested it against a multi-conn-aware client such as the Linux kernel >= 4.9. This implements NBD_FLAG_CAN_MULTI_CONN, described in the protocol doc as: "NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to the client. In the absence of this flag, clients SHOULD NOT multiplex their commands over more than one connection to the export." This is necessary to support the Linux nbd client -C option. The only plugin which sets the flag so far is file. The ocaml, sh and nbd plugins allow the flag to be controlled or passed through. Notable also is that the blocksize filter has to filter out this flag, because I'm not convinced that the bounce buffer is safe. However I believe the other filters *are* safe (although not really certain about the fua filter, and the new cache filter is tricky too). My feeling is that we should set the flag unconditionally for all readonly connections, but I can think of nasty corner cases where it might not work. We should most probably set it in all plugins that are readonly (eg. nbdkit-pattern-plugin). Rich.

2 15

[nbdkit PATCH 0/7] Implement structured replies in nbd plugin
by Eric Blake 14 May '19

14 May '19

I'm hoping to implement .extents for the nbd plugin; this is a prerequisite. I'm not sure about patch 3 - if we like it, I'll squash it to 2, if we don't, I think we are okay just dropping it. I'm also wondering if we have to worry about malicious plugins that don't populate the entire .pread buffer in an effort to get nbdkit to expose portions of the heap; my patch 7 loses some time on extra memset()s to avoid that, although it's probably in the noise compared to the time spent talking over sockets to the remote server. Eric Blake (7): protocol: Send correct structured reply error for non-Linux protocol: Support EOVERFLOW RFC: protocol: Only send EOVERFLOW when valid nbd: Honor server global flags on little-endian protocol: Add helpers for error response handling nbd: Implement NBD_OPT_GO client request nbd: Implement structured replies common/protocol/protocol.h | 21 +- plugins/nbd/nbd.c | 434 +++++++++++++++++++++++++++++++++---- server/protocol.c | 16 +- 3 files changed, 422 insertions(+), 49 deletions(-) -- 2.20.1

3 17

[PATCH 43 0/7] v2v: switch to ocaml-libvirt
by Pino Toscano 13 May '19

13 May '19

Hi, this series switches virt-2v to ocaml-libvirt, embedding the latest version of it from git. This way, it is possible to improve the way v2v connects to libvirt for both input, and output modules, and interacts with libvirt (e.g. no more virsh calls needed in virt-v2v). As side effect, virt-v2v now requires libvirt, as keeping it optional would create too much burden. I could not test all the libvirt input modes (like VDDK, and Xen), but VMware and libvirtxml work fine as before. Changes from v3: - rebase on master - update libvirt-ocaml from its master branch - use $(srcdir) in common/mllibvirt/Makefile.am - parse_libvirt_xml: open a libvirt connection only when needed (and only once) Changes from v2: - rebase on master Changes from v1: - rebase on master - update ocaml-libvirt from libvirt-ocaml.git on libvirt.org, and adjust the code to it - pass again the URI to input_libvirt_vddk, so an error message is preserved Pino Toscano (7): v2v: require libvirt common: Bundle the libvirt-ocaml library for use by virt-v2v v2v: switch to ocaml-libvirt v2v: -o libvirt: use a Lazy for the connection v2v: -o libvirt: switch away from virsh v2v: test-harness: stop using the external ocaml-libvirt build: stop looking for ocaml-libvirt .gitignore | 2 + Makefile.am | 5 +- common/mllibvirt/Makefile.am | 102 ++ common/mllibvirt/generator.pl | 908 +++++++++++++ common/mllibvirt/libvirt.README | 9 + common/mllibvirt/libvirt.ml | 1673 ++++++++++++++++++++++++ common/mllibvirt/libvirt.mli | 1647 ++++++++++++++++++++++++ common/mllibvirt/libvirt_c_epilogue.c | 462 +++++++ common/mllibvirt/libvirt_c_oneoffs.c | 1714 +++++++++++++++++++++++++ common/mllibvirt/libvirt_c_prologue.c | 134 ++ configure.ac | 1 + docs/C_SOURCE_FILES | 1 - m4/guestfs-ocaml.m4 | 4 - po/POTFILES | 1 - v2v/Makefile.am | 26 +- v2v/copy_to_local.ml | 7 +- v2v/input_libvirt.ml | 20 +- v2v/input_libvirt_other.ml | 27 +- v2v/input_libvirt_other.mli | 5 +- v2v/input_libvirt_vcenter_https.ml | 13 +- v2v/input_libvirt_vcenter_https.mli | 2 +- v2v/input_libvirt_vddk.ml | 15 +- v2v/input_libvirt_vddk.mli | 4 +- v2v/input_libvirt_xen_ssh.ml | 13 +- v2v/input_libvirt_xen_ssh.mli | 2 +- v2v/libvirt_utils-c.c | 539 -------- v2v/libvirt_utils.ml | 95 +- v2v/libvirt_utils.mli | 51 +- v2v/output_libvirt.ml | 56 +- v2v/parse_libvirt_xml.ml | 23 +- v2v/parse_libvirt_xml.mli | 11 +- v2v/test-harness/Makefile.am | 5 +- v2v/v2v.ml | 12 +- 33 files changed, 6893 insertions(+), 696 deletions(-) create mode 100644 common/mllibvirt/Makefile.am create mode 100755 common/mllibvirt/generator.pl create mode 100644 common/mllibvirt/libvirt.README create mode 100644 common/mllibvirt/libvirt.ml create mode 100644 common/mllibvirt/libvirt.mli create mode 100644 common/mllibvirt/libvirt_c_epilogue.c create mode 100644 common/mllibvirt/libvirt_c_oneoffs.c create mode 100644 common/mllibvirt/libvirt_c_prologue.c delete mode 100644 v2v/libvirt_utils-c.c -- 2.20.1

2 14

[PATCH 00/11] v2v: Change virt-v2v to use nbdkit for input in several modes.
by Richard W.M. Jones 10 May '19

10 May '19

This series (except the last one) changes virt-v2v to use nbdkit for several input modes: -i vmx -it vddk: No change in functionality, as this already uses nbdkit-vddk-plugin, but the code is refactored for the other modes to use. -i libvirtxml: Use nbdkit-curl-plugin instead of qemu curl. vCenter: Use nbdkit-curl-plugin instead of qemu curl. xen: Use nbdkit-ssh-plugin instead of qemu ssh. -i vmx -it ssh: Use nbdkit-ssh-plugin instead of qemu ssh. I was hoping to post this patch series when it was longer and showed some benefits. However those patches aren't ready quite yet, but these ones are ready for review. However some of the future benefits will be: * using nbdkit-rate-plugin to bandwidth limit the input side * better support for modern SSH encryption * possibility of SSH password auth (instead of requiring ssh-agent) I have tested this locally. Unfortunately none of the existing tests cover any of these input cases. The last patch reduces the minimum required version of nbdkit from 1.12 (not yet available) to 1.11, so this patch can be tested in Fedora 30. It is obviously not meant to be applied. Rich.

2 12

[nbdkit PATCH 0/3] Fix data integrity in vddk plugin
by Martin Kletzander 30 Apr '19

30 Apr '19

Couple of fixes to return correct data and one nice-to-have clean-up which is not needed. I just find it nicer to read. Martin Kletzander (3): vddk: Use a separate handle for single-link=true vddk: Do not report hole extents to be zero with single-link=true vddk: Eliminate one needless goto plugins/vddk/vddk.c | 48 +++++++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 12 deletions(-) -- 2.21.0

2 5

[nbdkit PATCH 0/2] Let nbd plugin connect to TCP socket
by Eric Blake 30 Apr '19

30 Apr '19

Accepting only Unix sockets can be a bit limiting; let's be more flexible. Eric Blake (2): nbd: Refactor Unix socket connection nbd: Support TCP socket plugins/nbd/nbdkit-nbd-plugin.pod | 36 ++++-- plugins/nbd/nbd.c | 175 ++++++++++++++++++++++-------- TODO | 3 - 3 files changed, 161 insertions(+), 53 deletions(-) -- 2.20.1

2 3