August 2018 - Libguestfs - libguestfs List Archives

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones 29 Nov '19

29 Nov '19

My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 18

[hivex PATCH] Re-allocating unused blocks before assigning new blocks
by Shreyas Khare 03 Jan '19

03 Jan '19

Hello Richard As discussed in the IRC channel, when merging a moderately large reg file (~35MB) to a hiv file (~118 MB); hivex generates a huge hiv file (~580 MB). These changes address that by creating a list of unallocated blocks and reassigning unused blocks. I used https://github.com/msuhanov/regf/blob/master/Windows%20registry%20file%20fo… as a reference for the structure of the hiv file (in addition to the source code itself) Attaching the patch file. Shreyas -- NOTICE OF CONFIDENTIALITY: At Rapid7, the privacy of our customers, partners, and employees is paramount. If you received this email in error, please notify the sender and delete it from your inbox right away. Learn how Rapid7 handles privacy at rapid7.com/privacy-policy <https://www.rapid7.com/privacy-policy/>. To opt-out of Rapid7 marketing emails, please click here <https://information.rapid7.com/manage-subscription.html> or email privacy(a)rapid7.com <mailto:mailto:privacy@rapid7.com>.

2 3

[PATCH] v2v: ovf: add firmware and machine type element
by Tomáš Golembiovský 10 Oct '18

10 Oct '18

Add oVirt specific elemnt to OVF. It represents the combination of machine type (i440fx/q35) and firmware (BIOS/UEFI). Signed-off-by: Tomáš Golembiovský <tgolembi(a)redhat.com> --- v2v/create_ovf.ml | 20 +++++++++++++++++++- v2v/create_ovf.mli | 2 +- v2v/output_rhv.ml | 6 ++---- v2v/output_rhv_upload.ml | 4 ++-- v2v/output_vdsm.ml | 6 ++---- v2v/test-v2v-o-rhv.ovf.expected | 1 + v2v/test-v2v-o-vdsm-options.ovf.expected | 1 + 7 files changed, 28 insertions(+), 12 deletions(-) diff --git a/v2v/create_ovf.ml b/v2v/create_ovf.ml index 2cf610333..1cab11dfd 100644 --- a/v2v/create_ovf.ml +++ b/v2v/create_ovf.ml @@ -462,6 +462,22 @@ let origin_of_source_hypervisor = function *) | _ -> None +let get_ovirt_biostype guestcaps target_firmware = + let uefi_firmware = + match target_firmware with + | TargetBIOS -> None + | TargetUEFI -> Some (find_uefi_firmware guestcaps.gcaps_arch) in + let secure_boot_required = + match uefi_firmware with + | Some { Uefi.flags = flags } + when List.mem Uefi.UEFI_FLAG_SECURE_BOOT_REQUIRED flags -> true + | _ -> false in + match target_firmware, secure_boot_required with + | TargetUEFI, true -> 3 (* q35 + UEFI + secure boot *) + | TargetUEFI, _ -> 2 (* q35 + UEFI *) + (* 1 is q35 + SeaBIOS *) + | _, _ -> 0 (* i440fx + SeaBIOS *) + (* Generate the .meta file associated with each volume. *) let create_meta_files output_alloc sd_uuid image_uuids targets = (* Note: Upper case in the .meta, mixed case in the OVF. *) @@ -506,7 +522,7 @@ let create_meta_files output_alloc sd_uuid image_uuids targets = ) (List.combine targets image_uuids) (* Create the OVF file. *) -let rec create_ovf source targets guestcaps inspect +let rec create_ovf source targets guestcaps inspect target_firmware output_alloc sd_uuid image_uuids vol_uuids vm_uuid ovf_flavour = assert (List.length targets = List.length vol_uuids); @@ -515,6 +531,7 @@ let rec create_ovf source targets guestcaps inspect let vmtype = get_vmtype inspect in let vmtype = match vmtype with `Desktop -> "0" | `Server -> "1" in let ostype = get_ostype inspect in + let biostype = get_ovirt_biostype guestcaps target_firmware in let ovf : doc = doc "ovf:Envelope" [ @@ -562,6 +579,7 @@ let rec create_ovf source targets guestcaps inspect e "VmType" [] [PCData vmtype]; (* See https://bugzilla.redhat.com/show_bug.cgi?id=1260590#c17 *) e "DefaultDisplayType" [] [PCData "1"]; + e "BiosType" [] [PCData (string_of_int biostype)]; ] in (match source.s_cpu_model with diff --git a/v2v/create_ovf.mli b/v2v/create_ovf.mli index 8200b76f9..cb6c12690 100644 --- a/v2v/create_ovf.mli +++ b/v2v/create_ovf.mli @@ -43,7 +43,7 @@ val ovf_flavour_to_string : ovf_flavour -> string create OVF for another target management system then we would need to heavily modify or even duplicate this code. *) -val create_ovf : Types.source -> Types.target list -> Types.guestcaps -> Types.inspect -> Types.output_allocation -> string -> string list -> string list -> string -> ovf_flavour -> DOM.doc +val create_ovf : Types.source -> Types.target list -> Types.guestcaps -> Types.inspect -> Types.target_firmware -> Types.output_allocation -> string -> string list -> string list -> string -> ovf_flavour -> DOM.doc (** Create the OVF file. Actually a {!DOM} document is created, not a file. It can be written diff --git a/v2v/output_rhv.ml b/v2v/output_rhv.ml index 5260ab030..ce57f0309 100644 --- a/v2v/output_rhv.ml +++ b/v2v/output_rhv.ml @@ -115,7 +115,7 @@ object method as_options = sprintf "-o rhv -os %s" os - method supported_firmware = [ TargetBIOS ] + method supported_firmware = [ TargetBIOS; TargetUEFI ] (* RHV doesn't support serial consoles. This causes the conversion * step to remove it. @@ -270,12 +270,10 @@ object (* This is called after conversion to write the OVF metadata. *) method create_metadata source targets _ guestcaps inspect target_firmware = - (* See #supported_firmware above. *) - assert (target_firmware = TargetBIOS); (* Create the metadata. *) let ovf = Create_ovf.create_ovf source targets guestcaps inspect - output_alloc esd_uuid image_uuids vol_uuids vm_uuid + target_firmware output_alloc esd_uuid image_uuids vol_uuids vm_uuid Create_ovf.RHVExportStorageDomain in (* Write it to the metadata file. *) diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml index 63fa2411a..3dcd6d952 100644 --- a/v2v/output_rhv_upload.ml +++ b/v2v/output_rhv_upload.ml @@ -253,7 +253,7 @@ object sprintf " -oc %s -op %s -os %s" output_conn output_password output_storage - method supported_firmware = [ TargetBIOS ] + method supported_firmware = [ TargetBIOS; TargetUEFI ] (* rhev-apt.exe will be installed (if available). *) method install_rhev_apt = true @@ -407,7 +407,7 @@ If the messages above are not sufficient to diagnose the problem then add the (* Create the metadata. *) let ovf = Create_ovf.create_ovf source targets guestcaps inspect - output_alloc + target_firmware output_alloc sd_uuid image_uuids vol_uuids vm_uuid OVirt in let ovf = DOM.doc_to_string ovf in diff --git a/v2v/output_vdsm.ml b/v2v/output_vdsm.ml index 9a1b748bc..f32acedae 100644 --- a/v2v/output_vdsm.ml +++ b/v2v/output_vdsm.ml @@ -123,7 +123,7 @@ object | flav -> sprintf "-oo vdsm-ovf-flavour=%s" (Create_ovf.ovf_flavour_to_string flav)) - method supported_firmware = [ TargetBIOS ] + method supported_firmware = [ TargetBIOS; TargetUEFI ] (* RHV doesn't support serial consoles. This causes the conversion * step to remove it. @@ -240,11 +240,9 @@ object (* This is called after conversion to write the OVF metadata. *) method create_metadata source targets _ guestcaps inspect target_firmware = - (* See #supported_firmware above. *) - assert (target_firmware = TargetBIOS); - (* Create the metadata. *) let ovf = Create_ovf.create_ovf source targets guestcaps inspect + target_firmware output_alloc dd_uuid vdsm_options.image_uuids vdsm_options.vol_uuids diff --git a/v2v/test-v2v-o-rhv.ovf.expected b/v2v/test-v2v-o-rhv.ovf.expected index 7bcc456c5..c6bd05c56 100644 --- a/v2v/test-v2v-o-rhv.ovf.expected +++ b/v2v/test-v2v-o-rhv.ovf.expected @@ -25,6 +25,7 @@ <IsStateless>False</IsStateless> <VmType>0</VmType> <DefaultDisplayType>1</DefaultDisplayType> + <BiosType>0</BiosType> <Section ovf:id='#VM_ID#' ovf:required='false' xsi:type='ovf:OperatingSystemSection_Type'> <Info>Microsoft Windows 7 Phony Edition</Info> <Description>Windows7</Description> diff --git a/v2v/test-v2v-o-vdsm-options.ovf.expected b/v2v/test-v2v-o-vdsm-options.ovf.expected index abaf37e54..f0d418b46 100644 --- a/v2v/test-v2v-o-vdsm-options.ovf.expected +++ b/v2v/test-v2v-o-vdsm-options.ovf.expected @@ -25,6 +25,7 @@ <IsStateless>False</IsStateless> <VmType>0</VmType> <DefaultDisplayType>1</DefaultDisplayType> + <BiosType>0</BiosType> <OperatingSystemSection ovf:id='VM' ovf:required='false' ovirt:id='11'> <Info>Microsoft Windows 7 Phony Edition</Info> <Description>Windows7</Description> -- 2.18.0

3 5

[PATCH 0/7] RFC: switch v2v to ocaml-libvirt
by Pino Toscano 03 Sep '18

03 Sep '18

Hi, this is a mostly done attempt to switch to ocaml-libvirt, embedding the latest version of it from git. This way, it is possible to improve the way v2v connects to libvirt for both input, and output modules, and interacts with libvirt (e.g. no more virsh calls needed in virt-v2v). As side effect, virt-v2v now requires libvirt, as keeping it optional would create too much burden. I could not test all the libvirt input modes (like VDDK, and Xen), but VMware and libvirtxml work fine as before. Thanks, Pino Toscano (7): v2v: require libvirt common: Bundle the ocaml-libvirt library for use by virt-v2v v2v: switch to ocaml-libvirt v2v: -o libvirt: use a Lazy for the connection v2v: -o libvirt: switch away from virsh v2v: test-harness: stop using the external ocaml-libvirt build: stop looking for ocaml-libvirt .gitignore | 2 + Makefile.am | 5 +- common/mllibvirt/Makefile.am | 102 ++ common/mllibvirt/generator.pl | 890 ++++++++++++++ common/mllibvirt/libvirt.README | 9 + common/mllibvirt/libvirt.ml | 1624 +++++++++++++++++++++++++ common/mllibvirt/libvirt.mli | 1537 +++++++++++++++++++++++ common/mllibvirt/libvirt_c_epilogue.c | 420 +++++++ common/mllibvirt/libvirt_c_oneoffs.c | 1550 +++++++++++++++++++++++ common/mllibvirt/libvirt_c_prologue.c | 129 ++ configure.ac | 1 + docs/C_SOURCE_FILES | 1 - m4/guestfs-ocaml.m4 | 4 - po/POTFILES | 1 - v2v/Makefile.am | 26 +- v2v/copy_to_local.ml | 7 +- v2v/dummy.c | 2 + v2v/input_libvirt.ml | 20 +- v2v/input_libvirt_other.ml | 27 +- v2v/input_libvirt_other.mli | 5 +- v2v/input_libvirt_vcenter_https.ml | 13 +- v2v/input_libvirt_vcenter_https.mli | 2 +- v2v/input_libvirt_vddk.ml | 21 +- v2v/input_libvirt_vddk.mli | 4 +- v2v/input_libvirt_xen_ssh.ml | 13 +- v2v/input_libvirt_xen_ssh.mli | 2 +- v2v/input_libvirtxml.ml | 3 +- v2v/libvirt_utils-c.c | 539 -------- v2v/libvirt_utils.ml | 95 +- v2v/libvirt_utils.mli | 51 +- v2v/output_libvirt.ml | 56 +- v2v/parse_libvirt_xml.ml | 14 +- v2v/parse_libvirt_xml.mli | 11 +- v2v/test-harness/Makefile.am | 5 +- v2v/v2v.ml | 12 +- 35 files changed, 6500 insertions(+), 703 deletions(-) create mode 100644 common/mllibvirt/Makefile.am create mode 100755 common/mllibvirt/generator.pl create mode 100644 common/mllibvirt/libvirt.README create mode 100644 common/mllibvirt/libvirt.ml create mode 100644 common/mllibvirt/libvirt.mli create mode 100644 common/mllibvirt/libvirt_c_epilogue.c create mode 100644 common/mllibvirt/libvirt_c_oneoffs.c create mode 100644 common/mllibvirt/libvirt_c_prologue.c create mode 100644 v2v/dummy.c delete mode 100644 v2v/libvirt_utils-c.c -- 2.17.1

2 8

[PATCH v2 0/2] v2v: Add -o openstack target.
by Richard W.M. Jones 03 Sep '18

03 Sep '18

v1 was here: https://www.redhat.com/archives/libguestfs/2018-August/thread.html#00287 v2: - The -oa option now gives an error; apparently Cinder cannot generally control sparse/preallocated behaviour, although certain Cinder backends can. - The -os option maps to Cinder volume type; suggested by Matt Booth. - Add a simple test.

1 3

[PATCH] build: Pass CFLAGS & LDFLAGS to final supermin link (RHBZ#1624175).
by Richard W.M. Jones 31 Aug '18

31 Aug '18

We also use -runtime-variant _pic which selects the OCaml runtime linked with -fPIC. This will cause a performance regression on i686 although that probably doesn't matter now. A bigger issue is that it will stop supermin from building with older versions of OCaml (<= 4.02.2). We might instead try detecting if it's the old version in ./configure but that gets a bit fragile. --- src/supermin-link.sh.in | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/supermin-link.sh.in b/src/supermin-link.sh.in index 29b84a1..b7fded6 100644 --- a/src/supermin-link.sh.in +++ b/src/supermin-link.sh.in @@ -21,4 +21,8 @@ # Hack automake to link 'supermin' binary properly. There is no other # way to add the -cclib parameter to the end of the command line. -exec "$@" -linkpkg -cclib '@EXT2FS_LIBS@ @COM_ERR_LIBS@ @LIBRPM_LIBS@' +exec "$@" \ + -linkpkg \ + -runtime-variant _pic \ + -ccopt '@CFLAGS@' \ + -cclib '@LDFLAGS@ @EXT2FS_LIBS@ @COM_ERR_LIBS@ @LIBRPM_LIBS@' -- 2.18.0

2 2

[PATCH 0/2] v2v: Add -o openstack target.
by Richard W.M. Jones 29 Aug '18

29 Aug '18

This patch implements output to OpenStack Cinder volumes using OpenStack APIs. It has only been lightly tested, but appears to work. There are some important things to understand about how this works: (1) You must run virt-v2v in a conversion appliance running on top of OpenStack. And you must supply the name or UUID of this appliance to virt-v2v using the ‘-oo server-id=NAME|UUID’ parameter. (See the manual for more details) (2) Virt-v2v will use the same environment variables as the openstack command line tool (eg. OS_USERNAME, OS_PASSWORD etc). You can also supply the same authentication data on the command line using ‘-oo os-username=NAME’ etc. (3) There is no definitive list of authentication parameters consumed by OpenStack, *and any wrapper software around virt-v2v should not assume one*. It should be able to handle any arbitrary (key, value) pairs and simply pass them through. (4) You can run multiple parallel conversions in a single conversion appliance, but OpenStack has some nasty, low limits which will affect this. See the manual for details. (5) Virt-v2v will try to clean up Cinder volumes on failure, but as with RHV this relies on virt-v2v exiting cleanly (not crashing). In case virt-v2v crashes, Cinder volumes which were not successfully converted have the description field "virt-v2v temporary volume for ...", so hopefully this can be used for some kind of garbage collection. (6) Only raw output format is supported. Other output formats wouldn't seem to make any sense since format is internal to Cinder (I think?) This patch seems largely complete. There is hopefully not any major extra work to do. I haven't tested it with guests with more than one disk, and there are some questions about whether OpenStack can really handle this. I haven't tested full end to end (from VMware) conversions. Rich.

1 2

[PATCH RHEL 7.6 LP] RHEL 7.6 LP: Convert Python 3 to Python 2.
by Richard W.M. Jones 28 Aug '18

28 Aug '18

Nir, can you confirm this is correct for RHEL 7 / Python 2? Rich.

2 2

[PATCH] ppc64le: Remove bogus __powerpc64le__.
by Richard W.M. Jones 28 Aug '18

28 Aug '18

Not a thing - tested on P9. --- common/qemuopts/qemuopts.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/common/qemuopts/qemuopts.c b/common/qemuopts/qemuopts.c index 49550bb21..3a8dcae70 100644 --- a/common/qemuopts/qemuopts.c +++ b/common/qemuopts/qemuopts.c @@ -552,7 +552,7 @@ qemuopts_set_binary_by_arch (struct qemuopts *qopts, const char *arch) binary = strdup ("qemu-system-aarch64"); #elif defined(__arm__) binary = strdup ("qemu-system-arm"); -#elif defined(__powerpc64__) || defined(__powerpc64le__) +#elif defined(__powerpc64__) binary = strdup ("qemu-system-ppc64"); #elif defined(__s390x__) binary = strdup ("qemu-system-s390x"); -- 2.18.0

1 1