March 2020 - Libguestfs - libguestfs List Archives

[PATCH libguestfs 0/2] build: Allow programs using libguestfs to be compiled from against build dir.
by Richard W.M. Jones 11 Mar '20

11 Mar '20

These changes allow virt-v2v to be compiled against the build directory of a non-installed libguestfs. Note that some small changes are also required to common and virt-v2v itself. I will post those separately. Rich.

3 7

[PATCH common] common: Use LIBGUESTFS_CFLAGS when compiling.
by Richard W.M. Jones 11 Mar '20

11 Mar '20

Allows virt-v2v to be compiled against the libguestfs build directory. We don't actually need to link these libraries, so LIBGUESTFS_LIBS is not needed here. --- mltools/Makefile.am | 4 +++- mlutils/Makefile.am | 3 ++- options/Makefile.am | 3 ++- utils/Makefile.am | 3 ++- 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mltools/Makefile.am b/mltools/Makefile.am index ce4d1ac..3b4172d 100644 --- a/mltools/Makefile.am +++ b/mltools/Makefile.am @@ -103,7 +103,9 @@ libmltools_a_CPPFLAGS = \ -I$(top_srcdir)/common/mlutils libmltools_a_CFLAGS = \ $(WARN_CFLAGS) $(WERROR_CFLAGS) \ - $(LIBVIRT_CFLAGS) $(LIBXML2_CFLAGS) \ + $(LIBVIRT_CFLAGS) \ + $(LIBXML2_CFLAGS) \ + $(LIBGUESTFS_CFLAGS) \ $(JANSSON_CFLAGS) \ -fPIC diff --git a/mlutils/Makefile.am b/mlutils/Makefile.am index e86d7bb..38a683f 100644 --- a/mlutils/Makefile.am +++ b/mlutils/Makefile.am @@ -63,7 +63,8 @@ libmlcutils_a_CPPFLAGS = \ -I$(shell $(OCAMLC) -where) libmlcutils_a_CFLAGS = \ $(WARN_CFLAGS) $(WERROR_CFLAGS) \ - -fPIC + -fPIC \ + $(LIBGUESTFS_CFLAGS) BOBJECTS = $(SOURCES_ML:.ml=.cmo) XOBJECTS = $(BOBJECTS:.cmo=.cmx) diff --git a/options/Makefile.am b/options/Makefile.am index 394f668..b38fedc 100644 --- a/options/Makefile.am +++ b/options/Makefile.am @@ -45,7 +45,8 @@ liboptions_la_CPPFLAGS = \ liboptions_la_CFLAGS = \ $(WARN_CFLAGS) $(WERROR_CFLAGS) \ $(LIBCONFIG_CFLAGS) \ - $(LIBXML2_CFLAGS) + $(LIBXML2_CFLAGS) \ + $(LIBGUESTFS_CFLAGS) liboptions_la_LIBADD = \ $(top_builddir)/common/utils/libutils.la \ $(LIBCONFIG_LIBS) \ diff --git a/utils/Makefile.am b/utils/Makefile.am index 4878e3b..b47285d 100644 --- a/utils/Makefile.am +++ b/utils/Makefile.am @@ -37,4 +37,5 @@ libutils_la_CPPFLAGS = \ libutils_la_CFLAGS = \ $(WARN_CFLAGS) $(WERROR_CFLAGS) \ $(GCC_VISIBILITY_HIDDEN) \ - $(LIBXML2_CFLAGS) + $(LIBXML2_CFLAGS) \ + $(LIBGUESTFS_CFLAGS) -- 2.24.1

2 1

[v2v PATCH] bash: remove extra registrations
by Pino Toscano 10 Mar '20

10 Mar '20

The virt-v2v-copy-to-local completion script definitely does not need to register completions for libguestfs-test-tool, and virt-win-reg, as they are libguestfs tools. Also, rename the helper _guestfs_options_only to _v2v_options_only, to avoid potential clashes with the libguestfs completion scripts. --- bash/virt-v2v-copy-to-local | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/bash/virt-v2v-copy-to-local b/bash/virt-v2v-copy-to-local index 8360b0b9..c14bbc47 100644 --- a/bash/virt-v2v-copy-to-local +++ b/bash/virt-v2v-copy-to-local @@ -15,7 +15,7 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -_guestfs_options_only () +_v2v_options_only () { local cur prev words cword split local shortopts longopts tool="$1" @@ -42,18 +42,6 @@ _guestfs_options_only () _virt_v2v_copy_to_local () { - _guestfs_options_only "virt-v2v-copy-to-local" + _v2v_options_only "virt-v2v-copy-to-local" } && complete -o default -F _virt_v2v_copy_to_local virt-v2v-copy-to-local - -_virt_win_reg () -{ - _guestfs_options_only "virt-win-reg" -} && -complete -o default -F _virt_win_reg virt-win-reg - -_libguestfs-test-tool () -{ - _guestfs_options_only "libguestfs-test-tool" -} && -complete -o default -F _libguestfs-test-tool libguestfs-test-tool -- 2.24.1

2 1

ANNOUNCE: libguestfs 1.42 - tools for accessing and modifying virtual machine disk images
by Richard W.M. Jones 10 Mar '20

10 Mar '20

I'm pleased to announce the release of libguestfs 1.42, a library and set of tools for accessing and modifying virtual machine disk images. You can find out more about the project here: http://libguestfs.org/ Release notes (http://libguestfs.org/guestfs-release-notes-1.42.1.html) Please note that virt-v2v, virt-p2v and the boot analysis tools were moved out of the libguestfs package / repository into separate projects. See: · https://github.com/libguestfs/virt-v2v · https://github.com/libguestfs/virt-p2v · https://github.com/libguestfs/libguestfs-analysis-tools Those projects are now on a separate release cycle. Release notes are published separately. New features Add support for the Vala language (Corentin Noël). Add support for the Rust language (Hiroyuki Katsura, sponsored by Google Summer of Code). Reproducible builds (Chris Lamb). Advanced Format disks (which use 4K sectors) can now be processed by libguestfs. Use the new optional "blocksize" parameter to "guestfs_add_drive_opts", and the --blocksize option added to several tools (Nikolay Ivanets). Windows CompactOS (NTFS file compression) is now supported. New features in existing tools Advanced machine readable output in various virt tools now supports JSON output, and writing logs to a numbered file descriptor (Pino Toscano). virt-filesystems tool now more accurately reports filesystem total size, in the case where for example the filesystem does not occupy the whole of its containing device (Pino Toscano). Language bindings Python ≥ 2.7 is required. Python bindings are now more extensively tested (Hiroyuki Katsura). Python bindings should now work with Python ≥ 3.8. Multiple fixes to avoid crashes in the Python bindings (Sam Eiderman). OCaml bindings are compiled with "-DCAML_NAME_SPACE" to ensure no non-caml_-namespaced symbols are used (Pino Toscano). OCaml bindings should now work with OCaml 4.09 and 4.10. Erlang has deprecated the old "erl_interface" library which we use for Erlang bindings. It will be removed in OTP 22, which will permanently break Erlang support unless someone steps forward to update the Erlang bindings in libguestfs. Fix compatibility with newer cgo for Go bindings (Tomáš Golembiovský). Go API error handling is now more idiomatic (Csaba Henk). Update documentation about how to handle boolean parameters in Ruby bindings (Csaba Henk). Inspection Libguestfs can now extract icons for Gentoo, newer SUSE and OpenMandriva guests. Improved extraction of RHEL icons. (Pino Toscano). Inspection returns the osinfo IDs of newer Windows versions, and rolling Linux distros (Pino Toscano). OpenMandriva is now detected (Bernhard Rosenkränzer). Arch Linux detection improvements (Pino Toscano). Architectures and platforms Libguestfs can now be compiled on OpenMandriva (Bernhard Rosenkränzer). Ubuntu fix for enabling network support (Ioanna Alifieraki). Other Use of deprecated libguestfs APIs will now cause compiler deprecation warnings by default, whereas previously you have to enable these warnings. To turn off these warnings use "-DGUESTFS_NO_WARN_DEPRECATED=1". A new feature is that you can now disable deprecated APIs completely if you want, using "-DGUESTFS_NO_DEPRECATED=1" (Pino Toscano). Fix hostname in virt-builder templates (Pino Toscano). The single large release notes file is now split into multiple documents organized by version. The "--key" option of several tools was broken, especially if you tried to use the option to pass multiple keys. This has been fixed and a regression test added. Augeas exceptions occurring in daemon (eg. during inspection) will now produced detailed errors (Pino Toscano). Security No security issues were found in this release. API New APIs "luks_uuid" returns the UUID of a LUKS device (Pino Toscano). Other API changes Cope with new output from the "btrfs scrub" command. There is no visible change at the API level. New optional parameter "blocksize" added to "guestfs_add_drive_opts" to support Advanced Format disks (Nikolay Ivanets). Build changes Minimum version of Augeas required is 1.2.0 (Pino Toscano). "common/" subdirectory is now a git submodule shared with the virt-v2v project. Supermin ≥ 5.1.18 is required. There were several bugs in libvirt 5.10.0 and 6.0 which affected libguestfs. It is highly advisable to use libvirt 6.1 where these bugs are fixed (versions of libvirt < 5.10.0 are also fine). Many gnulib modules which we were not using or which we used in only a few places have been removed (Pino Toscano). Multiple fixes for builddir != sourcedir builds (Pino Toscano). A new "bundled/" subdirectory contains one bundled library that we currently use, "ocaml-augeas" (Pino Toscano). Fix compatibility issues when being compiled with GCC 10. Internals When linking OCaml programs, automake silent rules and the "V=1" flag are both now obeyed (Pino Toscano). The "guestfs_device_index" API was reimplemented in library code instead of in the daemon. There should be no observable effect of this change. Since Linux 5.6 enumerates "/dev/sdX" devices in parallel, libguestfs was changed to map API device names to appliance device names, and then do the reverse mapping when returning names from the appliance. Hopefully this should have no observable effect, but can produce somewhat confusing debugging messages since device names in the debug might not match device names passed in to APIs. Furthermore the "root=UUID=..." option is always passed to the appliance (instead of naming the device containing the root filesystem). We expect to move from the Zanata service to the Weblate service for translated strings. For more information see https://bugzilla.redhat.com/1787301 Bugs fixed https://bugzilla.redhat.com/1773520 Segfault in python bindings for guestfs_int_py_event_callback_wrapper https://bugzilla.redhat.com/1746699 Can't import guest from export domain to data domain on rhv4.3 due to error "Invalid parameter: 'DiskType=1'" https://bugzilla.redhat.com/1733168 virt-v2v: Use scp -T in -i vmx -it ssh mode https://bugzilla.redhat.com/1723305 Delete info"export PATH=/path/to/nbdkit-1.1.x:$PATH" in virt-v2v-input-vmware manual page https://bugzilla.redhat.com/1705482 python 3.8 requires using -embed variant of pkgconf file, causing libguestfs to fail to build Python 3.8 bindings https://bugzilla.redhat.com/1703463 libguestfs cannot inspect disks using WIndows 10 NTFS compression, error "unsupported reparse point" https://bugzilla.redhat.com/1694268 [PATCH] Add OpenMandriva support https://bugzilla.redhat.com/1691659 virt-v2v should show a message when qemu-guest-agent is installed in guest successfully during conversion https://bugzilla.redhat.com/1690574 virt-v2v fails to import a guest while cannot find `file_architecture` for a file https://bugzilla.redhat.com/1680361 [v2v][RHV][Scale] v2v Migration to RHV failed on timed out waiting for transfer to finalize https://bugzilla.redhat.com/1661871 python bindings wrongly using str for APIs with RBufferOut/FBuffer on Python 3 https://bugzilla.redhat.com/1626503 Unable to maintain static IP address configuration post VM migration https://bugzilla.redhat.com/1612653 Guest has no disk after rhv-upload converting if target data domain has similar name with other data domain on rhv4.2 https://bugzilla.redhat.com/1605242 Update nbdkit info for vddk in v2v man page https://bugzilla.redhat.com/1584678 On W2K12r2 rhev-apt does not run non-interactively, causing race when starting rhev-apt service from the command line https://bugzilla.redhat.com/1518539 Macvtap network will be lost during v2v conversion https://bugzilla.redhat.com/1351000 virt-customize --commands-from-file will fail if there is an extra blank in the file AUTHORS Ioanna Alifieraki Sam Eiderman Tomáš Golembiovský Csaba Henk Nikolay Ivanets Richard W.M. Jones Hiroyuki Katsura Chris Lamb Corentin Noël Bernhard Rosenkränzer Pino Toscano -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

2 2

[PATCH virt-v2v] v2v: Require libguestfs >= 1.42 to build.
by Richard W.M. Jones 10 Mar '20

10 Mar '20

Should we do this yet? It would of course only be for upstream. It also requires an associated patch to common which I will post separately. Rich.

2 3

[PATCH v4] windows: delay installation of qemu-ga MSI
by Tomáš Golembiovský 10 Mar '20

10 Mar '20

Instead of running firstboot script during early boot schedule a task delayed for 2 minutes. During the first boot, after virt-v2v conversion, Windows installs the drivers injected by virt-v2v. When this installation is finished Windows enforces some kind of internal reboot. This unfortunately terminates any running firstboot scripts thus killing the installation of qemu-ga MSI. This is just a best-effort mitigation. It can still happen (e.g. with slow disk drives) that the drivers are not yet installed when the delayed installation starts. On the other hand we cannot delay it too much otherwise we risk that the users logs in and will be doing some work when the MSI installation starts. After MSI installation finishes the VM needs to be rebooted which would be annoying if that would happen under users hands. Although this is not a best fix (that may come later as it is more complex, e.g. introducing waiting mechanism), the delay as it is defined works in most cases. And it dramaticaly improves the situations -- originaly I experienced more than 90% failure rate. Signed-off-by: Tomáš Golembiovský <tgolembi(a)redhat.com> --- common | 2 +- v2v/convert_windows.ml | 17 +++++++---------- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/common b/common index ea10827b..5371257c 160000 --- a/common +++ b/common @@ -1 +1 @@ -Subproject commit ea10827b4cfb3cfe5f782421c01d2902e5f73f90 +Subproject commit 5371257c3cf27fb09d5f2e31ba378b0e6ccf5df6 diff --git a/v2v/convert_windows.ml b/v2v/convert_windows.ml index 0fda1d4e..9b90f611 100644 --- a/v2v/convert_windows.ml +++ b/v2v/convert_windows.ml @@ -428,16 +428,13 @@ popd and configure_qemu_ga files = List.iter ( fun msi_path -> - let fb_script = "\ -echo Installing qemu-ga from " ^ msi_path ^ " -\"\\" ^ msi_path ^ "\" /norestart /qn /l+*vx \"%~dpn0.log\" -set elvl=!errorlevel! -echo Done installing qemu-ga error_level=!elvl! -if !elvl! == 0 ( - echo Restarting Windows... - shutdown /r /f /c \"rebooted by firstboot script\" -) -" in + let fb_script = sprintf "\ +echo Removing any previously scheduled qemu-ga installation +schtasks.exe /Delete /TN Firstboot-qemu-ga /F +echo Scheduling delayed installation of qemu-ga from %s +powershell.exe -command \"$d = (get-date).AddSeconds(120); schtasks.exe /Create /SC ONCE /ST $d.ToString('HH:mm') /SD $d.ToString('MM/dd/yyyy') /RU SYSTEM /TN Firstboot-qemu-ga /TR \\\"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\\\"\" + " + msi_path msi_path msi_path in Firstboot.add_firstboot_script g inspect.i_root ("install " ^ msi_path) fb_script; ) files -- 2.25.0

2 3

ANNOUNCE: supermin 5.2.0 - a tool for creating and distributing tiny appliances
by Richard W.M. Jones 10 Mar '20

10 Mar '20

Supermin is a tool that we use to create and distribute the tiny libguestfs appliance. The tool has been in development since 2009 and was renamed from ‘febootstrap’ to ‘supermin’ in 2013. To find out more about this tool, see: http://libguestfs.org/supermin.1.html I have just released supermin 5.2.0. This is a stable branch version. Previously we didn't really have a split between stable and development releases, but I intend to maintain a 5.2 stable branch and a 5.3 development branch with stablising patches being periodically backported to the 5.2 branch. The just-released libguestfs 1.42 requires supermin >= 5.1.18. Ideally if you are packaging libguestfs 1.42, please use supermin 5.2.0 with it since it contains many patches which fix problems that are seen only with newer Linux kernels. (However this is not required.) Also supermin 5.2.0 will work with earlier versions of libguestfs, so upgrading should generally be safe. I have added supermin 5.2.0 to Fedora 31, 32 and Rawhide. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

1 0

[PATCH v3 0/3] Switch augeas APIs to OCaml
by Pino Toscano 10 Mar '20

10 Mar '20

This reimplements the augeas APIs using ocaml-augeas (dropping all the C code). The behaviour seems unchanged, although I may have not tested all the various corner cases. Changes from v2: - dropped patch #1, as it was applied already (was a real bugfix) - rebased on master Pino Toscano (3): Revert "Revert "daemon: implement OptString for OCaml APIs"" daemon: move augeas APIs to OCaml daemon: drop usage of C augeas library .gitignore | 1 + daemon/Makefile.am | 7 +- daemon/aug.ml | 143 +++++++++++ daemon/aug_utils.ml | 42 +++ daemon/aug_utils.mli | 32 +++ daemon/augeas.c | 493 ------------------------------------ daemon/cleanups.c | 11 - daemon/daemon-c.c | 18 ++ daemon/daemon-c.h | 1 + daemon/daemon.h | 35 --- daemon/lvm-filter.c | 2 - daemon/mount.c | 3 +- daemon/mount_utils.ml | 2 +- docs/C_SOURCE_FILES | 1 - generator/actions_augeas.ml | 15 ++ generator/actions_core.ml | 2 + generator/daemon.ml | 18 +- po/POTFILES | 1 - 18 files changed, 279 insertions(+), 548 deletions(-) create mode 100644 daemon/aug.ml create mode 100644 daemon/aug_utils.ml create mode 100644 daemon/aug_utils.mli delete mode 100644 daemon/augeas.c -- 2.24.1

2 4

[PATCH] lib: Autodetect backing format and specify it explicitly.
by Richard W.M. Jones 09 Mar '20

09 Mar '20

In the guestfs_disk_create API we have traditionally allowed you to set backingfile without setting backingformat. The meaning of this is to let qemu autodetect the backing format when opening the overlay disk. However libvirt >= 6.0 refuses to even pass such disks to qemu (see https://bugzilla.redhat.com/show_bug.cgi?id=1798148) For this reason, move the autodetection earlier and make it explicit. We now autodetect the format of the backing disk at the time of creation of the overlay, and set that as the backing format in the overlay disk itself, allowing libvirt to open the disk later. --- lib/create.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/lib/create.c b/lib/create.c index e2a59b88d..e30286c39 100644 --- a/lib/create.c +++ b/lib/create.c @@ -255,6 +255,7 @@ disk_create_qcow2 (guestfs_h *g, const char *filename, int64_t size, const struct guestfs_disk_create_argv *optargs) { const char *backingformat = NULL; + CLEANUP_FREE char *backingformat_free = NULL; const char *preallocation = NULL; const char *compat = NULL; int clustersize = -1; @@ -302,6 +303,18 @@ disk_create_qcow2 (guestfs_h *g, const char *filename, int64_t size, } } + /* With libvirt >= 6.0 the backing format must be specified. */ + if (backingfile != NULL && backingformat == NULL) { + backingformat = backingformat_free = guestfs_disk_format (g, backingfile); + if (!backingformat) + return -1; + if (STREQ (backingformat, "unknown")) { + error (g, _("could not auto-detect the format of the backing file %s"), + backingfile); + return -1; + } + } + /* Assemble the qemu-img command line. */ guestfs_int_cmd_add_arg (cmd, "qemu-img"); guestfs_int_cmd_add_arg (cmd, "create"); -- 2.25.0

4 9

[PATCH nbdkit] server: When using --run, wait for captive nbdkit to exit.
by Richard W.M. Jones 07 Mar '20

07 Mar '20

I'd like to propose we backport this to 1.18 and some earlier stable branches too. Rich.

2 3