December 2019 - Libguestfs - libguestfs List Archives

[PATCH 0/2] Delay firstboot scripts to some later time
by Tomáš Golembiovský 20 Feb '20

20 Feb '20

When firstboot is used from virt-v2v the scripts, if not fast enough, can get killed by Windows. After windows installs virtio drivers injected by virt-v2v it performs some internall reboot, stopping all the running services and killing any running firstboot script. This is problem mostly for MSI installs (like qemu-ga that was added recently) that can take several seconds to finish. This change is little bit controversial in fact that it relies on powershell. This is not available in early versions (without service packs) of XP and 2003 and our support matrix still mentions these Windows versions. The change can also be problem for those who really wish to perform an action early in the boot... if there are such users. Tomáš Golembiovský (2): firstboot: use absolute path to rhsrvany firstboot: schedule firstboot as delayed task mlcustomize/firstboot.ml | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) -- 2.24.0

3 9

[PATCH libnbd 0/2] api: Add support for AF_VSOCK.
by Richard W.M. Jones 10 Feb '20

10 Feb '20

This is a series of patches to libnbd and nbdkit adding AF_VSOCK support. On the host side it allows you to start an nbdkit instance which listens on a virtio-vsock socket: $ ./nbdkit -fv --vsock memory 1G ... nbdkit: debug: bound to vsock 2:10809 On the guest side you can then use libnbd to connect to the server: $ ./run nbdsh -c 'h.connect_vsock(2, 10809)' -c 'print(h.get_size())' 1073741824 $ ./run nbdfuse mp --vsock 2 10809 & $ ll mp/ total 0 -rw-rw-rw-. 1 rjones rjones 1073741824 Oct 18 16:23 nbd $ dd if=/dev/random of=mp/nbd bs=1024 count=100 conv=notrunc,nocreat dd: warning: partial read (84 bytes); suggest iflag=fullblock 0+100 records in 0+100 records out 6851 bytes (6.9 kB, 6.7 KiB) copied, 0.013797 s, 497 kB/s (Performance of FUSE is not great, it should be better using raw libnbd.) I mainly wrote this to show that it can be done. It's unclear if this would be faster or slower than the usual way that NBD devices are exposed to guests via virtio-blk/-scsi. https://wiki.qemu.org/Features/VirtioVsock Thanks: Stefan Hajnoczi for help with debugging this.

3 10

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

[PATCH 0/2] Fixes and tweak to the installation of qemu-ga MSI
by Tomáš Golembiovský 05 Feb '20

05 Feb '20

This, together with the changes to common repo are fixes to the installation qemu-ga MSI. There is still an issue that I did not figure yet how to fix. On Windows 10 it fails to register the QEMU-GA service. Tomáš Golembiovský (2): windows: fix detection of qemu-ga installer on RHV windows: small tweaks of qemu-ga firstboot script v2v/convert_windows.ml | 8 +++++++- v2v/windows_virtio.ml | 5 ++--- 2 files changed, 9 insertions(+), 4 deletions(-) -- 2.24.0

3 8

[v2v PATCH] build: add --with-extra configure parameter
by Pino Toscano 09 Jan '20

09 Jan '20

Borrow this feature from libguestfs, so it is possible to "watermark" the virt-v2v executable with a extra string to the version string. --- configure.ac | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 00246f29..e2418564 100644 --- a/configure.ac +++ b/configure.ac @@ -15,7 +15,14 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -AC_INIT([virt-v2v],1.41.8) +# The major, minor, and release fields MUST be numbers. Packagers can +# add extra information using --with-extra="..." which may be any +# freeform string. +m4_define([v2v_major], [1]) +m4_define([v2v_minor], [41]) +m4_define([v2v_release], [8]) + +AC_INIT([virt-v2v],v2v_major.v2v_minor.v2v_release) # Headings within the configure script output. m4_define([HEADING], @@ -37,6 +44,25 @@ AC_CONFIG_MACRO_DIR([m4]) dnl Initialize libtool. LT_INIT +dnl Extra string, a freeform string defined by packagers. +AC_ARG_WITH([extra], + [AS_HELP_STRING([--with-extra], + [extra version string (for use by packagers)])], + [v2v_extra="$withval"], + [v2v_extra=] +) + +AC_MSG_NOTICE([virt-v2v version v2v_major.v2v_minor.v2v_release$v2v_extra]) + +dnl Split up the version string. +AC_DEFINE([PACKAGE_VERSION_MAJOR],[v2v_major],[Major version number.]) +AC_DEFINE([PACKAGE_VERSION_MINOR],[v2v_minor],[Minor version number.]) +AC_DEFINE([PACKAGE_VERSION_RELEASE],[v2v_release],[Release number.]) +AC_DEFINE_UNQUOTED([PACKAGE_VERSION_EXTRA],["$v2v_extra"],[Extra version string.]) +PACKAGE_VERSION_FULL="v2v_major.v2v_minor.v2v_release${v2v_extra}" +AC_DEFINE_UNQUOTED([PACKAGE_VERSION_FULL],["$PACKAGE_VERSION_FULL"],[Full version string.]) +AC_SUBST([PACKAGE_VERSION_FULL]) + dnl Early gnulib initialization. HEADING([Configuring Gnulib]) gl_EARLY @@ -79,8 +105,6 @@ dnl These are required to get common/*/Makefile.am files dnl to work. We should further decouple these in future XXX AM_CONDITIONAL([ENABLE_APPLIANCE],[true]) AM_CONDITIONAL([HAVE_PYTHON],[false]) dnl Disables a test -AC_DEFINE_UNQUOTED([PACKAGE_VERSION_FULL],["$PACKAGE_VERSION"],[Version string.]) -AC_SUBST([PACKAGE_VERSION_FULL], [$PACKAGE_VERSION]) dnl Produce output files. HEADING([Generating output files]) -- 2.23.0

2 2

[v2v PATCH] po: do not extract tests
by Pino Toscano 09 Jan '20

09 Jan '20

They do not contain messages. --- Makefile.am | 1 + po/POTFILES-ml | 11 ----------- 2 files changed, 1 insertion(+), 11 deletions(-) diff --git a/Makefile.am b/Makefile.am index 4dde5002..46a52ca3 100644 --- a/Makefile.am +++ b/Makefile.am @@ -115,6 +115,7 @@ po/POTFILES-ml: configure.ac grep -v '^common/mlprogress/' | \ grep -v '^common/mlvisit/' | \ grep -v '^v2v/config.ml$$' | \ + grep -v -E '.*_tests\.ml$$' | \ LC_ALL=C sort > $@-t mv $@-t $@ diff --git a/po/POTFILES-ml b/po/POTFILES-ml index dae9bd1a..7512d63b 100644 --- a/po/POTFILES-ml +++ b/po/POTFILES-ml @@ -3,32 +3,23 @@ common/mlcustomize/customize_cmdline.ml common/mlcustomize/firstboot.ml common/mlgettext/common_gettext.ml common/mlpcre/PCRE.ml -common/mlpcre/pcre_tests.ml common/mlstdutils/guestfs_config.ml common/mlstdutils/std_utils.ml -common/mlstdutils/std_utils_tests.ml common/mlstdutils/stringMap.ml common/mlstdutils/stringSet.ml common/mltools/JSON.ml common/mltools/JSON_parser.ml -common/mltools/JSON_parser_tests.ml -common/mltools/JSON_tests.ml common/mltools/URI.ml common/mltools/checksums.ml common/mltools/curl.ml common/mltools/getopt.ml -common/mltools/getopt_tests.ml -common/mltools/machine_readable_tests.ml common/mltools/planner.ml common/mltools/regedit.ml common/mltools/registry.ml -common/mltools/tools_messages_tests.ml common/mltools/tools_utils.ml -common/mltools/tools_utils_tests.ml common/mltools/urandom.ml common/mltools/xpath_helpers.ml common/mlutils/c_utils.ml -common/mlutils/c_utils_unit_tests.ml common/mlutils/unix_utils.ml common/mlv2v/uefi.ml common/mlxml/xml.ml @@ -86,9 +77,7 @@ v2v/target_bus_assignment.ml v2v/types.ml v2v/utils.ml v2v/v2v.ml -v2v/v2v_unit_tests.ml v2v/vCenter.ml v2v/var_expander.ml -v2v/var_expander_tests.ml v2v/windows.ml v2v/windows_virtio.ml -- 2.24.1

2 2

Re: [Libguestfs] Libguestfs with Yara rules error
by Luis Fueris 09 Jan '20

09 Jan '20

Sorry Richard. Now I will attach you debug file. El 21/12/2019 a las 16:38, Luis Fueris escribió: > > Hi Richard. > > Few days ago, I installed libyara a libguestfs properly. But when I > load a yara rule and scan it via guestfs_yara_scan, my binary > throughts following error: > > libguestfs: error: deserialise_yara_detection_list: Success > > And function exists with NULL value. As we can see this function is on > lib/yara.c from libguestfs git. I think that these yara functions are > an integration for yara C api. All debug and trace log is in > libguestfs-yara.debug file. > > > El 10/12/2019 a las 09:46, Richard W.M. Jones escribió: >> On Tue, Dec 10, 2019 at 09:19:47AM +0100, Luis wrote: >>> I am using libguestfs 1.40.2 and yara 3.11.0 but when I execute my program >>> it thoughts the following error: >>> >>> $> ./yara-guestfs >>> libguestfs: error: yara_load: feature 'libyara' is not available in this >>> build of libguestfs. Read 'AVAILABILITY' in the guestfs(3) man page for >>> >>> If we check the manual, in guestfs appears guestfs_yara_load function so >>> libyara is installed. I installed and compiled it from tarballs. >> This function: >> >> https://github.com/libguestfs/libguestfs/blob/4aa4edf972bc8df8869cdcaa4ab4f… >> >> returns 1 if HAVE_YARA is defined. This is defined by this >> ./configure test: >> >> https://github.com/libguestfs/libguestfs/blob/4aa4edf972bc8df8869cdcaa4ab4f… >> >> In other words it depends how libguestfs was configured. Where >> did you get libguestfs from? How was it compiled? >> >> Rich. >>

2 1

[common/libguestfs PATCH 0/2] Fix OCaml/Python linking
by Pino Toscano 09 Jan '20

09 Jan '20

Make sure they can build also in no as-neede setups. Pino Toscano (1): utils: split string list functions in own file build: use split stringlist functions from common/utils utils/Makefile.am | 2 + utils/guestfs-stringlists-utils.h | 30 +++++ utils/guestfs-utils.h | 7 +- utils/stringlists-utils.c | 197 ++++++++++++++++++++++++++++++ utils/utils.c | 165 ------------------------- 5 files changed, 230 insertions(+), 171 deletions(-) create mode 100644 utils/guestfs-stringlists-utils.h create mode 100644 utils/stringlists-utils.c -- 2.24.1

2 3

[PATCH] python: remove also __pycache__ directories on clean
by Pino Toscano 09 Jan '20

09 Jan '20

--- python/Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/Makefile.am b/python/Makefile.am index 9aca202b5..c2f57a6ba 100644 --- a/python/Makefile.am +++ b/python/Makefile.am @@ -159,4 +159,4 @@ CLEANFILES += \ utils.c clean-local: - rm -rf build dist + rm -rf build dist __pycache__ t/__pycache__ -- 2.24.1

2 1

[PATCH] daemon: reorder internal static libs to fix linking
by Pino Toscano 09 Jan '20

09 Jan '20

Move the interal static libraries as the last items in the list of libraries of guestfsd, to make sure their symbols are used for all the other libraries. This is because GCC resolves the symbols looking at the arguments from the beginning to the end of the command line. This currently does not cause failures, however it "just works" because of the tricky situation set up. The situation is the following: 1) common/utils contains few utility sources: one of them is utils.c, which contains various functions -- for example guestfs_int_free_string_list and guestfs_int_drive_name --, it is built as utils.o, and bundled in the static library libutils.a 2) common/mlutils builds a OCaml library with bindings for some utility functions in libutils.a, in particular guestfs_int_drive_name (but not guestfs_int_free_string_list); there are two versions of this library, one OCaml library (dllmlcutils.so) that links with libutils.a, and one static library (libmlcutils.a), which cannot specify the libraries it links to (as it is static) 3) when the daemon is linked, the command line was the following (simplified): $ gcc [...] -o guestfsd guestfsd-9p.o other_daemon_object.o [...] \ ../common/utils/.libs/libutils.a [...] -lmlcutils [...] Some of the objects of the daemon itself use guestfs_int_free_string_list, and thus the compiler opens libutils.a (it is after the objects in the command line) and picks utils.o, which contains also guestfs_int_drive_name (not used directly in the daemon); when linking later on with libmlcutils.a, the symbols for this static library (like guestfs_int_drive_name) are already resolved, and thus all the symbols are resolved, and the linking succeeds This fragile situation can be easily broken by moving e.g. guestfs_int_drive_name out of common/utils/utils.c to a new source (say utils2.c) still built as part of libutils.a: since nothing before -lmlcutils actually needs to pick utils2.o from libutils.a for symbols, then GCC will not be able to resolve all the symbols in libmlcutils.a. As solution, move libutils.a (and other internal static libraries) as last libraries to link guestfsd to: this way, GCC knows where to find all the symbols needed by all the objects and libraries specified in the command line. --- daemon/Makefile.am | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/daemon/Makefile.am b/daemon/Makefile.am index ab3019cc1..25948dbe9 100644 --- a/daemon/Makefile.am +++ b/daemon/Makefile.am @@ -217,9 +217,6 @@ guestfsd_LDFLAGS = \ -L../bundled/ocaml-augeas \ -L../common/mlpcre guestfsd_LDADD = \ - ../common/errnostring/liberrnostring.la \ - ../common/protocol/libprotocol.la \ - ../common/utils/libutils.la \ camldaemon.o \ $(ACL_LIBS) \ $(CAP_LIBS) \ @@ -236,7 +233,10 @@ guestfsd_LDADD = \ $(TSK_LIBS) \ $(RPC_LIBS) \ $(YARA_LIBS) \ - $(OCAML_LIBS) + $(OCAML_LIBS) \ + ../common/errnostring/liberrnostring.la \ + ../common/protocol/libprotocol.la \ + ../common/utils/libutils.la guestfsd_CPPFLAGS = \ -DCAML_NAME_SPACE \ -- 2.24.1

2 1