September 2019 - Libguestfs - libguestfs List Archives

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones 29 Nov '19

29 Nov '19

My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 18

[PATCH libnbd v2 0/2] Implement systemd socket activation.
by Richard W.M. Jones 15 Nov '19

15 Nov '19

v1 was posted here: https://www.redhat.com/archives/libguestfs/2019-September/thread.html#00337 v2: - Drop the first patch. - Hopefully fix the multiple issues with fork-safety and general behaviour on error paths. Note this requires execvpe for which there seems to be no equivalent on FreeBSD, except some kind of tedious path parsing (but can we assign to environ?) Rich.

2 8

[nbdkit PATCH 0/2] Fix nbdkit --run when nbdkit hits assertion
by Eric Blake 19 Oct '19

19 Oct '19

Found while working on the retry filter. Swap the order of the two patches to see nbdkit ignore assertion failures with status 0. Eric Blake (2): server: Propagate unexpected nbdkit failure with --run tests: Enhance captive test server/captive.c | 43 ++++++++++++++++++++++++++++++---------- tests/test-captive.sh | 46 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 75 insertions(+), 14 deletions(-) -- 2.21.0

2 6

[NBDKIT SECURITY] Denial of Service / Amplification Attack in nbdkit
by Richard W.M. Jones 01 Oct '19

01 Oct '19

We have discovered a potential Denial of Service / Amplification Attack in nbdkit. Lifecycle --------- Reported: 2019-09-11 Fixed: 2019-09-11 Published: 2019-09-12 There is no CVE number assigned for this issue yet, but the bug is being categorized and processed by Red Hat's security team which may result in a CVE being published later. Credit ------ Reported and patched by Richard W.M. Jones <rjones(a)redhat.com>. Reviewed by Eric Blake <eblake(a)redhat.com> and Daniel Berrangé <berrange(a)redhat.com>. Description ----------- nbdkit is a Network Block Device (NBD) server with multiple plugins. An attacker, by connecting to nbdkit’s TCP port, is able to make the current plugin’s .open() callback run. For some plugins this callback does a considerable amount of work, for example connecting to a remote machine (nbdkit-ssh-plugin) or launching a small VM (nbdkit-guestfs-plugin). Thus for a small amount of work done by the attacker (opening many TCP connections but not sending any data), a large amount of work is done by nbdkit (eg. making many full ssh connections to a remote machine or launching hundreds of VMs). A related problem is that if nbdkit is configured to use TLS for client authentication, even clients which are not authorized are able to attack nbdkit in this way because all they have to do is open a TCP connection. Mitigating this is that the current plugin is NOT selected by the attacker, but is part of the server configuration. Therefore the effectiveness of the attack depends on the particular configuration. There are two situations in which you will NOT be affected: * You are not exposing nbdkit over TCP (eg. using nbdkit -U), or the TCP port is not accessible to untrusted attackers (eg. private LAN, firewall rules). * The plugin you are using does not do much work in its .open() callback. For example the memory plugin does nothing in its callback so is not vulnerable: https://github.com/libguestfs/nbdkit/blob/afbe7f11098df256acf8eafa9d4ec1e3c… Test if nbdkit is a vulnerable version -------------------------------------- Run the following command (requires bash): nbdkit -fv null --run 'sleep 1 >/dev/tcp/localhost/10809' 2>&1 | grep -q 'open readonly' && echo vulnerable || echo patched Workarounds ----------- It is recommended to upgrade to a fixed version of nbdkit (see next section). However if this cannot be done, apply network filtering (eg. firewall, TCP wrappers, etc.) to ensure that untrusted clients cannot connect to nbdkit’s public TCP port. Or use a Unix domain socket instead of a loopback TCP connection. Fixes ----- This affects all versions of nbdkit. A fix is available for 1.12, 1.14 and the current development branch. * development branch (1.15) https://github.com/libguestfs/nbdkit/commit/c05686f9577fa91b6a3a4d8c065954c… or use nbdkit >= 1.15.1 from http://download.libguestfs.org/nbdkit/1.15-development/ * stable branch 1.14 https://github.com/libguestfs/nbdkit/commit/e06cde00659ff97182173d0e33fff78… or use nbdkit >= 1.14.1 from http://download.libguestfs.org/nbdkit/1.14-stable/ * stable branch 1.12 https://github.com/libguestfs/nbdkit/commit/22b30adb796bb6dca264a38598f80b8… or use nbdkit >= 1.12.7 from http://download.libguestfs.org/nbdkit/1.12-stable/ -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

2 3

[PATCH nbdkit v2 0/4] info: Add mode for sending back server time.
by Richard W.M. Jones 30 Sep '19

30 Sep '19

v1 was: https://www.redhat.com/archives/libguestfs/2019-September/thread.html#00361 v2: - Adds a patch to rename the reflection plugin to the info plugin. - Adds tests. Rich.

2 11

[nbdkit PATCH v2 0/7] Spec compliance patches
by Eric Blake 30 Sep '19

30 Sep '19

Since the v1 series (0/4, at [1]), I've applied patches 1 and 2, rewritten patch 3 [Forbid NUL in export and context names] into patch 4 here, patch 4 there turned into patch 6 here, and everything else here is new. [1]https://www.redhat.com/archives/libguestfs/2019-September/msg00180.html I don't know if there is a handy reusable function for checking whether a string contains valid UTF-8 while still complying with our licensing, so for now I don't bother to check that. But fixing this meant I also proposed a qemu 4.2 patch on export name length: https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg06925.html so part of the test I added only really covers nbdkit when using qemu 4.1, because otherwise qemu fails early without involving nbdkit. Eric Blake (7): server: Reject -e with too long of a name server: Expose -e in $exportname during --run server: Switch to fixed-length conn->exportname server: Improve check of string validity server: Allow longer NBD_OPT server: Fix OPT_GO on different export than SET_META_CONTEXT server: Better newstyle .open failure handling docs/nbdkit-captive.pod | 12 ++- server/internal.h | 4 +- server/Makefile.am | 1 + server/captive.c | 22 ++++- server/connections.c | 7 -- server/main.c | 6 ++ server/protocol-handshake-newstyle.c | 117 ++++++++++++++++++++------- server/protocol-handshake-oldstyle.c | 3 + tests/Makefile.am | 2 + tests/test-long-name.sh | 101 +++++++++++++++++++++++ 10 files changed, 231 insertions(+), 44 deletions(-) create mode 100755 tests/test-long-name.sh -- 2.21.0

2 16

[PATCH nbdkit v3 0/3] Add new retry filter.
by Richard W.M. Jones 30 Sep '19

30 Sep '19

v2 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00221.html I think this is more like "the one". It handles reopen failing correctly, and there is a second test for that. I also ran my sshd tests locally and it worked in all scenarios I could think up (except of course sshd not being available at the start, but we want that to fail). Rich.

2 15

[p2v PATCH] make-disk: set p2v.local as hostname
by Pino Toscano 28 Sep '19

28 Sep '19

Set p2v.local as hostname for the p2v disk image, so it will be fixed, instead of whatever is set in the used templates. This is also somewhat important during the distribution update, and installation of our packages: there may be distribution packages actually using the hostname to initialize stuff (e.g. configuration values), and thus the unset value "(none)" triggers errors. The additional command to set the hostname is for the appliance itself: the hostname will be set on the running kernel, which is the appliance own, so getting it later on with hostname(1) works. --- virt-p2v-make-disk.in | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/virt-p2v-make-disk.in b/virt-p2v-make-disk.in index 1bec4e3..132211d 100644 --- a/virt-p2v-make-disk.in +++ b/virt-p2v-make-disk.in @@ -233,11 +233,16 @@ fi # Run virt-builder. Note we controversially assume systemd here. We # could provide a sysvinit fallback if required. +# The manual 'hostname' invocation is needed to set the hostname +# also for the appliance itself, so scriptlets can properly use +# the hostname we want. virt-builder "$osversion" \ $verbose_option \ --output "$output" \ $arch_option \ $preinstall_args \ + --hostname p2v.local \ + --run-command 'hostname p2v.local' \ --update \ --install "$install" \ --root-password password:p2v \ -- 2.21.0

2 1

[PATCH nbdkit 1/2] common/include: Add function for subtracting struct timeval.
by Richard W.M. Jones 28 Sep '19

28 Sep '19

--- common/include/test-tvdiff.c | 75 +++++++++++++++++++++++++++++------- common/include/tvdiff.h | 13 ++++++- 2 files changed, 74 insertions(+), 14 deletions(-) diff --git a/common/include/test-tvdiff.c b/common/include/test-tvdiff.c index 9cbcfc0..abefb2e 100644 --- a/common/include/test-tvdiff.c +++ b/common/include/test-tvdiff.c @@ -36,7 +36,6 @@ #include <stdlib.h> #include <stdint.h> #include <inttypes.h> -#include <assert.h> #include <sys/time.h> #include "tvdiff.h" @@ -45,37 +44,87 @@ * implementation. */ +#define TEST_TVDIFF(tv1, tv2, expected) \ + do { \ + int64_t actual = tvdiff_usec (&(tv1), &(tv2)); \ + \ + if (actual != (expected)) { \ + fprintf (stderr, \ + "%s: unexpected result %" PRIi64 ", expecting %" PRIi64 "\n", \ + argv[0], actual, (int64_t) (expected)); \ + errors++; \ + } \ + } while (0) + +#define TEST_SUBTRACT(tv1, tv2, exp_sec, exp_usec) \ + do { \ + struct timeval z; \ + \ + subtract_timeval (&tv1, &tv2, &z); \ + if (z.tv_sec != (exp_sec) || z.tv_usec != (exp_usec)) { \ + fprintf (stderr, \ + "%s: unexpected (%ld, %d), expecting (%ld, %d)\n", \ + argv[0], \ + (long) z.tv_sec, (int) z.tv_usec, \ + (long) (exp_sec), (int) (exp_usec)); \ + errors++; \ + } \ + } while (0) + int -main (void) +main (int argc, char *argv[]) { struct timeval tv1, tv2; + unsigned errors = 0; tv1.tv_sec = 1000; tv1.tv_usec = 1; - assert (tvdiff_usec (&tv1, &tv1) == 0); + TEST_TVDIFF (tv1, tv1, 0); + TEST_SUBTRACT (tv1, tv1, 0, 0); + tv2.tv_sec = 1000; tv2.tv_usec = 2; - assert (tvdiff_usec (&tv1, &tv2) == 1); - assert (tvdiff_usec (&tv2, &tv1) == -1); + TEST_TVDIFF (tv1, tv2, 1); + TEST_SUBTRACT (tv1, tv2, 0, 1); + TEST_TVDIFF (tv2, tv1, -1); + TEST_SUBTRACT (tv2, tv1, 0, -1); + tv2.tv_sec = 1000; tv2.tv_usec = 3; - assert (tvdiff_usec (&tv1, &tv2) == 2); - assert (tvdiff_usec (&tv2, &tv1) == -2); + TEST_TVDIFF (tv1, tv2, 2); + TEST_SUBTRACT (tv1, tv2, 0, 2); + TEST_TVDIFF (tv2, tv1, -2); + TEST_SUBTRACT (tv2, tv1, 0, -2); + tv2.tv_sec = 1001; tv2.tv_usec = 0; - assert (tvdiff_usec (&tv1, &tv2) == 999999); - assert (tvdiff_usec (&tv2, &tv1) == -999999); + TEST_TVDIFF (tv1, tv2, 999999); + TEST_SUBTRACT (tv1, tv2, 0, 999999); + TEST_TVDIFF (tv2, tv1, -999999); + TEST_SUBTRACT (tv2, tv1, 0, -999999); tv1.tv_sec = 1000; tv1.tv_usec = 999999; tv2.tv_sec = 1001; tv2.tv_usec = 1; - assert (tvdiff_usec (&tv1, &tv2) == 2); - assert (tvdiff_usec (&tv2, &tv1) == -2); + TEST_TVDIFF (tv1, tv2, 2); + TEST_SUBTRACT (tv1, tv2, 0, 2); + TEST_TVDIFF (tv2, tv1, -2); + TEST_SUBTRACT (tv2, tv1, 0, -2); + + tv1.tv_sec = 1000; + tv1.tv_usec = 1; + tv2.tv_sec = 1001; + tv2.tv_usec = 2; + TEST_TVDIFF (tv1, tv2, 1000001); + TEST_SUBTRACT (tv1, tv2, 1, 1); + TEST_TVDIFF (tv2, tv1, -1000001); + TEST_SUBTRACT (tv2, tv1, -1, -1); /* Test that an arbitrary tv is equal to itself. */ gettimeofday (&tv1, NULL); - assert (tvdiff_usec (&tv1, &tv1) == 0); + TEST_TVDIFF (tv1, tv1, 0); + TEST_SUBTRACT (tv1, tv1, 0, 0); - exit (EXIT_SUCCESS); + exit (errors == 0 ? EXIT_SUCCESS : EXIT_FAILURE); } diff --git a/common/include/tvdiff.h b/common/include/tvdiff.h index db86b66..cec8345 100644 --- a/common/include/tvdiff.h +++ b/common/include/tvdiff.h @@ -40,7 +40,7 @@ #include <sys/time.h> /* Return the number of µs (microseconds) in y - x. */ -static int64_t +static inline int64_t tvdiff_usec (const struct timeval *x, const struct timeval *y) { int64_t usec; @@ -50,4 +50,15 @@ tvdiff_usec (const struct timeval *x, const struct timeval *y) return usec; } +/* Return timeval difference as another struct timeval. z = y - x. */ +static inline void +subtract_timeval (const struct timeval *x, const struct timeval *y, + struct timeval *z) +{ + int64_t usec = tvdiff_usec (x, y); + + z->tv_sec = usec / 1000000; + z->tv_usec = usec % 1000000; +} + #endif /* NBDKIT_TVDIFF_H */ -- 2.23.0

1 2