October 2019 - Libguestfs - libguestfs List Archives

[PATCH libnbd 0/2] api: Add support for AF_VSOCK.
by Richard W.M. Jones 10 Feb '20

10 Feb '20

This is a series of patches to libnbd and nbdkit adding AF_VSOCK support. On the host side it allows you to start an nbdkit instance which listens on a virtio-vsock socket: $ ./nbdkit -fv --vsock memory 1G ... nbdkit: debug: bound to vsock 2:10809 On the guest side you can then use libnbd to connect to the server: $ ./run nbdsh -c 'h.connect_vsock(2, 10809)' -c 'print(h.get_size())' 1073741824 $ ./run nbdfuse mp --vsock 2 10809 & $ ll mp/ total 0 -rw-rw-rw-. 1 rjones rjones 1073741824 Oct 18 16:23 nbd $ dd if=/dev/random of=mp/nbd bs=1024 count=100 conv=notrunc,nocreat dd: warning: partial read (84 bytes); suggest iflag=fullblock 0+100 records in 0+100 records out 6851 bytes (6.9 kB, 6.7 KiB) copied, 0.013797 s, 497 kB/s (Performance of FUSE is not great, it should be better using raw libnbd.) I mainly wrote this to show that it can be done. It's unclear if this would be faster or slower than the usual way that NBD devices are exposed to guests via virtio-blk/-scsi. https://wiki.qemu.org/Features/VirtioVsock Thanks: Stefan Hajnoczi for help with debugging this.

3 10

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones 29 Nov '19

29 Nov '19

My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 18

[PATCH libnbd v2 0/2] Implement systemd socket activation.
by Richard W.M. Jones 15 Nov '19

15 Nov '19

v1 was posted here: https://www.redhat.com/archives/libguestfs/2019-September/thread.html#00337 v2: - Drop the first patch. - Hopefully fix the multiple issues with fork-safety and general behaviour on error paths. Note this requires execvpe for which there seems to be no equivalent on FreeBSD, except some kind of tedious path parsing (but can we assign to environ?) Rich.

2 8

installing supermin on debian10
by Keresztes Péter-Zoltán 06 Nov '19

06 Nov '19

Hello I am trying to install supermin on debian 10 (buster) and I am getting the following error: checking /usr/sbin/mke2fs -t or -T... -t checking for EXT2FS... no configure: error: in `/root/supermin-5.1.16': configure: error: The pkg-config script could not be found or is too old. Make sure it is in your PATH or set the PKG_CONFIG environment variable to the full path to pkg-config. Alternatively, you may set the environment variables EXT2FS_CFLAGS and EXT2FS_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. To get pkg-config, see <http://pkg-config.freedesktop.org/>. See `config.log' for more details Note that I have installed all the required dependency packages which I could find in the README. Can anyone help making this work? Regards, Peter

3 2

[PATCH libnbd] api: Allow NBD URIs to be restricted.
by Richard W.M. Jones 04 Nov '19

04 Nov '19

Previous discussion: https://www.redhat.com/archives/libguestfs/2019-August/msg00102.html Last night I experimentally added support for URIs that contain the query parameter tls-psk-file, as part of rewriting the tests to cover more of the URI code. So you can now have a URI like: nbds://alice@localhost/?tls-psk-file=keys.psk However there's an obvious security problem here because now any libnbd program which takes URIs from less trusted sources will open a local file under the user's control. So it's time to restrict what can appear in URIs. I've added three new APIs for this purpose, see generator/generator in this patch for documentation. The defaults are fairly liberal, except we do prevent opening local files (except socket) by default. Rich.

2 3

[PATCH] fish: add option --blocksize for disks
by Tuan Hoang 29 Oct '19

29 Oct '19

When --blocksize <val> is provided, qemu command line would add physical_block_size=<val>,physical_logical_size=<val> to -device directive. Example: qemu-kvm \ -drive file=raw.img,cache=writeback,id=hd0,if=none \ -device scsi-hd,drive=hd0,physical_block_size=4096,logical_block_size=4096 \ Signed-off-by: Tuan Hoang <tmhoang(a)linux.ibm.com> --- fish/fish.c | 5 +++++ generator/c.ml | 2 ++ lib/launch-direct.c | 4 ++++ 3 files changed, 11 insertions(+) diff --git a/fish/fish.c b/fish/fish.c index 2070e37..dd58d23 100644 --- a/fish/fish.c +++ b/fish/fish.c @@ -124,6 +124,7 @@ usage (int status) " -h|--cmd-help List available commands\n" " -h|--cmd-help cmd Display detailed help on ‘cmd’\n" " -a|--add image Add image\n" + " --blocksize Specify physical and logical blocksize\n" " -c|--connect uri Specify libvirt URI for -d option\n" " --csh Make --listen csh-compatible\n" " -d|--domain guest Add disks from libvirt guest\n" @@ -190,6 +191,7 @@ main (int argc, char *argv[]) static const char options[] = "a:c:d:Df:h::im:nN:rvVwx"; static const struct option long_options[] = { { "add", 1, 0, 'a' }, + { "blocksize", 1, 0, 0 }, { "cmd-help", 2, 0, 'h' }, { "connect", 1, 0, 'c' }, { "csh", 0, 0, 0 }, @@ -259,6 +261,9 @@ main (int argc, char *argv[]) display_long_options (long_options); else if (STREQ (long_options[option_index].name, "short-options")) display_short_options (options); + else if (STREQ (long_options[option_index].name, "blocksize")) { + blocksize = strtol(optarg, NULL, 0); + } else if (STREQ (long_options[option_index].name, "listen")) remote_control_listen = 1; else if (STREQ (long_options[option_index].name, "remote")) { diff --git a/generator/c.ml b/generator/c.ml index 86f7d89..61aa50d 100644 --- a/generator/c.ml +++ b/generator/c.ml @@ -439,6 +439,8 @@ extern \"C\" { (__GNUC__ * 10000 + __GNUC_MINOR__ * 100 + __GNUC_PATCHLEVEL__) #endif +extern int blocksize; + /* Define GUESTFS_WARN_DEPRECATED=1 to warn about deprecated API functions. */ #define GUESTFS_DEPRECATED_NO_REPLACEMENT #define GUESTFS_DEPRECATED_REPLACED_BY(s) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index ee2dcb8..57d1ad6 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -49,6 +49,8 @@ #include "guestfs_protocol.h" #include "qemuopts.h" +int blocksize = 0; + /* Per-handle data. */ struct backend_direct_data { pid_t pid; /* Qemu PID. */ @@ -315,6 +317,8 @@ add_drive (guestfs_h *g, struct backend_direct_data *data, start_list ("-device") { append_list ("scsi-hd"); append_list_format ("drive=hd%zu", i); + append_list_format ("physical_block_size=%d", blocksize); + append_list_format ("logical_block_size=%d", blocksize); if (drv->disk_label) append_list_format ("serial=%s", drv->disk_label); } end_list (); -- 2.21.0

1 1

[nbdkit PATCH 0/2] Fix nbdkit --run when nbdkit hits assertion
by Eric Blake 19 Oct '19

19 Oct '19

Found while working on the retry filter. Swap the order of the two patches to see nbdkit ignore assertion failures with status 0. Eric Blake (2): server: Propagate unexpected nbdkit failure with --run tests: Enhance captive test server/captive.c | 43 ++++++++++++++++++++++++++++++---------- tests/test-captive.sh | 46 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 75 insertions(+), 14 deletions(-) -- 2.21.0

2 6

[PATCH nbdkit] Add support for AF_VSOCK.
by Richard W.M. Jones 18 Oct '19

18 Oct '19

This is a series of patches to libnbd and nbdkit adding AF_VSOCK support. On the host side it allows you to start an nbdkit instance which listens on a virtio-vsock socket: $ ./nbdkit -fv --vsock memory 1G ... nbdkit: debug: bound to vsock 2:10809 On the guest side you can then use libnbd to connect to the server: $ ./run nbdsh -c 'h.connect_vsock(2, 10809)' -c 'print(h.get_size())' 1073741824 $ ./run nbdfuse mp --vsock 2 10809 & $ ll mp/ total 0 -rw-rw-rw-. 1 rjones rjones 1073741824 Oct 18 16:23 nbd $ dd if=/dev/random of=mp/nbd bs=1024 count=100 conv=notrunc,nocreat dd: warning: partial read (84 bytes); suggest iflag=fullblock 0+100 records in 0+100 records out 6851 bytes (6.9 kB, 6.7 KiB) copied, 0.013797 s, 497 kB/s (Performance of FUSE is not great, it should be better using raw libnbd.) I mainly wrote this to show that it can be done. It's unclear if this would be faster or slower than the usual way that NBD devices are exposed to guests via virtio-blk/-scsi. https://wiki.qemu.org/Features/VirtioVsock Thanks: Stefan Hajnoczi for help with debugging this.

3 4

[PATCH nbdkit v2] Add support for AF_VSOCK.
by Richard W.M. Jones 18 Oct '19

18 Oct '19

v1 was discussed here: https://www.redhat.com/archives/libguestfs/2019-October/thread.html#00100 v2: - Bind to VMADDR_CID_ANY (instead of HOST) and update the documentation accordingly. - Don't bother with SOCK_CLOEXEC fallback path that can never be used. Rich.

1 1