May 2019 - Libguestfs - libguestfs List Archives

[nbdkit PATCH] nozero: Add notrim mode
by Eric Blake 13 May '19

13 May '19

It may be useful to test whether the client's use of NBD_CMD_FLAG_NO_HOLE makes a difference; do this by adding a mode to --filter=nozero to force a non-trimming zero write. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- filters/nozero/nbdkit-nozero-filter.pod | 19 +++++++---- filters/nozero/nozero.c | 45 ++++++++++++++++++++----- tests/test-nozero.sh | 27 +++++++++++++-- 3 files changed, 74 insertions(+), 17 deletions(-) diff --git a/filters/nozero/nbdkit-nozero-filter.pod b/filters/nozero/nbdkit-nozero-filter.pod index 7e06570..8e694bb 100644 --- a/filters/nozero/nbdkit-nozero-filter.pod +++ b/filters/nozero/nbdkit-nozero-filter.pod @@ -18,14 +18,19 @@ testing client or server fallbacks. =over 4 -=item B<zeromode=none|emulate> +=item B<zeromode=none|emulate|notrim> Optional, controls which mode the filter will use. Mode B<none> -(default) means that zero support is not advertised to the client; -mode B<emulate> means that zero support is emulated by the filter -using the plugin's C<pwrite> callback, regardless of whether the -plugin itself implemented the C<zero> callback with a more efficient -way to write zeroes. +(default) means that zero support is not advertised to the +client. Mode B<emulate> means that zero support is emulated by the +filter using the plugin's C<pwrite> callback, regardless of whether +the plugin itself implemented the C<zero> callback with a more +efficient way to write zeros. Since nbdkit E<ge> 1.13.4, mode +B<notrim> means that zero requests are forwarded on to the plugin, +except that the plugin will never see the NBDKIT_MAY_TRIM flag, to +determine if the client permitting trimming during zero operations +makes a difference (it is an error to request this mode if the plugin +does not support the C<zero> callback). =back @@ -55,4 +60,4 @@ Eric Blake =head1 COPYRIGHT -Copyright (C) 2018 Red Hat Inc. +Copyright (C) 2018-2019 Red Hat Inc. diff --git a/filters/nozero/nozero.c b/filters/nozero/nozero.c index 3ec6346..6e0ffa9 100644 --- a/filters/nozero/nozero.c +++ b/filters/nozero/nozero.c @@ -47,8 +47,12 @@ #define MAX_WRITE (64 * 1024 * 1024) -static char buffer[MAX_WRITE]; -static bool emulate; +static const char buffer[MAX_WRITE]; +static enum ZeroMode { + NONE, + EMULATE, + NOTRIM, +} zeromode; static int nozero_config (nbdkit_next_config *next, void *nxdata, @@ -56,7 +60,9 @@ nozero_config (nbdkit_next_config *next, void *nxdata, { if (strcmp (key, "zeromode") == 0) { if (strcmp (value, "emulate") == 0) - emulate = true; + zeromode = EMULATE; + else if (strcmp (value, "notrim") == 0) + zeromode = NOTRIM; else if (strcmp (value, "none") != 0) { nbdkit_error ("unknown zeromode '%s'", value); return -1; @@ -67,13 +73,31 @@ nozero_config (nbdkit_next_config *next, void *nxdata, } #define nozero_config_help \ - "zeromode=<MODE> Either 'none' (default) or 'emulate'.\n" \ + "zeromode=<MODE> Either 'none' (default), 'emulate', or 'notrim'.\n" \ + +/* Check that desired mode is supported by plugin. */ +static int +nozero_prepare (struct nbdkit_next_ops *next_ops, void *nxdata, void *handle) +{ + int r; + + if (zeromode == NOTRIM) { + r = next_ops->can_zero (nxdata); + if (r == -1) + return -1; + if (!r) { + nbdkit_error ("zeromode 'notrim' requires plugin zero support"); + return -1; + } + } + return 0; +} /* Advertise desired WRITE_ZEROES mode. */ static int nozero_can_zero (struct nbdkit_next_ops *next_ops, void *nxdata, void *handle) { - return emulate; + return zeromode != NONE; } static int @@ -81,11 +105,15 @@ nozero_zero (struct nbdkit_next_ops *next_ops, void *nxdata, void *handle, uint32_t count, uint64_t offs, uint32_t flags, int *err) { - assert (emulate); + assert (zeromode != NONE); + flags &= ~NBDKIT_FLAG_MAY_TRIM; + + if (zeromode == NOTRIM) + return next_ops->zero (nxdata, count, offs, flags, err); + while (count) { uint32_t size = MIN (count, MAX_WRITE); - if (next_ops->pwrite (nxdata, buffer, size, offs, - flags & ~NBDKIT_FLAG_MAY_TRIM, err) == -1) + if (next_ops->pwrite (nxdata, buffer, size, offs, flags, err) == -1) return -1; offs += size; count -= size; @@ -99,6 +127,7 @@ static struct nbdkit_filter filter = { .version = PACKAGE_VERSION, .config = nozero_config, .config_help = nozero_config_help, + .prepare = nozero_prepare, .can_zero = nozero_can_zero, .zero = nozero_zero, }; diff --git a/tests/test-nozero.sh b/tests/test-nozero.sh index 1282586..fc22420 100755 --- a/tests/test-nozero.sh +++ b/tests/test-nozero.sh @@ -39,12 +39,14 @@ sock3=`mktemp -u` sock4=`mktemp -u` sock5a=`mktemp -u` sock5b=`mktemp -u` +sock6=`mktemp -u` files="nozero1.img nozero1.log $sock1 nozero1.pid nozero2.img nozero2.log $sock2 nozero2.pid nozero3.img nozero3.log $sock3 nozero3.pid nozero4.img nozero4.log $sock4 nozero4.pid nozero5.img nozero5a.log nozero5b.log $sock5a $sock5b - nozero5a.pid nozero5b.pid" + nozero5a.pid nozero5b.pid + nozero6.img nozero6.log $sock6 nozero6.pid" rm -f $files # Prep images, and check that qemu-io understands the actions we plan on @@ -54,6 +56,7 @@ cp nozero1.img nozero2.img cp nozero1.img nozero3.img cp nozero1.img nozero4.img cp nozero1.img nozero5.img +cp nozero1.img nozero6.img if ! qemu-io -f raw -d unmap -c 'w -z -u 0 1M' nozero1.img; then echo "$0: missing or broken qemu-io" rm nozero?.img @@ -82,17 +85,20 @@ cleanup () cat nozero5a.log || : echo "Log 5b file contents:" cat nozero5b.log || : + echo "Log 6 file contents:" + cat nozero6.log || : rm -f $files } cleanup_fn cleanup -# Run four parallel nbdkit; to compare the logs and see what changes. +# Run several parallel nbdkit; to compare the logs and see what changes. # 1: unfiltered, to check that qemu-io sends ZERO request and plugin trims # 2: log before filter with zeromode=none (default), to ensure no ZERO request # 3: log before filter with zeromode=emulate, to ensure ZERO from client # 4: log after filter with zeromode=emulate, to ensure no ZERO to plugin # 5a/b: both sides of nbd plugin: even though server side does not advertise # ZERO, the client side still exposes it, and just skips calling nbd's .zero +# 6: log after filter with zeromode=notrim, to ensure plugin does not trim start_nbdkit -P nozero1.pid -U $sock1 --filter=log \ file logfile=nozero1.log nozero1.img start_nbdkit -P nozero2.pid -U $sock2 --filter=log --filter=nozero \ @@ -105,6 +111,8 @@ start_nbdkit -P nozero5a.pid -U $sock5a --filter=log --filter=nozero \ file logfile=nozero5a.log nozero5.img start_nbdkit -P nozero5b.pid -U $sock5b --filter=log \ nbd logfile=nozero5b.log socket=$sock5a +start_nbdkit -P nozero6.pid -U $sock6 --filter=nozero --filter=log \ + file logfile=nozero6.log nozero6.img zeromode=notrim # Perform the zero write. qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock1" @@ -112,6 +120,7 @@ qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock2" qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock3" qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock4" qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock5b" +qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock6" # Check for expected ZERO vs. WRITE results grep 'connection=1 Zero' nozero1.log @@ -129,9 +138,23 @@ if grep 'connection=1 Zero' nozero5a.log; then echo "nbdkit should have converted zero into write before nbd plugin" exit 1 fi +grep 'connection=1 Zero' nozero6.log # Sanity check on contents - all 5 files should read identically cmp nozero1.img nozero2.img cmp nozero2.img nozero3.img cmp nozero3.img nozero4.img cmp nozero4.img nozero5.img +cmp nozero5.img nozero6.img + +# Sanity check on sparseness; only image 1 should be sparse +if test "$(stat -c %b nozero1.img)" = "$(stat -c %b nozero2.img)"; then + echo "nozero1.img was not trimmed" + exit 1 +fi +for i in 3 4 5 6; do + if test "$(stat -c %b nozero2.img)" != "$(stat -c %b nozero$i.img)"; then + echo "nozero$i.img was trimmed by mistake" + exit 1 + fi +done -- 2.20.1

2 1

[nbdkit PATCH] plugins: Use static buffer for plugin_zeroes
by Eric Blake 13 May '19

13 May '19

No need to calloc/free a buffer every time NBD_CMD_WRITE_ZEROES has to fall back to a .pread call. Just reserve the maximum buffer up front in our bss. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- I noticed that buf was a candidate for CLEANUP_FREE, but then further noticed that we can avoid the calloc/free altogether if we don't mind the bss being 64M larger. server/plugins.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/server/plugins.c b/server/plugins.c index 947bb6d..acdfa1f 100644 --- a/server/plugins.c +++ b/server/plugins.c @@ -594,7 +594,7 @@ plugin_zero (struct backend *b, struct connection *conn, uint32_t count, uint64_t offset, uint32_t flags, int *err) { struct backend_plugin *p = container_of (b, struct backend_plugin, backend); - char *buf; + static const char buf[MAX_REQUEST_SIZE]; uint32_t limit; int r = -1; bool may_trim = flags & NBDKIT_FLAG_MAY_TRIM; @@ -639,12 +639,7 @@ plugin_zero (struct backend *b, struct connection *conn, assert (p->plugin.pwrite || p->plugin._pwrite_old); flags &= ~NBDKIT_FLAG_MAY_TRIM; threadlocal_set_error (0); - limit = count < MAX_REQUEST_SIZE ? count : MAX_REQUEST_SIZE; - buf = calloc (limit, 1); - if (!buf) { - *err = ENOMEM; - return -1; - } + limit = count < sizeof (buf) ? count : sizeof (buf); while (count) { r = plugin_pwrite (b, conn, buf, limit, offset, flags, err); @@ -656,7 +651,6 @@ plugin_zero (struct backend *b, struct connection *conn, } *err = errno; - free (buf); errno = *err; done: -- 2.20.1

2 1

Welcome Hiroyuki Katsura as Google Summer of Code 2019 student
by Pino Toscano 13 May '19

13 May '19

Dear libguestfs developers and users, it is my pleasure to welcome Hiroyuki Katsura as Google Summer of Code 2019 student for libguestfs! He will work on adding Rust bindings, to allow using libguestfs in Rust applications. You can read the project submission here: https://summerofcode.withgoogle.com/projects/#6730007286644736 I'm the main mentor of this project, with Rich Jones, and Martin Kletzander as co-mentors. Hiroyuki: feel free to ask away about anything, from design questions to doubts about the existing code, etc. Welcome again, Hiroyuki! -- Pino Toscano

2 1

Re: [Libguestfs] [libguestfs/libguestfs] virt-tar-out (#37)
by Richard W.M. Jones 13 May '19

13 May '19

On Sun, May 12, 2019 at 04:11:08PM -0700, Jun Aruga wrote: > I am using libguestfs tool on Fedora 30. > > libguestfs version: 1.40.2 > libguestfs-tools-c RPM package version: libguestfs-tools-c-1.40.2-4.fc30.x86_64 > > > I have a question about `virt-tar-out`. > > ``` > $ rpm -qf /usr/bin/virt-tar-out > libguestfs-tools-c-1.40.2-4.fc30.x86_64 > ``` > > Current master's `fish/virt-tar-out` https://github.com/libguestfs/libguestfs/blob/master/fish/virt-tar-out is almost same with my local `/usr/bin/virt-tar-out` except bash path. > > ``` > $ diff fish/virt-tar-out /usr/bin/virt-tar-out > 1c1 > < #!/bin/bash - > --- > > #!/usr/bin/bash - > ``` Something in Fedora itself rewrites this path, it's nothing to do with libguestfs and in any case this should not cause a problem. > ## case1: convert CentOS7 aarch64 iso image to the tar file. > > ``` > $ wget http://mirror.centos.org/altarch/7.6.1810/isos/aarch64/CentOS-7-aarch64-Min… > ``` > > ``` > $ virt-tar-out -a CentOS-7-aarch64-Minimal-1810.iso / foo.tar > virt-tar-out: no operating system was found on this disk libguestfs cannot inspect installation ISOs which is why this fails. > If using guestfish ‘-i’ option, remove this option and instead > use the commands ‘run’ followed by ‘list-filesystems’. > You can then mount filesystems you want by hand using the > ‘mount’ or ‘mount-ro’ command. > > If using guestmount ‘-i’, remove this option and choose the > filesystem(s) you want to see by manually adding ‘-m’ option(s). > Use ‘virt-filesystems’ to see what filesystems are available. > > If using other virt tools, this disk image won’t work > with these tools. Use the guestfish equivalent commands > (see the virt tool manual page). > ``` > > This works when running `guestfish` directly like this. > > ``` > $ guestfish -a CentOS-7-aarch64-Minimal-1810.iso --ro <<EOF > run > list-filesystems > EOF > /dev/sda: iso9660 > ``` > > ``` > $ guestfish --ro -a CentOS-7-aarch64-Minimal-1810.iso -m /dev/sda tar-out / foo.tar > > $ echo $? > 0 > > $ du -sh foo.tar > 768M foo.tar > ``` > > ## case2: convert CentOS7 ppc64 iso image to the tar file. > > ``` > $ wget http://mirror.centos.org/altarch/7.6.1810/isos/ppc64/CentOS-7-ppc64-Minimal… > ``` > > ``` > $ virt-tar-out -a CentOS-7-ppc64-Minimal-1810.iso foo.tar > virt-tar-out: no operating system was found on this disk > > If using guestfish ‘-i’ option, remove this option and instead > use the commands ‘run’ followed by ‘list-filesystems’. > You can then mount filesystems you want by hand using the > ‘mount’ or ‘mount-ro’ command. > > If using guestmount ‘-i’, remove this option and choose the > filesystem(s) you want to see by manually adding ‘-m’ option(s). > Use ‘virt-filesystems’ to see what filesystems are available. > > If using other virt tools, this disk image won’t work > with these tools. Use the guestfish equivalent commands > (see the virt tool manual page). > ``` > > This works when running `guestfish` directly like this. > > ``` > $ guestfish -a CentOS-7-ppc64-Minimal-1810.iso --ro <<EOF > run > list-filesystems > EOF > /dev/sda1: iso9660 > ``` > > ``` > $ guestfish --ro -a CentOS-7-ppc64-Minimal-1810.iso -m /dev/sda1 tar-out / foo.tar > > $ echo $? > 0 > > $ du -sh foo.tar > 877M foo.tar > ``` > > Why is `virt-tar-out` command not working? > I should use `guestfish` command directly? virt-tar-out is just a convenient, thin wrapper around guestfish, so use guestfish if that works for you. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v

1 0

[PATCH 00/11] v2v: Change virt-v2v to use nbdkit for input in several modes.
by Richard W.M. Jones 10 May '19

10 May '19

This series (except the last one) changes virt-v2v to use nbdkit for several input modes: -i vmx -it vddk: No change in functionality, as this already uses nbdkit-vddk-plugin, but the code is refactored for the other modes to use. -i libvirtxml: Use nbdkit-curl-plugin instead of qemu curl. vCenter: Use nbdkit-curl-plugin instead of qemu curl. xen: Use nbdkit-ssh-plugin instead of qemu ssh. -i vmx -it ssh: Use nbdkit-ssh-plugin instead of qemu ssh. I was hoping to post this patch series when it was longer and showed some benefits. However those patches aren't ready quite yet, but these ones are ready for review. However some of the future benefits will be: * using nbdkit-rate-plugin to bandwidth limit the input side * better support for modern SSH encryption * possibility of SSH password auth (instead of requiring ssh-agent) I have tested this locally. Unfortunately none of the existing tests cover any of these input cases. The last patch reduces the minimum required version of nbdkit from 1.12 (not yet available) to 1.11, so this patch can be tested in Fedora 30. It is obviously not meant to be applied. Rich.

2 12

Re: [Libguestfs] libguestfs Windows Build Support
by Richard W.M. Jones 01 May '19

01 May '19

On Wed, May 01, 2019 at 04:33:49PM +0300, Matan wrote: > Hello Richard, > > I've recently encountered with libguestfs while looking for a library > that provides parsing capabilities for VMDK formats, as part of a > utility I'm working on. I've read the documentation and and few of the > source code but couldn't find an answer for my question. > Then, I've seen your that you are active on the mailing list, so I'm > reaching out to you. > > Do you know of any way to build libguestfs on Windows? so that I'll > have a compiled PE image that will be able to parse and mount VMDK > files? It used to build on Windows at some time in the very distant past, but it doesn't now. I would suggest using virtualization similar to run up a Linux environment. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

1 0