November 2017 - Libguestfs - libguestfs List Archives

[PATCH REPOST 1/2] common/mlstdutils: Add return statement.
by Richard W.M. Jones 21 Nov '17

21 Nov '17

No change, just reposting without the "for discussion" tag. I think we should allow this as it seems like a nice coding style for a limited subset of imperative-style code. Rich.

1 2

[PATCH v3 0/2] common/mlstdutils: Extend the List module.
by Richard W.M. Jones 21 Nov '17

21 Nov '17

v2 -> v3: - Renamed List.assoc_ -> List.assoc_lbl. - Rebased on top of current master branch. Rich.

1 2

Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
by Max Reitz 20 Nov '17

20 Nov '17

On 2017-11-15 18:44, Gandalf Corvotempesta wrote: > XVA is a tar archive. > I don't think would be possible to directly use the image stored inside > without extracting and merging each chunks > > Any random reads would be impossible to do, only a huge sequential dump to > build the raw image Well, you can't just add support to qemu-img alone either. Every image format supported by qemu-img is one supported by qemu as a whole and thus needs a proper block driver that needs to support random accesses as well. Max

4 27

Effect of qemu-img convert -m and -W options
by Richard W.M. Jones 20 Nov '17

20 Nov '17

[CC to qemu-devel since I'm obviously doing something wrong here, I'm just not sure what.] I was getting ready to add multiple threads to ‘qemu-img convert’ (the longest part of v2v conversions) when I noticed that it had them already! (To be fair this was only added in February this year so no wonder we didn't notice.) To enable parallel convert we would need to use the ‘qemu-img convert -W’ option (which allows out-of-order writes to the destination) and ‘qemu-img convert -m <#num-coroutines>’ option to select the parallelism (defaults to 8). The documentation refers to coroutines but I verified from strace that it is using real threads. I did some testing to see what effect this has. For this I used a large guest image which is approximately a third full of random data (the rest being sparsely allocated): Source format: raw Source virtual size: 100 GB Source used space: 31 GB Target format: qcow2 Version: qemu-img-2.10.0-7.fc28.x86_64 Conversion command: rm -f /to/target time qemu-img convert [-W] [-m ##] -f raw source.img -O qcow2 /to/target Source and target are regular files on two different disks. The test machine is a Xeon E5 with 16 real cores. ---------------------------------------------------------------------- Non-preallocated output (times are in seconds) without -W -W -m 1 153 - -m 4 155 157 -m 8 [default] 158 231 -m 16 [max] 166 166 ---------------------------------------------------------------------- The documentation for ‘-W’ notes that this is only recommended for preallocated outputs (which the test above does not use). So let's try using a preallocated qcow2 output. Conversion command: # the same target file is reused each time time qemu-img convert -n [-W] [-m ##] -f raw source.img -O qcow2 /to/target ---------------------------------------------------------------------- Preallocated output (times are in seconds) without -W -W -m 1 147 - -m 4 146 145 -m 8 [default] 146 199 -m 16 [max] 147 146 ---------------------------------------------------------------------- Based on this there seems to be some issue with the ‘-W’ option -- I even thought I might have it backwards, but checking the code it does seem like ‘-W’ enables (rather than disables) out of order writes. Also some bizarre interaction between ‘-W’ and ‘-m 8’. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

3 8

[nbdkit PATCH v2 0/2] add nbd plugin
by Eric Blake 19 Nov '17

19 Nov '17

I'm still working on the interleaving (and Rich reminded me on IRC that we still don't have THREAD_MODEL_PARALLEL working anywhere yet, anyways). Since nbdkit doesn't really have a parallel plugin yet, my testing on that front will have to use qemu-nbd as the original server, as well as qemu-io as the driver (qemu-io's aio_read and aio_write commands can be used to trigger interleaved requests, and well-placed gdb breakpoints can make the triggering even easier to test). Do people like the name 'nbd-forward' instead of 'nbd'? The name of the plugin describes what it connects to ('nbdkit file ...' connects to a file; 'nbdkit streaming ...' connects to a streaming fd, so 'nbdkit nbd ...' connects to another nbd server); but the reuse of the term 'nbd' might be a bit confusing, where a longer name would make it more obvious on intended usage. Here's how I tested: term1$ rm -f sock; ./nbdkit -o -U sock [-r] file file=TODO term2$ ./nbdkit [-r] -n nbd socket=sock term3$ qemu-io -f raw nbd://localhost:10809/ [-r] which let me use qemu-io commands to peek (and poke) the TODO file through two layers of nbdkit processes, while upgrading from an old-style handshake to a newstyle (I also used -fv on nbdkit and --trace='nbd_*' on qemu-io to turn up the debugging messages, to actually watch the interactions). Patch 2 shows my thought process to switching things to fully parallel: patch 3/2 (still unwritten at the moment) would basically change a single transaction into a linked list, where nbd_request_full() adds another transaction to the list, nbd_read_raw() uses the cookie received from the server to look up a transaction to remove from the list, and where breaking the while(1) loop of nbd_reader() cleans up any remaining transactions still on the list. Eric Blake (2): nbd: Add new nbd forwarding plugin nbd: Split reading into separate thread configure.ac | 1 + plugins/Makefile.am | 1 + plugins/nbd/Makefile.am | 63 ++++ plugins/nbd/nbd.c | 629 ++++++++++++++++++++++++++++++++++++++ plugins/nbd/nbdkit-nbd-plugin.pod | 96 ++++++ 5 files changed, 790 insertions(+) create mode 100644 plugins/nbd/Makefile.am create mode 100644 plugins/nbd/nbd.c create mode 100644 plugins/nbd/nbdkit-nbd-plugin.pod -- 2.13.6

2 11

[PATCH] build: Reference RPC_CFLAGS where needed
by Martin Kletzander 19 Nov '17

19 Nov '17

Since glibc-2.26 removed the /usr/include/rpc/rpc.h, it caused problem even though probing for libtirpc has taken place and worked properly. The problem is that RPC_CFLAGS (usually `-I/usr/include/tirpc` on Linux) was not used in places where it was needed. So the build failed with messages like the following: In file included from guestfs_protocol.c:6:0: guestfs_protocol.h:9:10: fatal error: rpc/rpc.h: No such file or directory #include <rpc/rpc.h> ^~~~~~~~~~~ Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com> --- Notes: Using $(RPC_CFLAGS) in places where needed fixes the issue and libguestfs compiles cleanly with glibc-2.26 for me. Disclaimer: I wasn't able to build with everything enabled, so there might be some more build sources that need this, but I haven't found them. common/protocol/Makefile.am | 3 ++- fish/Makefile.am | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/common/protocol/Makefile.am b/common/protocol/Makefile.am index cfac2d8d375f..778758209d4c 100644 --- a/common/protocol/Makefile.am +++ b/common/protocol/Makefile.am @@ -34,7 +34,8 @@ noinst_LTLIBRARIES = libprotocol.la # warnings here. libprotocol_la_SOURCES = guestfs_protocol.c guestfs_protocol.h libprotocol_la_CFLAGS = \ - -Wall -Wno-unused -fno-strict-aliasing $(GCC_VISIBILITY_HIDDEN) + -Wall -Wno-unused -fno-strict-aliasing $(GCC_VISIBILITY_HIDDEN) \ + $(RPC_CFLAGS) if HAVE_RPCGEN diff --git a/fish/Makefile.am b/fish/Makefile.am index a6efbb43966c..8623e8ecae72 100644 --- a/fish/Makefile.am +++ b/fish/Makefile.am @@ -101,7 +101,7 @@ guestfish_SOURCES = \ # This convenience library is solely to compile its generated sources with # custom flags. librc_protocol_la_SOURCES = rc_protocol.c rc_protocol.h -librc_protocol_la_CFLAGS = -Wall -Wno-unused -fno-strict-aliasing +librc_protocol_la_CFLAGS = -Wall -Wno-unused -fno-strict-aliasing $(RPC_CFLAGS) # Build the command lookup perfect hash code. The generated code has # lots of warnings so we must compile it in a separate mini-library. -- 2.15.0

2 1

[RFC nbdkit PATCH 0/6] Enable full parallel request handling
by Eric Blake 19 Nov '17

19 Nov '17

I want to make my nbd forwarding plugin fully parallel - but to do that, I first need to make nbdkit itself fully parallel ;) With this series, I was finally able to demonstrate out-of-order responses when using qemu-io (which is great at sending back-to-back requests prior to waiting for responses) coupled with the nbd file plugin (which has a great feature of rdelay and wdelay, to make it obvious whether processing occurs in parallel among separate threads or in serial). 'make check' sometimes succeeds with this series, but more often than not I was able to crash or hand on the test-socket-activation test; I've posted another series for the crashes, as well as analysis of the hang. Until the hang is solved, I'm leaving this series as RFC, but review is welcome to see if my approach is sound. Eric Blake (6): connections: Simplify handle_request() connections: Avoid plugin when shutdown is imminent connections: Add read/write lock over client I/O connections: Add thread-safe status indicator connections: Set up thread pool for handling client requests Add --threads option for supporting true parallel requests TODO | 7 -- docs/nbdkit.pod | 12 ++- nbdkit.in | 2 +- plugins/file/file.c | 2 +- src/connections.c | 271 +++++++++++++++++++++++++++++++++++++--------------- src/internal.h | 2 + src/main.c | 20 +++- src/plugins.c | 8 ++ 8 files changed, 237 insertions(+), 87 deletions(-) -- 2.13.6

2 12

[nbdkit PATCH 0/4] thread-safety issues prior to parallel handling
by Eric Blake 17 Nov '17

17 Nov '17

These patches should be ready to go in now; I will also post my work-in-progress for enabling full parallel handling that depends on these, but with that series, I was still getting crashes or hangs with test-socket-activation (I think I've nailed all the crashes I've seen, but the hang is rather insidious; see my other email https://www.redhat.com/archives/libguestfs/2017-November/msg00139.html) Eric Blake (4): errors: Avoid interleaved errors from parallel threads threadlocal: Copy thread name plugins: Make plugin_name() reliable past unload sockets: Fix lifetime of thread_data src/errors.c | 30 +++++++++++++++++++++++++++++- src/plugins.c | 8 ++++++-- src/sockets.c | 27 ++++++++++++++++++--------- src/threadlocal.c | 17 +++++++++++++---- 4 files changed, 66 insertions(+), 16 deletions(-) -- 2.13.6

2 9

[PATCH 0/3] Alternate way to avoid race conditions when nbdkit exits.
by Richard W.M. Jones 17 Nov '17

17 Nov '17

This fixes the race conditions for me, using the test described here: https://www.redhat.com/archives/libguestfs/2017-September/msg00226.html Rich.

2 8

[nbdkit PATCH] maint: Rebuild nbdkit if nbdkit.in changes
by Eric Blake 16 Nov '17

16 Nov '17

Tell automake that we want ./nbdkit to be kept up-to-date in the developer's tree. Otherwise, after touching nbdkit.in, you would have to remember to manually run 'make nbdkit'. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- Makefile.am | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Makefile.am b/Makefile.am index 315aa36..7815258 100644 --- a/Makefile.am +++ b/Makefile.am @@ -39,6 +39,8 @@ EXTRA_DIST = \ valgrind-suppressions \ m4/.gitignore +noinst_SCRIPTS = nbdkit + SUBDIRS = \ docs \ include \ -- 2.13.6

2 2