Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
by Max Reitz
On 2017-11-15 18:44, Gandalf Corvotempesta wrote:
> XVA is a tar archive.
> I don't think would be possible to directly use the image stored inside
> without extracting and merging each chunks
>
> Any random reads would be impossible to do, only a huge sequential dump to
> build the raw image
Well, you can't just add support to qemu-img alone either. Every image
format supported by qemu-img is one supported by qemu as a whole and
thus needs a proper block driver that needs to support random accesses
as well.
Max
6 years, 6 months
Effect of qemu-img convert -m and -W options
by Richard W.M. Jones
[CC to qemu-devel since I'm obviously doing something wrong here,
I'm just not sure what.]
I was getting ready to add multiple threads to ‘qemu-img convert’ (the
longest part of v2v conversions) when I noticed that it had them
already! (To be fair this was only added in February this year so no
wonder we didn't notice.)
To enable parallel convert we would need to use the ‘qemu-img convert
-W’ option (which allows out-of-order writes to the destination) and
‘qemu-img convert -m <#num-coroutines>’ option to select the
parallelism (defaults to 8). The documentation refers to coroutines
but I verified from strace that it is using real threads.
I did some testing to see what effect this has. For this I used a
large guest image which is approximately a third full of random data
(the rest being sparsely allocated):
Source format: raw
Source virtual size: 100 GB
Source used space: 31 GB
Target format: qcow2
Version: qemu-img-2.10.0-7.fc28.x86_64
Conversion command:
rm -f /to/target
time qemu-img convert [-W] [-m ##] -f raw source.img -O qcow2 /to/target
Source and target are regular files on two different disks. The test
machine is a Xeon E5 with 16 real cores.
----------------------------------------------------------------------
Non-preallocated output
(times are in seconds)
without -W -W
-m 1 153 -
-m 4 155 157
-m 8 [default] 158 231
-m 16 [max] 166 166
----------------------------------------------------------------------
The documentation for ‘-W’ notes that this is only recommended for
preallocated outputs (which the test above does not use). So let's
try using a preallocated qcow2 output.
Conversion command:
# the same target file is reused each time
time qemu-img convert -n [-W] [-m ##] -f raw source.img -O qcow2 /to/target
----------------------------------------------------------------------
Preallocated output
(times are in seconds)
without -W -W
-m 1 147 -
-m 4 146 145
-m 8 [default] 146 199
-m 16 [max] 147 146
----------------------------------------------------------------------
Based on this there seems to be some issue with the ‘-W’ option -- I
even thought I might have it backwards, but checking the code it does
seem like ‘-W’ enables (rather than disables) out of order writes.
Also some bizarre interaction between ‘-W’ and ‘-m 8’.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
6 years, 6 months
[nbdkit PATCH v2 0/2] add nbd plugin
by Eric Blake
I'm still working on the interleaving (and Rich reminded me on IRC
that we still don't have THREAD_MODEL_PARALLEL working anywhere
yet, anyways). Since nbdkit doesn't really have a parallel plugin
yet, my testing on that front will have to use qemu-nbd as the
original server, as well as qemu-io as the driver (qemu-io's
aio_read and aio_write commands can be used to trigger interleaved
requests, and well-placed gdb breakpoints can make the triggering
even easier to test).
Do people like the name 'nbd-forward' instead of 'nbd'? The name
of the plugin describes what it connects to ('nbdkit file ...'
connects to a file; 'nbdkit streaming ...' connects to a streaming
fd, so 'nbdkit nbd ...' connects to another nbd server); but the
reuse of the term 'nbd' might be a bit confusing, where a longer
name would make it more obvious on intended usage.
Here's how I tested:
term1$ rm -f sock; ./nbdkit -o -U sock [-r] file file=TODO
term2$ ./nbdkit [-r] -n nbd socket=sock
term3$ qemu-io -f raw nbd://localhost:10809/ [-r]
which let me use qemu-io commands to peek (and poke) the TODO
file through two layers of nbdkit processes, while upgrading
from an old-style handshake to a newstyle (I also used -fv on
nbdkit and --trace='nbd_*' on qemu-io to turn up the debugging
messages, to actually watch the interactions).
Patch 2 shows my thought process to switching things to fully
parallel: patch 3/2 (still unwritten at the moment) would basically
change a single transaction into a linked list, where
nbd_request_full() adds another transaction to the list,
nbd_read_raw() uses the cookie received from the server to look
up a transaction to remove from the list, and where breaking the
while(1) loop of nbd_reader() cleans up any remaining transactions
still on the list.
Eric Blake (2):
nbd: Add new nbd forwarding plugin
nbd: Split reading into separate thread
configure.ac | 1 +
plugins/Makefile.am | 1 +
plugins/nbd/Makefile.am | 63 ++++
plugins/nbd/nbd.c | 629 ++++++++++++++++++++++++++++++++++++++
plugins/nbd/nbdkit-nbd-plugin.pod | 96 ++++++
5 files changed, 790 insertions(+)
create mode 100644 plugins/nbd/Makefile.am
create mode 100644 plugins/nbd/nbd.c
create mode 100644 plugins/nbd/nbdkit-nbd-plugin.pod
--
2.13.6
6 years, 6 months
[PATCH] build: Reference RPC_CFLAGS where needed
by Martin Kletzander
Since glibc-2.26 removed the /usr/include/rpc/rpc.h, it caused problem even
though probing for libtirpc has taken place and worked properly. The problem is
that RPC_CFLAGS (usually `-I/usr/include/tirpc` on Linux) was not used in places
where it was needed. So the build failed with messages like the following:
In file included from guestfs_protocol.c:6:0:
guestfs_protocol.h:9:10: fatal error: rpc/rpc.h: No such file or directory
#include <rpc/rpc.h>
^~~~~~~~~~~
Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com>
---
Notes:
Using $(RPC_CFLAGS) in places where needed fixes the issue and libguestfs
compiles cleanly with glibc-2.26 for me. Disclaimer: I wasn't able to build
with everything enabled, so there might be some more build sources that need
this, but I haven't found them.
common/protocol/Makefile.am | 3 ++-
fish/Makefile.am | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/common/protocol/Makefile.am b/common/protocol/Makefile.am
index cfac2d8d375f..778758209d4c 100644
--- a/common/protocol/Makefile.am
+++ b/common/protocol/Makefile.am
@@ -34,7 +34,8 @@ noinst_LTLIBRARIES = libprotocol.la
# warnings here.
libprotocol_la_SOURCES = guestfs_protocol.c guestfs_protocol.h
libprotocol_la_CFLAGS = \
- -Wall -Wno-unused -fno-strict-aliasing $(GCC_VISIBILITY_HIDDEN)
+ -Wall -Wno-unused -fno-strict-aliasing $(GCC_VISIBILITY_HIDDEN) \
+ $(RPC_CFLAGS)
if HAVE_RPCGEN
diff --git a/fish/Makefile.am b/fish/Makefile.am
index a6efbb43966c..8623e8ecae72 100644
--- a/fish/Makefile.am
+++ b/fish/Makefile.am
@@ -101,7 +101,7 @@ guestfish_SOURCES = \
# This convenience library is solely to compile its generated sources with
# custom flags.
librc_protocol_la_SOURCES = rc_protocol.c rc_protocol.h
-librc_protocol_la_CFLAGS = -Wall -Wno-unused -fno-strict-aliasing
+librc_protocol_la_CFLAGS = -Wall -Wno-unused -fno-strict-aliasing $(RPC_CFLAGS)
# Build the command lookup perfect hash code. The generated code has
# lots of warnings so we must compile it in a separate mini-library.
--
2.15.0
6 years, 6 months
[RFC nbdkit PATCH 0/6] Enable full parallel request handling
by Eric Blake
I want to make my nbd forwarding plugin fully parallel - but to do
that, I first need to make nbdkit itself fully parallel ;)
With this series, I was finally able to demonstrate out-of-order
responses when using qemu-io (which is great at sending back-to-back
requests prior to waiting for responses) coupled with the nbd file
plugin (which has a great feature of rdelay and wdelay, to make
it obvious whether processing occurs in parallel among separate
threads or in serial). 'make check' sometimes succeeds with this
series, but more often than not I was able to crash or hand on the
test-socket-activation test; I've posted another series for the
crashes, as well as analysis of the hang. Until the hang is solved,
I'm leaving this series as RFC, but review is welcome to see if
my approach is sound.
Eric Blake (6):
connections: Simplify handle_request()
connections: Avoid plugin when shutdown is imminent
connections: Add read/write lock over client I/O
connections: Add thread-safe status indicator
connections: Set up thread pool for handling client requests
Add --threads option for supporting true parallel requests
TODO | 7 --
docs/nbdkit.pod | 12 ++-
nbdkit.in | 2 +-
plugins/file/file.c | 2 +-
src/connections.c | 271 +++++++++++++++++++++++++++++++++++++---------------
src/internal.h | 2 +
src/main.c | 20 +++-
src/plugins.c | 8 ++
8 files changed, 237 insertions(+), 87 deletions(-)
--
2.13.6
6 years, 6 months
[nbdkit PATCH 0/4] thread-safety issues prior to parallel handling
by Eric Blake
These patches should be ready to go in now; I will also post my
work-in-progress for enabling full parallel handling that depends
on these, but with that series, I was still getting crashes or
hangs with test-socket-activation (I think I've nailed all the
crashes I've seen, but the hang is rather insidious; see my other
email https://www.redhat.com/archives/libguestfs/2017-November/msg00139.html)
Eric Blake (4):
errors: Avoid interleaved errors from parallel threads
threadlocal: Copy thread name
plugins: Make plugin_name() reliable past unload
sockets: Fix lifetime of thread_data
src/errors.c | 30 +++++++++++++++++++++++++++++-
src/plugins.c | 8 ++++++--
src/sockets.c | 27 ++++++++++++++++++---------
src/threadlocal.c | 17 +++++++++++++----
4 files changed, 66 insertions(+), 16 deletions(-)
--
2.13.6
6 years, 6 months
[nbdkit PATCH] maint: Rebuild nbdkit if nbdkit.in changes
by Eric Blake
Tell automake that we want ./nbdkit to be kept up-to-date in the
developer's tree. Otherwise, after touching nbdkit.in, you would
have to remember to manually run 'make nbdkit'.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
Makefile.am | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Makefile.am b/Makefile.am
index 315aa36..7815258 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -39,6 +39,8 @@ EXTRA_DIST = \
valgrind-suppressions \
m4/.gitignore
+noinst_SCRIPTS = nbdkit
+
SUBDIRS = \
docs \
include \
--
2.13.6
6 years, 6 months
