Re: [Libguestfs] [PATCH 6/6] v2v: nbdkit: Match qemu-img number of parallel coroutines

On Sat, Jan 23, 2021 at 12:45:24AM +0200, Nir Soffer wrote: > qemu-img is using 8 parallel coroutines by default. I tests up to 16 > parallel coroutines and it seems that 8 gives good results. > > nbdkit uses 16 threads by default. Testing nbdkit with qemu-img show > that 8 threads give good results. > > I think for rhv upload plugin matching the number of threads to the > number of connections would be optimal. We need to improve this later to > use the optimal number for the configured input and output plugins. > > Testing rhv-upload-plugin show small improvement (~6%) in total > connection time. Compared with last version using single connection, we > are now 50% faster. > > Results are not stable, we need to test this with bigger images and real > environment. > > [connection 1 ops, 3.561693 s] > [dispatch 550 ops, 2.808350 s] > [write 470 ops, 2.482875 s, 316.06 MiB, 127.30 MiB/s] > [zero 78 ops, 0.178174 s, 1.26 GiB, 7.05 GiB/s] > [flush 2 ops, 0.000211 s] > > [connection 1 ops, 3.561724 s] > [dispatch 543 ops, 2.836738 s] > [write 472 ops, 2.503561 s, 341.62 MiB, 136.46 MiB/s] > [zero 69 ops, 0.162465 s, 1.12 GiB, 6.89 GiB/s] > [flush 2 ops, 0.000181 s] > > [connection 1 ops, 3.566931 s] > [dispatch 536 ops, 2.807226 s] > [write 462 ops, 2.508345 s, 326.12 MiB, 130.02 MiB/s] > [zero 72 ops, 0.141442 s, 1.30 GiB, 9.20 GiB/s] > [flush 2 ops, 0.000158 s] > > [connection 1 ops, 3.564396 s] > [dispatch 563 ops, 2.853623 s] > [write 503 ops, 2.592482 s, 361.44 MiB, 139.42 MiB/s] > [zero 58 ops, 0.113708 s, 1.01 GiB, 8.88 GiB/s] > [flush 2 ops, 0.000149 s] > > Signed-off-by: Nir Soffer <nsoffer(a)redhat.com&gt; > --- > v2v/nbdkit.ml | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/v2v/nbdkit.ml b/v2v/nbdkit.ml > index 46b20c9d..caa76342 100644 > --- a/v2v/nbdkit.ml > +++ b/v2v/nbdkit.ml > @@ -137,6 +137,9 @@ let run_unix cmd = > add_arg "--pidfile"; add_arg pidfile; > add_arg "--unix"; add_arg sock; > > + (* Match qemu-img default number of parallel coroutines *) > + add_arg "--threads"; add_arg "8"; This will affect all nbdkit instances -- virt-v2v uses nbdkit for both the input and output sides, eg. when converting from VMware -- so it would be better to make this configurable as a parameter to the Nbdkit module (eg. Nbdkit.set_threads, like the way Nbdkit.set_verbose works).

Rich. > (* Reduce verbosity in nbdkit >= 1.17.4. *) > let version = version (config ()) in > if version >= (1, 17, 4) then ( > -- > 2.26.2 -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW