Re: [Libguestfs] [PATCH 6/6] v2v: nbdkit: Match qemu-img number of parallel coroutines
On Sat, Jan 23, 2021 at 12:45:24AM +0200, Nir Soffer wrote:
qemu-img is using 8 parallel coroutines by default. I tests up to 16
parallel coroutines and it seems that 8 gives good results.
nbdkit uses 16 threads by default. Testing nbdkit with qemu-img show
that 8 threads give good results.
I think for rhv upload plugin matching the number of threads to the
number of connections would be optimal. We need to improve this later to
use the optimal number for the configured input and output plugins.
Testing rhv-upload-plugin show small improvement (~6%) in total
connection time. Compared with last version using single connection, we
are now 50% faster.
Results are not stable, we need to test this with bigger images and real
environment.
[connection 1 ops, 3.561693 s]
[dispatch 550 ops, 2.808350 s]
[write 470 ops, 2.482875 s, 316.06 MiB, 127.30 MiB/s]
[zero 78 ops, 0.178174 s, 1.26 GiB, 7.05 GiB/s]
[flush 2 ops, 0.000211 s]
[connection 1 ops, 3.561724 s]
[dispatch 543 ops, 2.836738 s]
[write 472 ops, 2.503561 s, 341.62 MiB, 136.46 MiB/s]
[zero 69 ops, 0.162465 s, 1.12 GiB, 6.89 GiB/s]
[flush 2 ops, 0.000181 s]
[connection 1 ops, 3.566931 s]
[dispatch 536 ops, 2.807226 s]
[write 462 ops, 2.508345 s, 326.12 MiB, 130.02 MiB/s]
[zero 72 ops, 0.141442 s, 1.30 GiB, 9.20 GiB/s]
[flush 2 ops, 0.000158 s]
[connection 1 ops, 3.564396 s]
[dispatch 563 ops, 2.853623 s]
[write 503 ops, 2.592482 s, 361.44 MiB, 139.42 MiB/s]
[zero 58 ops, 0.113708 s, 1.01 GiB, 8.88 GiB/s]
[flush 2 ops, 0.000149 s]
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
v2v/nbdkit.ml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/v2v/nbdkit.ml b/v2v/nbdkit.ml
index 46b20c9d..caa76342 100644
--- a/v2v/nbdkit.ml
+++ b/v2v/nbdkit.ml
@@ -137,6 +137,9 @@ let run_unix cmd =
add_arg "--pidfile"; add_arg pidfile;
add_arg "--unix"; add_arg sock;
+ (* Match qemu-img default number of parallel coroutines *)
+ add_arg "--threads"; add_arg "8";
This will affect all nbdkit instances -- virt-v2v uses nbdkit for both
the input and output sides, eg. when converting from VMware -- so it
would be better to make this configurable as a parameter to the Nbdkit
module (eg. Nbdkit.set_threads, like the way Nbdkit.set_verbose works).
Rich.
(* Reduce verbosity in nbdkit >= 1.17.4. *)
let version = version (config ()) in
if version >= (1, 17, 4) then (
--
2.26.2
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW