Re: [Libguestfs] Virt-v2v performance benchmarking part 3
On 01/10/22 16:52, Richard W.M. Jones wrote:
For the raw format local disk to local disk conversion, it's possible
to regain most of the performance by adding
--request-size=$(( 16 * 1024 * 1024 )) to the nbdcopy command. The
patch below is not suitable for going upstream but it can be used for
testing:
diff --git a/v2v/v2v.ml b/v2v/v2v.ml
index 47e6e937..ece3b7d9 100644
--- a/v2v/v2v.ml
+++ b/v2v/v2v.ml
@@ -613,6 +613,7 @@ and nbdcopy output_alloc input_uri output_uri =
let cmd = ref [] in
List.push_back_list cmd [ "nbdcopy"; input_uri; output_uri ];
List.push_back cmd "--flush";
+ List.push_back cmd "--request-size=16777216";
(*List.push_back cmd "--verbose";*)
if not (quiet ()) then List.push_back cmd "--progress";
if output_alloc = Types.Preallocated then List.push_back cmd "--allocated";
The problem is of course this is a pessimisation for other
conversions. It's known to make at least qcow2 to qcow2, and all VDDK
conversions worse. So we'd have to make it conditional on doing a raw
format local conversion, which is a pretty ugly hack. Even worse, the
exact size (16M) varies for me when I test this on different machines
and HDDs vs SSDs. On my very fast AMD machine with an SSD, the
nbdcopy default request size (256K) is fastest and larger sizes are a
very slightly slower.
I can imagine an "adaptive nbdcopy" which adjusts these parameters
while copying in order to find the best performance. A little bit
hard to implement ...
I'm also still wondering exactly why a larger request size is better
in this case. You can easily reproduce the effect using the attached
test script and adjusting --request-size. You'll need to build the
standard test guest, see part 1.
(The following thought occurred to me last evening.)
In modular v2v, we use multi-threaded nbdkit instances, and
multi-threaded nbdcopy instances. (IIUC.) I think: that should result in
quite a bit of thrashing, on both source and destination disks, no? That
should be especially visible on HDDs, but perhaps also on SSDs
(dependent on request size as you mention above).
The worst is likely when both nbdcopy processes operate on the same
physical HDD (i.e., spinning rust).
qemu-img is single-threaded, so even if reads from and writes to the
same physical hard disk, it kind of generates two "parallel" request
streams, which both the disk and the kernel's IO scheduler could cope
with more easily. According to the nbdcopy manual, the default thread
count is "number of processor cores available", the "sliding window of
requests" with a high thread count is likely undistinguishable from real
random access.
Also I (vaguely?) gather that nbdcopy bypasses the page cache (or does
it only sync automatically at the end? I don't remember). If the page
cache is avoided, then the page cache has no chance to mitigate the
thrashing, especially on HDDs -- but even on SSDs, if the drive's
internal cache is not large enough (considering the individual request
size and the number of random requests flying in parallel), the
degradation should be visible.
Can you tweak (i.e., lower) the thread count of both nbdcopy processes;
let's say to "1", for starters?
Thanks!
Laszlo