Virt-v2v performance benchmarking part 3
Part 1: https://listman.redhat.com/archives/libguestfs/2022-January/msg00055.html Part 2: https://listman.redhat.com/archives/libguestfs/2022-January/msg00057.html This is part 3 of my performance analysis of virt-v2v over the last year. In this email I cover conversion from VMware to a local disk using VDDK. This is a more realistic test than doing local disk to local disk conversions. As you can see from the new chart in the attached file [LibreOffice Calc format] modular virt-v2v has got a little faster over all, with conversion taking slightly longer and copying being slightly faster. If you expand the hidden columns (between columns F & M) you will also see clearly the new flushing behaviour of nbdcopy, where it always flushes the output to disk, versus "qemu-img convert" which used the page cache (notice the Sync times in column L). This can make old virt-v2v appear to be much faster, but the appearance is not real. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Sun, Jan 09, 2022 at 06:51:42PM +0000, Richard W.M. Jones wrote:
Part 1: https://listman.redhat.com/archives/libguestfs/2022-January/msg00055.html Part 2: https://listman.redhat.com/archives/libguestfs/2022-January/msg00057.html
This is part 3 of my performance analysis of virt-v2v over the last year. In this email I cover conversion from VMware to a local disk using VDDK. This is a more realistic test than doing local disk to local disk conversions.
As you can see from the new chart in the attached file [LibreOffice Calc format] modular virt-v2v has got a little faster over all, with conversion taking slightly longer and copying being slightly faster.
If you expand the hidden columns (between columns F & M) you will also see clearly the new flushing behaviour of nbdcopy, where it always flushes the output to disk, versus "qemu-img convert" which used the page cache (notice the Sync times in column L). This can make old virt-v2v appear to be much faster, but the appearance is not real.
I meant to record the virt-v2v command I used, which was: $ virt-v2v -ic 'esx://root@example.com/?no_verify=1' \ -it vddk \ -io vddk-libdir=vddk-7.0.0/vmware-vix-disklib-distrib \ -io vddk-thumbprint=<thumbprint> \ -ip /tmp/vmware-passwd \ 'Fedora 35 standard test' -o local -os /var/tmp The version of VDDK was 7.0.0 in all tests. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On 01/09/22 19:51, Richard W.M. Jones wrote:
Part 1: https://listman.redhat.com/archives/libguestfs/2022-January/msg00055.html Part 2: https://listman.redhat.com/archives/libguestfs/2022-January/msg00057.html
This is part 3 of my performance analysis of virt-v2v over the last year. In this email I cover conversion from VMware to a local disk using VDDK. This is a more realistic test than doing local disk to local disk conversions.
As you can see from the new chart in the attached file [LibreOffice Calc format] modular virt-v2v has got a little faster over all, with conversion taking slightly longer and copying being slightly faster.
If you expand the hidden columns (between columns F & M) you will also see clearly the new flushing behaviour of nbdcopy, where it always flushes the output to disk, versus "qemu-img convert" which used the page cache (notice the Sync times in column L). This can make old virt-v2v appear to be much faster, but the appearance is not real.
So this confirms that there is no performance regression in the "VMware to a local disk using VDDK" conversion case -- this is the result we've been looking for, right? Thanks, Laszlo
On Mon, Jan 10, 2022 at 11:21:16AM +0100, Laszlo Ersek wrote:
On 01/09/22 19:51, Richard W.M. Jones wrote:
Part 1: https://listman.redhat.com/archives/libguestfs/2022-January/msg00055.html Part 2: https://listman.redhat.com/archives/libguestfs/2022-January/msg00057.html
This is part 3 of my performance analysis of virt-v2v over the last year. In this email I cover conversion from VMware to a local disk using VDDK. This is a more realistic test than doing local disk to local disk conversions.
As you can see from the new chart in the attached file [LibreOffice Calc format] modular virt-v2v has got a little faster over all, with conversion taking slightly longer and copying being slightly faster.
If you expand the hidden columns (between columns F & M) you will also see clearly the new flushing behaviour of nbdcopy, where it always flushes the output to disk, versus "qemu-img convert" which used the page cache (notice the Sync times in column L). This can make old virt-v2v appear to be much faster, but the appearance is not real.
So this confirms that there is no performance regression in the "VMware to a local disk using VDDK" conversion case -- this is the result we've been looking for, right?
Yes, but there could _appear_ to be a regression because of the substantial extra time taken to flush data to disk (the "Sync" column). I've added the Sync time to the total times, but if you were to omit that then old virt-v2v would appear to be ~ 50 seconds faster. I think flushing / not using the page cache is important for whole system throughput so I want to keep this behaviour. There is very definitely a regression for local disk to local disk raw format which I'm looking at now. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
For the raw format local disk to local disk conversion, it's possible to regain most of the performance by adding --request-size=$(( 16 * 1024 * 1024 )) to the nbdcopy command. The patch below is not suitable for going upstream but it can be used for testing: diff --git a/v2v/v2v.ml b/v2v/v2v.ml index 47e6e937..ece3b7d9 100644 --- a/v2v/v2v.ml +++ b/v2v/v2v.ml @@ -613,6 +613,7 @@ and nbdcopy output_alloc input_uri output_uri = let cmd = ref [] in List.push_back_list cmd [ "nbdcopy"; input_uri; output_uri ]; List.push_back cmd "--flush"; + List.push_back cmd "--request-size=16777216"; (*List.push_back cmd "--verbose";*) if not (quiet ()) then List.push_back cmd "--progress"; if output_alloc = Types.Preallocated then List.push_back cmd "--allocated"; The problem is of course this is a pessimisation for other conversions. It's known to make at least qcow2 to qcow2, and all VDDK conversions worse. So we'd have to make it conditional on doing a raw format local conversion, which is a pretty ugly hack. Even worse, the exact size (16M) varies for me when I test this on different machines and HDDs vs SSDs. On my very fast AMD machine with an SSD, the nbdcopy default request size (256K) is fastest and larger sizes are a very slightly slower. I can imagine an "adaptive nbdcopy" which adjusts these parameters while copying in order to find the best performance. A little bit hard to implement ... I'm also still wondering exactly why a larger request size is better in this case. You can easily reproduce the effect using the attached test script and adjusting --request-size. You'll need to build the standard test guest, see part 1. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 01/10/22 16:52, Richard W.M. Jones wrote:
For the raw format local disk to local disk conversion, it's possible to regain most of the performance by adding --request-size=$(( 16 * 1024 * 1024 )) to the nbdcopy command. The patch below is not suitable for going upstream but it can be used for testing:
diff --git a/v2v/v2v.ml b/v2v/v2v.ml index 47e6e937..ece3b7d9 100644 --- a/v2v/v2v.ml +++ b/v2v/v2v.ml @@ -613,6 +613,7 @@ and nbdcopy output_alloc input_uri output_uri = let cmd = ref [] in List.push_back_list cmd [ "nbdcopy"; input_uri; output_uri ]; List.push_back cmd "--flush"; + List.push_back cmd "--request-size=16777216"; (*List.push_back cmd "--verbose";*) if not (quiet ()) then List.push_back cmd "--progress"; if output_alloc = Types.Preallocated then List.push_back cmd "--allocated";
The problem is of course this is a pessimisation for other conversions. It's known to make at least qcow2 to qcow2, and all VDDK conversions worse. So we'd have to make it conditional on doing a raw format local conversion, which is a pretty ugly hack. Even worse, the exact size (16M) varies for me when I test this on different machines and HDDs vs SSDs. On my very fast AMD machine with an SSD, the nbdcopy default request size (256K) is fastest and larger sizes are a very slightly slower.
I can imagine an "adaptive nbdcopy" which adjusts these parameters while copying in order to find the best performance. A little bit hard to implement ...
I'm also still wondering exactly why a larger request size is better in this case. You can easily reproduce the effect using the attached test script and adjusting --request-size. You'll need to build the standard test guest, see part 1.
(The following thought occurred to me last evening.) In modular v2v, we use multi-threaded nbdkit instances, and multi-threaded nbdcopy instances. (IIUC.) I think: that should result in quite a bit of thrashing, on both source and destination disks, no? That should be especially visible on HDDs, but perhaps also on SSDs (dependent on request size as you mention above). The worst is likely when both nbdcopy processes operate on the same physical HDD (i.e., spinning rust). qemu-img is single-threaded, so even if reads from and writes to the same physical hard disk, it kind of generates two "parallel" request streams, which both the disk and the kernel's IO scheduler could cope with more easily. According to the nbdcopy manual, the default thread count is "number of processor cores available", the "sliding window of requests" with a high thread count is likely undistinguishable from real random access. Also I (vaguely?) gather that nbdcopy bypasses the page cache (or does it only sync automatically at the end? I don't remember). If the page cache is avoided, then the page cache has no chance to mitigate the thrashing, especially on HDDs -- but even on SSDs, if the drive's internal cache is not large enough (considering the individual request size and the number of random requests flying in parallel), the degradation should be visible. Can you tweak (i.e., lower) the thread count of both nbdcopy processes; let's say to "1", for starters? Thanks! Laszlo
On 01/11/22 08:00, Laszlo Ersek wrote:
On 01/10/22 16:52, Richard W.M. Jones wrote:
For the raw format local disk to local disk conversion, it's possible to regain most of the performance by adding --request-size=$(( 16 * 1024 * 1024 )) to the nbdcopy command. The patch below is not suitable for going upstream but it can be used for testing:
diff --git a/v2v/v2v.ml b/v2v/v2v.ml index 47e6e937..ece3b7d9 100644 --- a/v2v/v2v.ml +++ b/v2v/v2v.ml @@ -613,6 +613,7 @@ and nbdcopy output_alloc input_uri output_uri = let cmd = ref [] in List.push_back_list cmd [ "nbdcopy"; input_uri; output_uri ]; List.push_back cmd "--flush"; + List.push_back cmd "--request-size=16777216"; (*List.push_back cmd "--verbose";*) if not (quiet ()) then List.push_back cmd "--progress"; if output_alloc = Types.Preallocated then List.push_back cmd "--allocated";
The problem is of course this is a pessimisation for other conversions. It's known to make at least qcow2 to qcow2, and all VDDK conversions worse. So we'd have to make it conditional on doing a raw format local conversion, which is a pretty ugly hack. Even worse, the exact size (16M) varies for me when I test this on different machines and HDDs vs SSDs. On my very fast AMD machine with an SSD, the nbdcopy default request size (256K) is fastest and larger sizes are a very slightly slower.
I can imagine an "adaptive nbdcopy" which adjusts these parameters while copying in order to find the best performance. A little bit hard to implement ...
I'm also still wondering exactly why a larger request size is better in this case. You can easily reproduce the effect using the attached test script and adjusting --request-size. You'll need to build the standard test guest, see part 1.
(The following thought occurred to me last evening.)
In modular v2v, we use multi-threaded nbdkit instances, and multi-threaded nbdcopy instances. (IIUC.) I think: that should result in quite a bit of thrashing, on both source and destination disks, no? That should be especially visible on HDDs, but perhaps also on SSDs (dependent on request size as you mention above).
The worst is likely when both nbdcopy processes operate on the same physical HDD (i.e., spinning rust).
qemu-img is single-threaded, so even if reads from and writes to the same physical hard disk, it kind of generates two "parallel" request streams, which both the disk and the kernel's IO scheduler could cope with more easily. According to the nbdcopy manual, the default thread count is "number of processor cores available", the "sliding window of requests" with a high thread count is likely undistinguishable from real random access.
Also I (vaguely?) gather that nbdcopy bypasses the page cache (or does it only sync automatically at the end? I don't remember). If the page cache is avoided, then the page cache has no chance to mitigate the thrashing, especially on HDDs -- but even on SSDs, if the drive's internal cache is not large enough (considering the individual request size and the number of random requests flying in parallel), the degradation should be visible.
Can you tweak (i.e., lower) the thread count of both nbdcopy processes; let's say to "1", for starters?
I meant to add: the "--request-size=16777216" option, *where it helps*, effectively serializes the request stream. A single request is not broken up into smaller *parallel* requests, thus, if you have huge (like 16MiB) requests, then that effectively de-randomizes the accesses in the sliding window. Thanks Laszlo
On 01/11/22 08:00, Laszlo Ersek wrote:
On 01/10/22 16:52, Richard W.M. Jones wrote:
For the raw format local disk to local disk conversion, it's possible to regain most of the performance by adding --request-size=$(( 16 * 1024 * 1024 )) to the nbdcopy command. The patch below is not suitable for going upstream but it can be used for testing:
diff --git a/v2v/v2v.ml b/v2v/v2v.ml index 47e6e937..ece3b7d9 100644 --- a/v2v/v2v.ml +++ b/v2v/v2v.ml @@ -613,6 +613,7 @@ and nbdcopy output_alloc input_uri output_uri = let cmd = ref [] in List.push_back_list cmd [ "nbdcopy"; input_uri; output_uri ]; List.push_back cmd "--flush"; + List.push_back cmd "--request-size=16777216"; (*List.push_back cmd "--verbose";*) if not (quiet ()) then List.push_back cmd "--progress"; if output_alloc = Types.Preallocated then List.push_back cmd "--allocated";
The problem is of course this is a pessimisation for other conversions. It's known to make at least qcow2 to qcow2, and all VDDK conversions worse. So we'd have to make it conditional on doing a raw format local conversion, which is a pretty ugly hack. Even worse, the exact size (16M) varies for me when I test this on different machines and HDDs vs SSDs. On my very fast AMD machine with an SSD, the nbdcopy default request size (256K) is fastest and larger sizes are a very slightly slower.
I can imagine an "adaptive nbdcopy" which adjusts these parameters while copying in order to find the best performance. A little bit hard to implement ...
I'm also still wondering exactly why a larger request size is better in this case. You can easily reproduce the effect using the attached test script and adjusting --request-size. You'll need to build the standard test guest, see part 1.
(The following thought occurred to me last evening.)
In modular v2v, we use multi-threaded nbdkit instances, and multi-threaded nbdcopy instances. (IIUC.) I think: that should result in quite a bit of thrashing, on both source and destination disks, no? That should be especially visible on HDDs, but perhaps also on SSDs (dependent on request size as you mention above).
The worst is likely when both nbdcopy processes operate on the same physical HDD (i.e., spinning rust).
qemu-img is single-threaded,
hmmmm, not necessarily; according to the manual, "qemu-img convert" uses (by default) 8 co-routines. There's also the -W flag ("out of order writes"), which I don't know if the original virt-v2v used. Laszlo
so even if reads from and writes to the same physical hard disk, it kind of generates two "parallel" request streams, which both the disk and the kernel's IO scheduler could cope with more easily. According to the nbdcopy manual, the default thread count is "number of processor cores available", the "sliding window of requests" with a high thread count is likely undistinguishable from real random access.
Also I (vaguely?) gather that nbdcopy bypasses the page cache (or does it only sync automatically at the end? I don't remember). If the page cache is avoided, then the page cache has no chance to mitigate the thrashing, especially on HDDs -- but even on SSDs, if the drive's internal cache is not large enough (considering the individual request size and the number of random requests flying in parallel), the degradation should be visible.
Can you tweak (i.e., lower) the thread count of both nbdcopy processes; let's say to "1", for starters?
Thanks! Laszlo
On Tue, Jan 11, 2022 at 08:07:39AM +0100, Laszlo Ersek wrote:
hmmmm, not necessarily; according to the manual, "qemu-img convert" uses (by default) 8 co-routines. There's also the -W flag ("out of order writes"), which I don't know if the original virt-v2v used.
I'm never sure how qemu coroutines map to threads. I assume it's not 1-1, and it's somehow connected to the iothread setting? The -W flag was only used for -o rhv-upload and not for any other input or output method. See write_out_of_order in the 1.44 code. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 01/11/22 11:56, Richard W.M. Jones wrote:
On Tue, Jan 11, 2022 at 08:07:39AM +0100, Laszlo Ersek wrote:
hmmmm, not necessarily; according to the manual, "qemu-img convert" uses (by default) 8 co-routines. There's also the -W flag ("out of order writes"), which I don't know if the original virt-v2v used.
I'm never sure how qemu coroutines map to threads. I assume it's not 1-1, and it's somehow connected to the iothread setting?
(I don't know the answer, but) this difference should not matter here I believe; the QEMU co-routines submit async IO one way or another (IIRC), so once those are in the kernel's hands, it shouldn't matter (from an IO performance perspective) whether the userspace process waits on their completions with heavy-weight threads, or co-routines or whatever. What matters is that the kernel has multiple IO reqs in flight at the same time. Thanks, Laszlo
The -W flag was only used for -o rhv-upload and not for any other input or output method. See write_out_of_order in the 1.44 code.
Rich.
On Wed, Jan 12, 2022 at 1:12 PM Laszlo Ersek <lersek@redhat.com> wrote:
On 01/11/22 11:56, Richard W.M. Jones wrote:
On Tue, Jan 11, 2022 at 08:07:39AM +0100, Laszlo Ersek wrote:
hmmmm, not necessarily; according to the manual, "qemu-img convert" uses (by default) 8 co-routines. There's also the -W flag ("out of order writes"), which I don't know if the original virt-v2v used.
I'm never sure how qemu coroutines map to threads. I assume it's not 1-1, and it's somehow connected to the iothread setting?
(I don't know the answer, but) this difference should not matter here I believe; the QEMU co-routines submit async IO one way or another (IIRC), so once those are in the kernel's hands, it shouldn't matter (from an IO performance perspective) whether the userspace process waits on their completions with heavy-weight threads, or co-routines or whatever. What matters is that the kernel has multiple IO reqs in flight at the same time.
In practice this matters a lot. Using -W can be 6x times faster with qemu-img convert. See how I tested this here: https://bugzilla.redhat.com/1511891#c57 Nir
On 01/12/22 15:04, Nir Soffer wrote:
On Wed, Jan 12, 2022 at 1:12 PM Laszlo Ersek <lersek@redhat.com> wrote:
On 01/11/22 11:56, Richard W.M. Jones wrote:
On Tue, Jan 11, 2022 at 08:07:39AM +0100, Laszlo Ersek wrote:
hmmmm, not necessarily; according to the manual, "qemu-img convert" uses (by default) 8 co-routines. There's also the -W flag ("out of order writes"), which I don't know if the original virt-v2v used.
I'm never sure how qemu coroutines map to threads. I assume it's not 1-1, and it's somehow connected to the iothread setting?
(I don't know the answer, but) this difference should not matter here I believe; the QEMU co-routines submit async IO one way or another (IIRC), so once those are in the kernel's hands, it shouldn't matter (from an IO performance perspective) whether the userspace process waits on their completions with heavy-weight threads, or co-routines or whatever. What matters is that the kernel has multiple IO reqs in flight at the same time.
In practice this matters a lot. Using -W can be 6x times faster with qemu-img convert.
See how I tested this here: https://bugzilla.redhat.com/1511891#c57
Sorry, I didn't mean the difference between "-W" and "no -W". I meant the difference between "NPTL threads submitting async IO requests" and "coroutines submitting async IO requests". Thanks Laszlo
On Tue, Jan 11, 2022 at 1:00 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Jan 11, 2022 at 08:07:39AM +0100, Laszlo Ersek wrote:
hmmmm, not necessarily; according to the manual, "qemu-img convert" uses (by default) 8 co-routines. There's also the -W flag ("out of order writes"), which I don't know if the original virt-v2v used.
I'm never sure how qemu coroutines map to threads. I assume it's not 1-1, and it's somehow connected to the iothread setting?
In qemu-img and qemu-nbd, when using cache=none and aio=native there is one thread submitting io via libaio. When using aio=threads we can see multiple threads doing the I/O. I don't know how qemu maps request to threads, but it seems that we have one thread per inflight request, but never more than 16 (qmu-nbd never handle more than 16 inflight requests per client). I tested with "nbdcopy --connections 1 --requests ..." and we see the same number of threads in qemu-nbd plus one idle thread when requests < 16. Nir
On Tue, Jan 11, 2022 at 08:00:57AM +0100, Laszlo Ersek wrote:
(The following thought occurred to me last evening.)
In modular v2v, we use multi-threaded nbdkit instances, and multi-threaded nbdcopy instances. (IIUC.) I think: that should result in quite a bit of thrashing, on both source and destination disks, no? That should be especially visible on HDDs, but perhaps also on SSDs (dependent on request size as you mention above).
This is very possible. Also the HDD machine has only 2 cores / 4 threads (the SSD machine has 12 cores / 24 threads) so we are heavily overcommitting software threads to hardware threads. In contrast, in the VDDK to local disk case (which if you recall performs fine) nbdcopy won't be using multiple threads. This is because multi-conn is not enabled on the input (VDDK) side [see: https://listman.redhat.com/archives/libguestfs/2021-December/msg00172.html] so nbdcopy will use a single thread and a single NBD connection to both input and output sides. Within nbdkit-vddk-plugin we're using some threads to handle parallel requests on the same NBD connection (using VDDK's Async method). Within nbdkit-file-plugin on the output side it will also use multiple threads to handle parallel requests, again on a single NBD connection. But the copy is generally sequential.
The worst is likely when both nbdcopy processes operate on the same physical HDD (i.e., spinning rust).
qemu-img is single-threaded, so even if reads from and writes to the same physical hard disk, it kind of generates two "parallel" request streams, which both the disk and the kernel's IO scheduler could cope with more easily. According to the nbdcopy manual, the default thread count is "number of processor cores available", the "sliding window of requests" with a high thread count is likely undistinguishable from real random access.
Also I (vaguely?) gather that nbdcopy bypasses the page cache (or does it only sync automatically at the end? I don't remember).
Yes, both nbdcopy and nbdkit-file-cache + cache=none (which we are using) will attempt to minimize use of the page cache.
If the page cache is avoided, then the page cache has no chance to mitigate the thrashing, especially on HDDs -- but even on SSDs, if the drive's internal cache is not large enough (considering the individual request size and the number of random requests flying in parallel), the degradation should be visible.
I don't understand the mechanism by which this could happen. We're reading and writing blocks which are much larger than the filesystem block size (fs block size is probably 1K or 4K, minimum nbdcopy block size in any test is 256K). And we read and write each block exactly once, and we never revisit that data after reading/writing. Can the page cache help?
Can you tweak (i.e., lower) the thread count of both nbdcopy processes; let's say to "1", for starters?
Using nbdcopy -T 1 and leaving --request-size at the default does actually improve performance quite a bit, although not as much as increasing the request size. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On 01/11/22 11:53, Richard W.M. Jones wrote:
On Tue, Jan 11, 2022 at 08:00:57AM +0100, Laszlo Ersek wrote:
(The following thought occurred to me last evening.)
In modular v2v, we use multi-threaded nbdkit instances, and multi-threaded nbdcopy instances. (IIUC.) I think: that should result in quite a bit of thrashing, on both source and destination disks, no? That should be especially visible on HDDs, but perhaps also on SSDs (dependent on request size as you mention above).
This is very possible. Also the HDD machine has only 2 cores / 4 threads (the SSD machine has 12 cores / 24 threads) so we are heavily overcommitting software threads to hardware threads.
In contrast, in the VDDK to local disk case (which if you recall performs fine) nbdcopy won't be using multiple threads. This is because multi-conn is not enabled on the input (VDDK) side [see: https://listman.redhat.com/archives/libguestfs/2021-December/msg00172.html] so nbdcopy will use a single thread and a single NBD connection to both input and output sides. Within nbdkit-vddk-plugin we're using some threads to handle parallel requests on the same NBD connection (using VDDK's Async method). Within nbdkit-file-plugin on the output side it will also use multiple threads to handle parallel requests, again on a single NBD connection. But the copy is generally sequential.
The worst is likely when both nbdcopy processes operate on the same physical HDD (i.e., spinning rust).
qemu-img is single-threaded, so even if reads from and writes to the same physical hard disk, it kind of generates two "parallel" request streams, which both the disk and the kernel's IO scheduler could cope with more easily. According to the nbdcopy manual, the default thread count is "number of processor cores available", the "sliding window of requests" with a high thread count is likely undistinguishable from real random access.
Also I (vaguely?) gather that nbdcopy bypasses the page cache (or does it only sync automatically at the end? I don't remember).
Yes, both nbdcopy and nbdkit-file-cache + cache=none (which we are using) will attempt to minimize use of the page cache.
If the page cache is avoided, then the page cache has no chance to mitigate the thrashing, especially on HDDs -- but even on SSDs, if the drive's internal cache is not large enough (considering the individual request size and the number of random requests flying in parallel), the degradation should be visible.
I don't understand the mechanism by which this could happen. We're reading and writing blocks which are much larger than the filesystem block size (fs block size is probably 1K or 4K, minimum nbdcopy block size in any test is 256K). And we read and write each block exactly once, and we never revisit that data after reading/writing. Can the page cache help?
I think it could: even if the (internally contiguous) requests from nbdcopy have 256K sizes, their boundaries (falling on filesystem block boundaries) are still somewhat random with regard to each other, and the page cache (naively: collecting a number of requests before starting to write them out) could help linearize these (collaborating with the IO scheduler the kernel uses for the disk, I guess?) Thanks, Laszlo
Can you tweak (i.e., lower) the thread count of both nbdcopy processes; let's say to "1", for starters?
Using nbdcopy -T 1 and leaving --request-size at the default does actually improve performance quite a bit, although not as much as increasing the request size.
Rich.
participants (3)
-
Laszlo Ersek -
Nir Soffer -
Richard W.M. Jones