Re: [Libguestfs] More parallelism in VDDK driver
On 8/5/20 7:47 AM, Richard W.M. Jones wrote:
Here are some results anyway. The command I'm using is:
$ ./nbdkit -r -U - vddk \
libdir=/path/to/vmware-vix-disklib-distrib \
user=root password='***' \
server='***' thumbprint=aa:bb:cc:... \
vm=moref=3 \
file='[datastore1] Fedora 28/Fedora 28.vmdk' \
--run 'time /var/tmp/threaded-reads $unixsocket'
Source for threaded-reads is attached.
Tests (1) and (2) are about the same within noise.
Test (3) is making 8 times as many requests as test (1), so I think
it's fair to compare the 8 x time taken by test (1) (ie. the time it
would have taking to make 80,000 requests):
Test (1) * 8 = 11m28
Test (3) = 7m11
So if we had a client which could actually use multi-conn then this
would be a reasonable win. It seems like there's still a lot of
locking going on somewhere, perhaps inside VDDK or in the server.
It's certainly nowhere near a linear speedup.
If I'm reading
https://code.vmware.com/docs/11750/virtual-disk-development-kit-programmi...
correctly, we cannot reuse a single VDDK handle for two parallel
requests, but we CAN have two VDDK handles open, with requests open on
both handles at the same time. That's what SERIALIZE_REQUESTS buys us -
a client that opens multiple connections (taking advantage of
multi-conn) now has two NBD handles and therefore two VDDK handles, and
the reads separated across those two handles can proceed in parallel.
But I also don't see anything that prevents a single NBD connection from
opening multiple VDDK handles under the hood, or even from having all of
those handles opened as coordinated through a single helper thread.
That is, if nbdkit were to provide a way for a plugin to know the
maximum number of threads that will be used in parallel, then vddk's
.after_fork could spawn a dedicated thread for running VDDK handle
open/close requests (protected by a mutex), the .open callback can then
obtain the mutex, call into the helper thread to open N handles, the
thread model is advertised as PARALLEL, and in all other calls (.pread,
.pwrite, ...) we pick any one of the N handles for that NBD connection
that is not currently in use. The client application would not even
have to take advantage of multi-conn, but gets the full benefit of
out-of-order thread access for a parallel speedup.
The patch does at least seem stable. I'll post it in a minute.
Whether we do all VixDiskLib_Open from a single dedicated helper thread
created during .after_fork, or rely on pthread mutex locking so that at
most one .open is calling Open or Close, is different from whether we
open multiple VDDK handles per single NBD connection in PARALLEL mode,
vs. one VDDK handle per NBD connection in SERIALIZE_REQUESTS (offloading
the parallelism to the multi-conn client). We could probably do both,
but while opening multiple VDDK handles per NBD connection appears to be
fully compliant with VDDK docs (patch not written yet), the former
(using a mutex to serialize open calls, but open is not happening in the
same thread all the time) is indeed risky. And having the speedup
available to all clients, not just multi-conn aware clients, seems like
it might be nice to have, even if the speedup is not linear.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org