Re: [Libguestfs] More parallelism in VDDK driver (was: Re: CFME-5.11.7.3 Perf. Tests)
On Wed, Aug 05, 2020 at 03:39:15PM +0300, Nir Soffer wrote:
On Wed, Aug 5, 2020 at 3:28 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
>
>
> Nir, BTW what are you using for performance testing?
virt-v2v with local image, or imageio client with local image.
> As far as I can tell it's not possible to make qemu-img convert use
> multi-conn when connecting to the source (which is going to be a
> problem if we want to use this stuff in virt-v2v).
But do we need multiple connections? qemu can send multiple requests
on one connection.
As implemented now there is only one VDDK handle per connection and
VDDK doesn't allow multiple requests at the same time on one handle,
so with a single NBD connection everything will be serialized.
Now if we were to implement a thread pool inside nbdkit-vddk-plugin we
could get around that restriction (with a lot of complexity). But the
results I posted a moment ago show that we wouldn't get anything like
a linear speed up. It hardly seems worth it to me.
Did you try to copy an image from nbdkit file plugin to another
nbdkit
file plugin using qemu-img convert?
nbdkit file pluing -> qemu-img convert -W nbd:///?socket=src.sock
nbd:///?socket=dst.sock-> nbdkit file plugin
I did not try it but I will be surprised if we don't get all 8 threads
busy in both sides
Likely, but that's because the file plugin uses PARALLEL as thread
model, so even on a single NBD connection it can keep all threads
inside nbdkit busy.
The reason we use multiple connection in imageio is that we
don't
support async I/O in http client, http server, and nbd client, and
it is much easier to open new connection with the entire stuck
compared to rewriting the http server and nbd client.
It also much hard to provide easy to use interface for users
supporting async I/O.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW