More parallelism in VDDK driver (was: Re: CFME-5.11.7.3 Perf. Tests)
[NB: Adding PUBLIC mailing list because this is upstream discussion] On Mon, Aug 03, 2020 at 06:27:04PM +0100, Richard W.M. Jones wrote:
On Mon, Aug 03, 2020 at 06:03:23PM +0300, Nir Soffer wrote:
On Mon, Aug 3, 2020 at 5:47 PM Richard W.M. Jones <rjones@redhat.com> wrote: All this make sense, but when we upload 10 disks we have 10 connections but still we cannot push data fast enough. Avoiding copies will help, but I don't expect huge difference.
My guess is the issue is on the other side - pulling data from vmware.
I can believe this too. VDDK is really slow, and especially the way we use it is probably not optimal either -- but it has a confusing threading model and I don't know if we can safely use a more parallel thread model:
https://github.com/libguestfs/nbdkit/blob/89a36b1fab8302ddc370695d386a28a03a...
I may have a play around with this tomorrow.
The threading model allowed by VDDK is restrictive. The rules are here: https://code.vmware.com/docs/11750/virtual-disk-development-kit-programming-... I did a bit of testing, and it's possible to do better than what we are doing at the moment. Not sure at present if this will be easy or will add a lot of complexity. Read on ... I found through experimentation that it is possible to open multiple VDDK handles pointing to the same disk. This would allow us to use SERIALIZE_REQUESTS (instead of SERIALIZE_ALL_REQUESTS) and have overlapping calls through different handles all pointing back to the same server/disk. We should have to change all open/close calls to make the request through a single background thread - see document above for why. Adding a background thread and all the RPC needed to marshall these calls is the part which would add the complexity. However I suspect we might be able to get away with just adding a mutex around open/close. The open/close requests would happen on different threads but would not overlap. This is contrary to the rules above, but it could be sufficient. This is what I'm testing at the moment. It is definitely *not* possible to move to PARALLEL since nbdkit would make requests in parallel on the same VDDK handle, which is not allowed. (I did try this to see if the document above was serious, and it crashed in all kinds of strange ways, so I guess yes they are serious.) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Wed, Aug 5, 2020 at 2:15 PM Richard W.M. Jones <rjones@redhat.com> wrote:
[NB: Adding PUBLIC mailing list because this is upstream discussion]
On Mon, Aug 03, 2020 at 06:27:04PM +0100, Richard W.M. Jones wrote:
On Mon, Aug 03, 2020 at 06:03:23PM +0300, Nir Soffer wrote:
On Mon, Aug 3, 2020 at 5:47 PM Richard W.M. Jones <rjones@redhat.com> wrote: All this make sense, but when we upload 10 disks we have 10 connections but still we cannot push data fast enough. Avoiding copies will help, but I don't expect huge difference.
My guess is the issue is on the other side - pulling data from vmware.
I can believe this too. VDDK is really slow, and especially the way we use it is probably not optimal either -- but it has a confusing threading model and I don't know if we can safely use a more parallel thread model:
https://github.com/libguestfs/nbdkit/blob/89a36b1fab8302ddc370695d386a28a03a...
I may have a play around with this tomorrow.
The threading model allowed by VDDK is restrictive. The rules are here:
https://code.vmware.com/docs/11750/virtual-disk-development-kit-programming-...
I did a bit of testing, and it's possible to do better than what we are doing at the moment. Not sure at present if this will be easy or will add a lot of complexity. Read on ...
I found through experimentation that it is possible to open multiple VDDK handles pointing to the same disk. This would allow us to use SERIALIZE_REQUESTS (instead of SERIALIZE_ALL_REQUESTS) and have overlapping calls through different handles all pointing back to the same server/disk. We should have to change all open/close calls to make the request through a single background thread - see document above for why.
Adding a background thread and all the RPC needed to marshall these calls is the part which would add the complexity.
However I suspect we might be able to get away with just adding a mutex around open/close. The open/close requests would happen on different threads but would not overlap. This is contrary to the rules above, but it could be sufficient. This is what I'm testing at the moment.
It is definitely *not* possible to move to PARALLEL since nbdkit would make requests in parallel on the same VDDK handle, which is not allowed. (I did try this to see if the document above was serious, and it crashed in all kinds of strange ways, so I guess yes they are serious.)
Can we use something like the file plugin? thread pool of workers, each keeping open vddk handle, and serving requests in parallel from the same nbd socket? This is kind of ugly but simple, and it works great for the file plugin - we get better performance than qemu-nbd. But since we get low throughput even when we have 10 concurrent handles for 10 different disks, I'm sure this will help, and the issue may be deeper in vmware. Maybe they intentionally throttle the clients?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote:
Can we use something like the file plugin? thread pool of workers, each keeping open vddk handle, and serving requests in parallel from the same nbd socket?
Yes, but this isn't implemented in the plugins, it's implemented in the server. The server always uses a thread pool, but plugins can opt for more or less concurrency by adjusting the thread model: http://libguestfs.org/nbdkit-plugin.3.html#Threads The file plugin uses PARALLEL: $ nbdkit file --dump-plugin | grep thread max_thread_model=parallel thread_model=parallel The VDDK plugin currently uses SERIALIZE_ALL_REQUESTS: $ nbdkit vddk --dump-plugin | grep thread max_thread_model=serialize_all_requests thread_model=serialize_all_requests The proposal is to use SERIALIZE_REQUESTS, with an extra mutex added by the plugin around VixDiskLib_Open and _Close calls. PARALLEL is not possible.
This is kind of ugly but simple, and it works great for the file plugin - we get better performance than qemu-nbd.
But since we get low throughput even when we have 10 concurrent handles for 10 different disks, I'm sure this will help, and the issue may be deeper in vmware. Maybe they intentionally throttle the clients?
The whole server side seems very heavyweight, judging by how long it takes to answer single requests. It might just be poor implementation rather than throttling though. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote:
Can we use something like the file plugin? thread pool of workers, each keeping open vddk handle, and serving requests in parallel from the same nbd socket?
Yes, but this isn't implemented in the plugins, it's implemented in the server. The server always uses a thread pool, but plugins can opt for more or less concurrency by adjusting the thread model:
http://libguestfs.org/nbdkit-plugin.3.html#Threads
The file plugin uses PARALLEL:
$ nbdkit file --dump-plugin | grep thread max_thread_model=parallel thread_model=parallel
The VDDK plugin currently uses SERIALIZE_ALL_REQUESTS:
$ nbdkit vddk --dump-plugin | grep thread max_thread_model=serialize_all_requests thread_model=serialize_all_requests
The proposal is to use SERIALIZE_REQUESTS, with an extra mutex added by the plugin around VixDiskLib_Open and _Close calls.
I'm not sure what is the difference between SERIALIZE_REQUESTS and SERIALIZE_ALL_REQUESTS, but it sounds to me like we need PARALLEL. With parallel we will have multiple threads using the same vddk_handle, which can be thread safe since we control this struct. The struct can be: struct vddk_item { VixDiskLibConnectParams *params; /* connection parameters */ VixDiskLibConnection connection; /* connection */ VixDiskLibHandle handle; /* disk handle */ struct vddk_item *next; /* next handle in the list */ } struct vddk_handle { struct vddm_items *pool; pthread_mutex_t *mutex; } open() will initialize the pool of vddk_item. pread() will: - lock the mutex - take an item from the pool - unlock the mutex - perform a single request - lock the mutex - return the item to the pool - unlock the mutex Since we don't need lot of connections, and most of the time is spent waiting on I/O, the time to lock/unlock the pool should not be significant. The server thread pool should probably use the same size of the pool, so there is always a free item in the pool for every thread. Or maybe something simpler, every thread will create a vddk_item and keep it in thread local storage, no locking required (maybe only for open and close). Do you see any reason why this will not work?
PARALLEL is not possible.
This is kind of ugly but simple, and it works great for the file plugin - we get better performance than qemu-nbd.
But since we get low throughput even when we have 10 concurrent handles for 10 different disks, I'm sure this will help, and the issue may be deeper in vmware. Maybe they intentionally throttle the clients?
The whole server side seems very heavyweight, judging by how long it takes to answer single requests. It might just be poor implementation rather than throttling though.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Wed, Aug 05, 2020 at 03:40:43PM +0300, Nir Soffer wrote:
On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote:
Can we use something like the file plugin? thread pool of workers, each keeping open vddk handle, and serving requests in parallel from the same nbd socket?
Yes, but this isn't implemented in the plugins, it's implemented in the server. The server always uses a thread pool, but plugins can opt for more or less concurrency by adjusting the thread model:
http://libguestfs.org/nbdkit-plugin.3.html#Threads
The file plugin uses PARALLEL:
$ nbdkit file --dump-plugin | grep thread max_thread_model=parallel thread_model=parallel
The VDDK plugin currently uses SERIALIZE_ALL_REQUESTS:
$ nbdkit vddk --dump-plugin | grep thread max_thread_model=serialize_all_requests thread_model=serialize_all_requests
The proposal is to use SERIALIZE_REQUESTS, with an extra mutex added by the plugin around VixDiskLib_Open and _Close calls.
I'm not sure what is the difference between SERIALIZE_REQUESTS and SERIALIZE_ALL_REQUESTS,
SERIALIZE_ALL_REQUESTS serializes requests across all handles. SERIALIZE_REQUESTS serializes requests within each handle, but multiple parallel requests can happen to different handles.
but it sounds to me like we need PARALLEL.
With parallel we will have multiple threads using the same vddk_handle, which can be thread safe since we control this struct.
The struct can be:
struct vddk_item { VixDiskLibConnectParams *params; /* connection parameters */ VixDiskLibConnection connection; /* connection */ VixDiskLibHandle handle; /* disk handle */ struct vddk_item *next; /* next handle in the list */ }
struct vddk_handle { struct vddm_items *pool; pthread_mutex_t *mutex; }
open() will initialize the pool of vddk_item.
pread() will: - lock the mutex - take an item from the pool - unlock the mutex - perform a single request - lock the mutex - return the item to the pool - unlock the mutex
Since we don't need lot of connections, and most of the time is spent waiting on I/O, the time to lock/unlock the pool should not be significant.
The server thread pool should probably use the same size of the pool, so there is always a free item in the pool for every thread.
Or maybe something simpler, every thread will create a vddk_item and keep it in thread local storage, no locking required (maybe only for open and close).
Do you see any reason why this will not work?
It sounds plausible, but since VDDK itself seems to be serialized internally (or more probably in the server) I doubt you'll see much performance improvement. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Wed, Aug 5, 2020 at 4:28 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Aug 05, 2020 at 03:40:43PM +0300, Nir Soffer wrote:
On Wed, Aug 5, 2020 at 2:58 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Aug 05, 2020 at 02:39:44PM +0300, Nir Soffer wrote:
Can we use something like the file plugin? thread pool of workers, each keeping open vddk handle, and serving requests in parallel from the same nbd socket?
Yes, but this isn't implemented in the plugins, it's implemented in the server. The server always uses a thread pool, but plugins can opt for more or less concurrency by adjusting the thread model:
http://libguestfs.org/nbdkit-plugin.3.html#Threads
The file plugin uses PARALLEL:
$ nbdkit file --dump-plugin | grep thread max_thread_model=parallel thread_model=parallel
The VDDK plugin currently uses SERIALIZE_ALL_REQUESTS:
$ nbdkit vddk --dump-plugin | grep thread max_thread_model=serialize_all_requests thread_model=serialize_all_requests
The proposal is to use SERIALIZE_REQUESTS, with an extra mutex added by the plugin around VixDiskLib_Open and _Close calls.
I'm not sure what is the difference between SERIALIZE_REQUESTS and SERIALIZE_ALL_REQUESTS,
SERIALIZE_ALL_REQUESTS serializes requests across all handles. SERIALIZE_REQUESTS serializes requests within each handle, but multiple parallel requests can happen to different handles.
I see, can change the python plugin to support multiple connections to imageio using SERIALIZE_REQUESTS? The GiL should not limit us since the GIL is released when you write to imageio socket, and this is likely where the plugin spends most of the time. I'm not sure what triggers using multiple connections in qemu-img and how it decide how many connections should be used, but we report the number of writers in OPTIONS: http://ovirt.github.io/ovirt-imageio/images.html#max_writers There is a hard limit in vdsm, because it runs qemu-nbd with --shared=8, so you should not use more than 8 connections, they will just block on qemu-nbd forever. We use 4 connections by default, giving about 100% speed up compared with one connection. 2 connections give about 80% speed up. If the number of connections is related to the number of coroutines, you can use -m 4 to use 4 coroutines. Using -W will improve performance. In this mode every coroutine will do the I/O when it is ready, instead of waiting for other coroutines and submit the I/O in the right order.
but it sounds to me like we need PARALLEL.
With parallel we will have multiple threads using the same vddk_handle, which can be thread safe since we control this struct.
The struct can be:
struct vddk_item { VixDiskLibConnectParams *params; /* connection parameters */ VixDiskLibConnection connection; /* connection */ VixDiskLibHandle handle; /* disk handle */ struct vddk_item *next; /* next handle in the list */ }
struct vddk_handle { struct vddm_items *pool; pthread_mutex_t *mutex; }
open() will initialize the pool of vddk_item.
pread() will: - lock the mutex - take an item from the pool - unlock the mutex - perform a single request - lock the mutex - return the item to the pool - unlock the mutex
Since we don't need lot of connections, and most of the time is spent waiting on I/O, the time to lock/unlock the pool should not be significant.
The server thread pool should probably use the same size of the pool, so there is always a free item in the pool for every thread.
Or maybe something simpler, every thread will create a vddk_item and keep it in thread local storage, no locking required (maybe only for open and close).
Do you see any reason why this will not work?
It sounds plausible, but since VDDK itself seems to be serialized internally (or more probably in the server) I doubt you'll see much performance improvement.
If we get better performance with multiple connections, we are likely to get the same performance using multiple threads, but it seems that using multiple connections is much simpler.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
I see, can change the python plugin to support multiple connections to imageio using SERIALIZE_REQUESTS?
The GiL should not limit us since the GIL is released when you write to imageio socket, and this is likely where the plugin spends most of the time.
It's an interesting question and one I'd not really considered at all. Does the Python GIL actively mutex different threads if they call into Python code at the same time? If it's truly a lock, then it should, in which case it should be safe to change the Python plugin to PARALLEL ... I'll try it out and get back to you.
I'm not sure what triggers using multiple connections in qemu-img and how it decide how many connections should be used, but we report the number of writers in OPTIONS: http://ovirt.github.io/ovirt-imageio/images.html#max_writers
There is a hard limit in vdsm, because it runs qemu-nbd with --shared=8, so you should not use more than 8 connections, they will just block on qemu-nbd forever.
It's different for qemu NBD client and server. Eric told me on IRC a few minutes ago that qemu NBD client does not "yet" support multi-conn. However it is implemented by qemu-nbd (the server) for readonly connections. Note that multi-conn and --shared are (a bit) different. Multi-conn is a flag which the server can send to clients to indicate not only that multiple connections are possible, but also that they are come with certain guarantees: bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to the client. In the absence of this flag, clients SHOULD NOT multiplex their commands over more than one connection to the export. “--shared” limits the number of connections that the server will accept at the same time (like the nbdkit limit filter). But it would still be possible for an NBD server to accept multiple connections from a single client but not be multi-conn safe. Also NBD lets you multiplex commands on a single connection (which does not require multi-conn or --shared). BTW I found that multi-conn is a big win with the Linux kernel NBD client.
We use 4 connections by default, giving about 100% speed up compared with one connection. 2 connections give about 80% speed up. If the number of connections is related to the number of coroutines, you can use -m 4 to use 4 coroutines.
Using -W will improve performance. In this mode every coroutine will do the I/O when it is ready, instead of waiting for other coroutines and submit the I/O in the right order.
I think Eric might have a better idea about what -m and -W really do for qemu NBD client. Maybe improve multiplexing? They don't enable multi-conn :-( Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Wed, Aug 5, 2020 at 5:10 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
I see, can change the python plugin to support multiple connections to imageio using SERIALIZE_REQUESTS?
The GiL should not limit us since the GIL is released when you write to imageio socket, and this is likely where the plugin spends most of the time.
It's an interesting question and one I'd not really considered at all. Does the Python GIL actively mutex different threads if they call into Python code at the same time?
Yes, only one thread can run python code at the same time, there is no way around this. But before calling most syscalls, python releases the GIL. When the syscall returns, python acquires the GIL again. But thread can be switched at any point in time, so must use locking in the python if you modify or access shared state.
If it's truly a lock, then it should, in which case it should be safe to change the Python plugin to PARALLEL ...
Using multiple threads on the same http socket will not work, you must have multiple sockets to imageio.
I'll try it out and get back to you.
I'm not sure what triggers using multiple connections in qemu-img and how it decide how many connections should be used, but we report the number of writers in OPTIONS: http://ovirt.github.io/ovirt-imageio/images.html#max_writers
There is a hard limit in vdsm, because it runs qemu-nbd with --shared=8, so you should not use more than 8 connections, they will just block on qemu-nbd forever.
It's different for qemu NBD client and server. Eric told me on IRC a few minutes ago that qemu NBD client does not "yet" support multi-conn. However it is implemented by qemu-nbd (the server) for readonly connections.
Note that multi-conn and --shared are (a bit) different. Multi-conn is a flag which the server can send to clients to indicate not only that multiple connections are possible, but also that they are come with certain guarantees:
bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to the client. In the absence of this flag, clients SHOULD NOT multiplex their commands over more than one connection to the export.
“--shared” limits the number of connections that the server will accept at the same time (like the nbdkit limit filter). But it would still be possible for an NBD server to accept multiple connections from a single client but not be multi-conn safe.
Also NBD lets you multiplex commands on a single connection (which does not require multi-conn or --shared).
BTW I found that multi-conn is a big win with the Linux kernel NBD client.
We use 4 connections by default, giving about 100% speed up compared with one connection. 2 connections give about 80% speed up. If the number of connections is related to the number of coroutines, you can use -m 4 to use 4 coroutines.
Using -W will improve performance. In this mode every coroutine will do the I/O when it is ready, instead of waiting for other coroutines and submit the I/O in the right order.
I think Eric might have a better idea about what -m and -W really do for qemu NBD client. Maybe improve multiplexing? They don't enable multi-conn :-(
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Wed, Aug 5, 2020 at 5:18 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Aug 5, 2020 at 5:10 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
I see, can change the python plugin to support multiple connections to imageio using SERIALIZE_REQUESTS?
The GiL should not limit us since the GIL is released when you write to imageio socket, and this is likely where the plugin spends most of the time.
It's an interesting question and one I'd not really considered at all. Does the Python GIL actively mutex different threads if they call into Python code at the same time?
Yes, only one thread can run python code at the same time, there is no way around this. But before calling most syscalls, python releases the GIL. When the syscall returns, python acquires the GIL again.
But thread can be switched at any point in time, so must use locking in the python if you modify or access shared state.
If it's truly a lock, then it should, in which case it should be safe to change the Python plugin to PARALLEL ...
Using multiple threads on the same http socket will not work, you must have multiple sockets to imageio.
And we also have to change the rhv-upload-pluign, since we create the disk and validate the current hosts in open(). These operation should be done once. So the flow will be: 1. pre check 2. prepare overlay 3. create disk 4. check if the current host can do the upload 5. run qemu-img convert 6. post (create vm or delete disk on failure)
I'll try it out and get back to you.
I'm not sure what triggers using multiple connections in qemu-img and how it decide how many connections should be used, but we report the number of writers in OPTIONS: http://ovirt.github.io/ovirt-imageio/images.html#max_writers
There is a hard limit in vdsm, because it runs qemu-nbd with --shared=8, so you should not use more than 8 connections, they will just block on qemu-nbd forever.
It's different for qemu NBD client and server. Eric told me on IRC a few minutes ago that qemu NBD client does not "yet" support multi-conn. However it is implemented by qemu-nbd (the server) for readonly connections.
Note that multi-conn and --shared are (a bit) different. Multi-conn is a flag which the server can send to clients to indicate not only that multiple connections are possible, but also that they are come with certain guarantees:
bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to the client. In the absence of this flag, clients SHOULD NOT multiplex their commands over more than one connection to the export.
“--shared” limits the number of connections that the server will accept at the same time (like the nbdkit limit filter). But it would still be possible for an NBD server to accept multiple connections from a single client but not be multi-conn safe.
Also NBD lets you multiplex commands on a single connection (which does not require multi-conn or --shared).
BTW I found that multi-conn is a big win with the Linux kernel NBD client.
We use 4 connections by default, giving about 100% speed up compared with one connection. 2 connections give about 80% speed up. If the number of connections is related to the number of coroutines, you can use -m 4 to use 4 coroutines.
Using -W will improve performance. In this mode every coroutine will do the I/O when it is ready, instead of waiting for other coroutines and submit the I/O in the right order.
I think Eric might have a better idea about what -m and -W really do for qemu NBD client. Maybe improve multiplexing? They don't enable multi-conn :-(
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 8/5/20 9:10 AM, Richard W.M. Jones wrote:
On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
I see, can change the python plugin to support multiple connections to imageio using SERIALIZE_REQUESTS?
The GiL should not limit us since the GIL is released when you write to imageio socket, and this is likely where the plugin spends most of the time.
It's an interesting question and one I'd not really considered at all. Does the Python GIL actively mutex different threads if they call into Python code at the same time? If it's truly a lock, then it should, in which case it should be safe to change the Python plugin to PARALLEL ...
I'll try it out and get back to you.
Yeah, I would not be surprised if we could make the Python plugin more performant, but matching our glue code to the python docs for embedding with C code is not trivial, so I haven't spent the time trying.
Also NBD lets you multiplex commands on a single connection (which does not require multi-conn or --shared).
BTW I found that multi-conn is a big win with the Linux kernel NBD client.
We use 4 connections by default, giving about 100% speed up compared with one connection. 2 connections give about 80% speed up. If the number of connections is related to the number of coroutines, you can use -m 4 to use 4 coroutines.
Using -W will improve performance. In this mode every coroutine will do the I/O when it is ready, instead of waiting for other coroutines and submit the I/O in the right order.
I think Eric might have a better idea about what -m and -W really do for qemu NBD client. Maybe improve multiplexing? They don't enable multi-conn :-(
Correct. Using -W doesn't make sense without -m (if you only have one worker, you might as well proceed linearly than trying to randomize access, but even when you have multiple threads, there are cases where linear operations are still useful, such as 'nbdkit streaming'. But -m is definitely the knob that controls how many outstanding I/O requests qemu-img is willing to use; and once you are using -m, using -W makes life easier for those coroutines to stay active. The default -m1 says that at most one request is outstanding, so parallelism in the server is not utilized. With higher -m, qemu-img issues up to that many requests without waiting for server answers, but all on the same NBD connection. Ideally, you'll get the maximum behavior as 'qemu-img -m' and 'nbdkit --threads' choose the same values; if either side has fewer in-flight operations permitted than the other, then that side has the potential to become a bottleneck. Right now, nbdkit defaults to 16 threads (that is, up to 16 in-flight operations) for any PARALLEL plugin. And someday, I'd love to improve nbdkit's PARALLEL mode to make its thread-pool more of an on-demand setup (right now, we pre-create all 16 threads up front, even if the client never reaches 16 in-flight operations at once, which is a bit wasteful), but other than potential performance improvements, it should be a transparent change to both plugins and clients. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On 8/5/20 9:38 AM, Eric Blake wrote:
Correct. Using -W doesn't make sense without -m (if you only have one worker, you might as well proceed linearly than trying to randomize access, but even when you have multiple threads, there are cases where linear operations are still useful, such as 'nbdkit streaming'. But -m is definitely the knob that controls how many outstanding I/O requests qemu-img is willing to use; and once you are using -m, using -W makes life easier for those coroutines to stay active. The default -m1 says that at most one request is outstanding, so parallelism in the server is not utilized. With higher -m, qemu-img issues up to that many requests without waiting for server answers, but all on the same NBD connection. Ideally, you'll get the maximum behavior as 'qemu-img -m' and 'nbdkit --threads' choose the same values; if either side has fewer in-flight operations permitted than the other, then that side has the potential to become a bottleneck. Right now, nbdkit defaults to 16 threads (that is, up to 16 in-flight operations) for any PARALLEL plugin.
This reminds me of a previous thread: we probably want to add a new mode to nbdkit parallelism: https://www.redhat.com/archives/libguestfs/2019-May/msg00089.html Use of qemu-nbd -m without -W is like my proposed NBDKIT_THREAD_MODEL_SERIALIZE_RETIREMENT (okay, I just invented that name; I did not name it in that thread). Use of qemu-nbd -m with -W is then NBDKIT_THREAD_MODEL_PARALLEL. Use of qemu-nbd without -m (at which point -W makes no difference) is like NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS. The middle mode is where you allow multiple in-flight requests, but serialize their retirement (different than serializing their issuance), for a client that can't deal with out-of-order replies. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Wed, Aug 5, 2020 at 5:38 PM Eric Blake <eblake@redhat.com> wrote:
On 8/5/20 9:10 AM, Richard W.M. Jones wrote:
On Wed, Aug 05, 2020 at 04:49:04PM +0300, Nir Soffer wrote:
I see, can change the python plugin to support multiple connections to imageio using SERIALIZE_REQUESTS?
The GiL should not limit us since the GIL is released when you write to imageio socket, and this is likely where the plugin spends most of the time.
It's an interesting question and one I'd not really considered at all. Does the Python GIL actively mutex different threads if they call into Python code at the same time? If it's truly a lock, then it should, in which case it should be safe to change the Python plugin to PARALLEL ...
I'll try it out and get back to you.
Yeah, I would not be surprised if we could make the Python plugin more performant, but matching our glue code to the python docs for embedding with C code is not trivial, so I haven't spent the time trying.
Using multiple threads is easy when you call C from pyhton. I'm not sure how it can work when you embed python into C. When the python pwrite() is called, the C extension is holding the GIL. When pwrite call socket.send() python will release the GIL. Theoretically, the thread that called pwrite is blocked and another thread can call Python C API, but it may also break horribly if the blocked thread modified some global state and is not expecting other code to run until the python code returns. I never tried to do this. A better way is to separate the management code (using oVirt python SDK) from the transfer code (using http client). The management code should run before the transfer, setting up the transfer, and after the transfer (finalling it). The transfer code can be written in C using libcurl using parallel mode.
Also NBD lets you multiplex commands on a single connection (which does not require multi-conn or --shared).
BTW I found that multi-conn is a big win with the Linux kernel NBD client.
We use 4 connections by default, giving about 100% speed up compared with one connection. 2 connections give about 80% speed up. If the number of connections is related to the number of coroutines, you can use -m 4 to use 4 coroutines.
Using -W will improve performance. In this mode every coroutine will do the I/O when it is ready, instead of waiting for other coroutines and submit the I/O in the right order.
I think Eric might have a better idea about what -m and -W really do for qemu NBD client. Maybe improve multiplexing? They don't enable multi-conn :-(
Correct. Using -W doesn't make sense without -m (if you only have one worker, you might as well proceed linearly than trying to randomize access, but even when you have multiple threads, there are cases where linear operations are still useful, such as 'nbdkit streaming'. But -m is definitely the knob that controls how many outstanding I/O requests qemu-img is willing to use; and once you are using -m, using -W makes life easier for those coroutines to stay active. The default -m1 says that at most one request is outstanding, so parallelism in the server is not utilized. With higher -m, qemu-img issues up to that many requests without waiting for server answers, but all on the same NBD connection. Ideally, you'll get the maximum behavior as 'qemu-img -m' and 'nbdkit --threads' choose the same values; if either side has fewer in-flight operations permitted than the other, then that side has the potential to become a bottleneck. Right now, nbdkit defaults to 16 threads (that is, up to 16 in-flight operations) for any PARALLEL plugin.
And someday, I'd love to improve nbdkit's PARALLEL mode to make its thread-pool more of an on-demand setup (right now, we pre-create all 16 threads up front, even if the client never reaches 16 in-flight operations at once, which is a bit wasteful), but other than potential performance improvements, it should be a transparent change to both plugins and clients.
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Nir, BTW what are you using for performance testing? As far as I can tell it's not possible to make qemu-img convert use multi-conn when connecting to the source (which is going to be a problem if we want to use this stuff in virt-v2v). Instead I've hacked up a copy of this program from libnbd: https://github.com/libguestfs/libnbd/blob/master/examples/threaded-reads-and... so that it only does reads and aligns requests to 512 bytes. At least this is testing multi-conn, but there should be an easier way ... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Wed, Aug 5, 2020 at 3:28 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Nir, BTW what are you using for performance testing?
virt-v2v with local image, or imageio client with local image.
As far as I can tell it's not possible to make qemu-img convert use multi-conn when connecting to the source (which is going to be a problem if we want to use this stuff in virt-v2v).
But do we need multiple connections? qemu can send multiple requests on one connection. Did you try to copy an image from nbdkit file plugin to another nbdkit file plugin using qemu-img convert? nbdkit file pluing -> qemu-img convert -W nbd:///?socket=src.sock nbd:///?socket=dst.sock-> nbdkit file plugin I did not try it but I will be surprised if we don't get all 8 threads busy in both sides The reason we use multiple connection in imageio is that we don't support async I/O in http client, http server, and nbd client, and it is much easier to open new connection with the entire stuck compared to rewriting the http server and nbd client. It also much hard to provide easy to use interface for users supporting async I/O.
Instead I've hacked up a copy of this program from libnbd:
https://github.com/libguestfs/libnbd/blob/master/examples/threaded-reads-and...
so that it only does reads and aligns requests to 512 bytes.
At least this is testing multi-conn, but there should be an easier way ...
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Wed, Aug 05, 2020 at 03:39:15PM +0300, Nir Soffer wrote:
On Wed, Aug 5, 2020 at 3:28 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Nir, BTW what are you using for performance testing?
virt-v2v with local image, or imageio client with local image.
As far as I can tell it's not possible to make qemu-img convert use multi-conn when connecting to the source (which is going to be a problem if we want to use this stuff in virt-v2v).
But do we need multiple connections? qemu can send multiple requests on one connection.
As implemented now there is only one VDDK handle per connection and VDDK doesn't allow multiple requests at the same time on one handle, so with a single NBD connection everything will be serialized. Now if we were to implement a thread pool inside nbdkit-vddk-plugin we could get around that restriction (with a lot of complexity). But the results I posted a moment ago show that we wouldn't get anything like a linear speed up. It hardly seems worth it to me.
Did you try to copy an image from nbdkit file plugin to another nbdkit file plugin using qemu-img convert?
nbdkit file pluing -> qemu-img convert -W nbd:///?socket=src.sock nbd:///?socket=dst.sock-> nbdkit file plugin
I did not try it but I will be surprised if we don't get all 8 threads busy in both sides
Likely, but that's because the file plugin uses PARALLEL as thread model, so even on a single NBD connection it can keep all threads inside nbdkit busy.
The reason we use multiple connection in imageio is that we don't support async I/O in http client, http server, and nbd client, and it is much easier to open new connection with the entire stuck compared to rewriting the http server and nbd client.
It also much hard to provide easy to use interface for users supporting async I/O.
Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Here are some results anyway. The command I'm using is: $ ./nbdkit -r -U - vddk \ libdir=/path/to/vmware-vix-disklib-distrib \ user=root password='***' \ server='***' thumbprint=aa:bb:cc:... \ vm=moref=3 \ file='[datastore1] Fedora 28/Fedora 28.vmdk' \ --run 'time /var/tmp/threaded-reads $unixsocket' Source for threaded-reads is attached. (1) Existing nbdkit VDDK plugin. NR_MULTI_CONN = 1 NR_CYCLES = 10000 Note this is making 10,000 pread requests. real 1m26.103s user 0m0.283s sys 0m0.571s (2) VDDK plugin patched to support SERIALIZE_REQUESTS. NR_MULTI_CONN = 1 NR_CYCLES = 10000 Note this is making 10,000 pread requests. real 1m26.755s user 0m0.230s sys 0m0.539s (3) VDDK plugin same as in (2). NR_MULTI_CONN = 8 NR_CYCLES = 10000 Note this is making 80,000 pread requests in total. real 7m11.729s user 0m2.891s sys 0m6.037s My observations: Tests (1) and (2) are about the same within noise. Test (3) is making 8 times as many requests as test (1), so I think it's fair to compare the 8 x time taken by test (1) (ie. the time it would have taking to make 80,000 requests): Test (1) * 8 = 11m28 Test (3) = 7m11 So if we had a client which could actually use multi-conn then this would be a reasonable win. It seems like there's still a lot of locking going on somewhere, perhaps inside VDDK or in the server. It's certainly nowhere near a linear speedup. The patch does at least seem stable. I'll post it in a minute. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 8/5/20 7:47 AM, Richard W.M. Jones wrote:
Here are some results anyway. The command I'm using is:
$ ./nbdkit -r -U - vddk \ libdir=/path/to/vmware-vix-disklib-distrib \ user=root password='***' \ server='***' thumbprint=aa:bb:cc:... \ vm=moref=3 \ file='[datastore1] Fedora 28/Fedora 28.vmdk' \ --run 'time /var/tmp/threaded-reads $unixsocket'
Source for threaded-reads is attached.
Tests (1) and (2) are about the same within noise.
Test (3) is making 8 times as many requests as test (1), so I think it's fair to compare the 8 x time taken by test (1) (ie. the time it would have taking to make 80,000 requests):
Test (1) * 8 = 11m28 Test (3) = 7m11
So if we had a client which could actually use multi-conn then this would be a reasonable win. It seems like there's still a lot of locking going on somewhere, perhaps inside VDDK or in the server. It's certainly nowhere near a linear speedup.
If I'm reading https://code.vmware.com/docs/11750/virtual-disk-development-kit-programming-... correctly, we cannot reuse a single VDDK handle for two parallel requests, but we CAN have two VDDK handles open, with requests open on both handles at the same time. That's what SERIALIZE_REQUESTS buys us - a client that opens multiple connections (taking advantage of multi-conn) now has two NBD handles and therefore two VDDK handles, and the reads separated across those two handles can proceed in parallel. But I also don't see anything that prevents a single NBD connection from opening multiple VDDK handles under the hood, or even from having all of those handles opened as coordinated through a single helper thread. That is, if nbdkit were to provide a way for a plugin to know the maximum number of threads that will be used in parallel, then vddk's .after_fork could spawn a dedicated thread for running VDDK handle open/close requests (protected by a mutex), the .open callback can then obtain the mutex, call into the helper thread to open N handles, the thread model is advertised as PARALLEL, and in all other calls (.pread, .pwrite, ...) we pick any one of the N handles for that NBD connection that is not currently in use. The client application would not even have to take advantage of multi-conn, but gets the full benefit of out-of-order thread access for a parallel speedup.
The patch does at least seem stable. I'll post it in a minute.
Whether we do all VixDiskLib_Open from a single dedicated helper thread created during .after_fork, or rely on pthread mutex locking so that at most one .open is calling Open or Close, is different from whether we open multiple VDDK handles per single NBD connection in PARALLEL mode, vs. one VDDK handle per NBD connection in SERIALIZE_REQUESTS (offloading the parallelism to the multi-conn client). We could probably do both, but while opening multiple VDDK handles per NBD connection appears to be fully compliant with VDDK docs (patch not written yet), the former (using a mutex to serialize open calls, but open is not happening in the same thread all the time) is indeed risky. And having the speedup available to all clients, not just multi-conn aware clients, seems like it might be nice to have, even if the speedup is not linear. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Wed, Aug 5, 2020 at 3:47 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Here are some results anyway. The command I'm using is:
$ ./nbdkit -r -U - vddk \ libdir=/path/to/vmware-vix-disklib-distrib \ user=root password='***' \ server='***' thumbprint=aa:bb:cc:... \ vm=moref=3 \ file='[datastore1] Fedora 28/Fedora 28.vmdk' \ --run 'time /var/tmp/threaded-reads $unixsocket'
Source for threaded-reads is attached.
(1) Existing nbdkit VDDK plugin.
NR_MULTI_CONN = 1 NR_CYCLES = 10000
Note this is making 10,000 pread requests.
real 1m26.103s user 0m0.283s sys 0m0.571s
(2) VDDK plugin patched to support SERIALIZE_REQUESTS.
NR_MULTI_CONN = 1 NR_CYCLES = 10000
Note this is making 10,000 pread requests.
real 1m26.755s user 0m0.230s sys 0m0.539s
(3) VDDK plugin same as in (2).
NR_MULTI_CONN = 8 NR_CYCLES = 10000
Note this is making 80,000 pread requests in total.
real 7m11.729s user 0m2.891s sys 0m6.037s
My observations:
Tests (1) and (2) are about the same within noise.
Test (3) is making 8 times as many requests as test (1), so I think it's fair to compare the 8 x time taken by test (1) (ie. the time it would have taking to make 80,000 requests):
Test (1) * 8 = 11m28 Test (3) = 7m11
That's pretty good results, 62% faster. What is the request size used? I would test 1, 2, 4, 8 MiB reads.
So if we had a client which could actually use multi-conn then this would be a reasonable win. It seems like there's still a lot of locking going on somewhere, perhaps inside VDDK or in the server. It's certainly nowhere near a linear speedup.
The patch does at least seem stable. I'll post it in a minute.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Wed, Aug 05, 2020 at 05:40:18PM +0300, Nir Soffer wrote:
On Wed, Aug 5, 2020 at 3:47 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Here are some results anyway. The command I'm using is:
$ ./nbdkit -r -U - vddk \ libdir=/path/to/vmware-vix-disklib-distrib \ user=root password='***' \ server='***' thumbprint=aa:bb:cc:... \ vm=moref=3 \ file='[datastore1] Fedora 28/Fedora 28.vmdk' \ --run 'time /var/tmp/threaded-reads $unixsocket'
Source for threaded-reads is attached.
(1) Existing nbdkit VDDK plugin.
NR_MULTI_CONN = 1 NR_CYCLES = 10000
Note this is making 10,000 pread requests.
real 1m26.103s user 0m0.283s sys 0m0.571s
(2) VDDK plugin patched to support SERIALIZE_REQUESTS.
NR_MULTI_CONN = 1 NR_CYCLES = 10000
Note this is making 10,000 pread requests.
real 1m26.755s user 0m0.230s sys 0m0.539s
(3) VDDK plugin same as in (2).
NR_MULTI_CONN = 8 NR_CYCLES = 10000
Note this is making 80,000 pread requests in total.
real 7m11.729s user 0m2.891s sys 0m6.037s
My observations:
Tests (1) and (2) are about the same within noise.
Test (3) is making 8 times as many requests as test (1), so I think it's fair to compare the 8 x time taken by test (1) (ie. the time it would have taking to make 80,000 requests):
Test (1) * 8 = 11m28 Test (3) = 7m11
That's pretty good results, 62% faster.
What is the request size used? I would test 1, 2, 4, 8 MiB reads.
Randomly 512 bytes and 1M requests. Yes we should probably test reads of different sizes. Rich.
So if we had a client which could actually use multi-conn then this would be a reasonable win. It seems like there's still a lot of locking going on somewhere, perhaps inside VDDK or in the server. It's certainly nowhere near a linear speedup.
The patch does at least seem stable. I'll post it in a minute.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
participants (3)
-
Eric Blake -
Nir Soffer -
Richard W.M. Jones