Two multi-conn questions
Hi Eric, A couple of semi-related multi-conn questions. (1) nbdkit-curl-plugin does not currently advertise multi-conn. However forcing multi-conn makes it faster, eg: $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' █ 100% [****************************************] real 1m15.438s user 0m2.513s sys 0m4.063s $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' █ 100% [****************************************] real 0m59.635s user 0m3.010s sys 0m4.538s (Note that this is line speed - wget also takes 59 seconds) I believe that multi-conn should be safe to enable in the curl plugin, certainly if we're in read-only mode (which is the most common case for this plugin). It's probably not safe when operating in read-write mode, because HTTP doesn't have any concept of flushing, but almost no web server supports writing. (2) qemu's NBD client still doesn't seem to support multi-conn. This makes qemu-img convert very slow when reading from nbdkit, as in the example below. (Note the '.img' file on that website is actually a qcow2 file.) $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' qemu-img convert -W -m 16 -p -f qcow2 $uri -O raw jammy-server-cloudimg-amd64.img.raw ' (100.00/100%) real 3m29.412s user 0m17.856s sys 0m8.218s Since we know from (1) above that copying could go at line speed when multi-conn is enabled, and also that qemu-img convert from the same local file takes only about 2 seconds, it seems like qemu-img convert above could go a lot faster if we allowed multi-conn. Is there any work on adding multi-conn support to qemu's NBD client? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Wed, Feb 01, 2023 at 12:40:45PM +0000, Richard W.M. Jones wrote:
Hi Eric,
A couple of semi-related multi-conn questions.
(1) nbdkit-curl-plugin does not currently advertise multi-conn. However forcing multi-conn makes it faster, eg:
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' █ 100% [****************************************]
real 1m15.438s user 0m2.513s sys 0m4.063s
I'm jealous of your connection. Even with bare wget (without nbdkit in the middle), for some reason my computer is failing to connect when trying IPv6. I then tried 'wget -4 ...', and didn't get the full file until 5m37s. And when I tried to reproduce your nbdkit line, I got: nbdkit: curl[1]: error: pread: curl_easy_perform: Failed sending data to the peer: Connection died, tried 5 times before giving up nbdcopy: read at offset 245104640 failed: Input/output error at 6 minutes in. So slow speeds and some connection flakiness is not helping me.
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '
Yay - I'm glad my multi-conn filter makes it easier to test things like this! Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention that despite multi-conn-mode=unsafe being unsafe for a plugin that does not have consistency, it is useful for timing tests on a plugin where we suspect consistency is available in order to test timing to see if it actually makes a difference?
█ 100% [****************************************]
real 0m59.635s user 0m3.010s sys 0m4.538s
(Note that this is line speed - wget also takes 59 seconds)
I believe that multi-conn should be safe to enable in the curl plugin, certainly if we're in read-only mode (which is the most common case for this plugin). It's probably not safe when operating in read-write mode, because HTTP doesn't have any concept of flushing, but almost no web server supports writing.
Agreed on both fronts; I see you have proposed this patch (and I acked it) in the meantime.
(2) qemu's NBD client still doesn't seem to support multi-conn. This makes qemu-img convert very slow when reading from nbdkit, as in the example below. (Note the '.img' file on that website is actually a qcow2 file.)
Correct - qemu does NOT do multi-queue (or multipath) client connections. It is blindly using exactly one NBD client connection per block device, and ignoring the server's advertisement of multi-conn support. Teaching qemu to support a multipath paradigm is not a trivial task. I'm wondering if the effort might be easier by teaching libblkio how to do arbitrary multipath for any underlying block device, coupled with current efforts to teach qemu how to target any arbitrary libblkio target, so that qemu proper doesn't have to worry about the intricacies of multipath; from there, we'd need a way for libblkio to target NBD devices (probably by libnbd). In other words, instead of trying to directly code client MULTI_CONN efforts into qemu proper, it may be better to segregate things into more modular building blocks.
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' qemu-img convert -W -m 16 -p -f qcow2 $uri -O raw jammy-server-cloudimg-amd64.img.raw ' (100.00/100%)
real 3m29.412s user 0m17.856s sys 0m8.218s
Since we know from (1) above that copying could go at line speed when multi-conn is enabled, and also that qemu-img convert from the same local file takes only about 2 seconds, it seems like qemu-img convert above could go a lot faster if we allowed multi-conn.
Is there any work on adding multi-conn support to qemu's NBD client?
Not that I'm aware of at the moment, but we have proof that it may prove fruitful to have someone spend time on. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Thu, Feb 02, 2023 at 10:04:20AM -0600, Eric Blake wrote:
On Wed, Feb 01, 2023 at 12:40:45PM +0000, Richard W.M. Jones wrote:
Hi Eric,
A couple of semi-related multi-conn questions.
(1) nbdkit-curl-plugin does not currently advertise multi-conn. However forcing multi-conn makes it faster, eg:
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' █ 100% [****************************************]
real 1m15.438s user 0m2.513s sys 0m4.063s
I'm jealous of your connection. Even with bare wget (without nbdkit in the middle), for some reason my computer is failing to connect when trying IPv6. I then tried 'wget -4 ...', and didn't get the full file until 5m37s. And when I tried to reproduce your nbdkit line, I got:
nbdkit: curl[1]: error: pread: curl_easy_perform: Failed sending data to the peer: Connection died, tried 5 times before giving up nbdcopy: read at offset 245104640 failed: Input/output error
at 6 minutes in. So slow speeds and some connection flakiness is not helping me.
This was the original complaint in: https://github.com/kubevirt/containerized-data-importer/issues/2561
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '
Yay - I'm glad my multi-conn filter makes it easier to test things like this!
Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention that despite multi-conn-mode=unsafe being unsafe for a plugin that does not have consistency, it is useful for timing tests on a plugin where we suspect consistency is available in order to test timing to see if it actually makes a difference?
Yes, sounds good.
█ 100% [****************************************]
real 0m59.635s user 0m3.010s sys 0m4.538s
(Note that this is line speed - wget also takes 59 seconds)
I believe that multi-conn should be safe to enable in the curl plugin, certainly if we're in read-only mode (which is the most common case for this plugin). It's probably not safe when operating in read-write mode, because HTTP doesn't have any concept of flushing, but almost no web server supports writing.
Agreed on both fronts; I see you have proposed this patch (and I acked it) in the meantime.
(2) qemu's NBD client still doesn't seem to support multi-conn. This makes qemu-img convert very slow when reading from nbdkit, as in the example below. (Note the '.img' file on that website is actually a qcow2 file.)
Correct - qemu does NOT do multi-queue (or multipath) client connections. It is blindly using exactly one NBD client connection per block device, and ignoring the server's advertisement of multi-conn support.
Teaching qemu to support a multipath paradigm is not a trivial task. I'm wondering if the effort might be easier by teaching libblkio how to do arbitrary multipath for any underlying block device, coupled with current efforts to teach qemu how to target any arbitrary libblkio target, so that qemu proper doesn't have to worry about the intricacies of multipath; from there, we'd need a way for libblkio to target NBD devices (probably by libnbd). In other words, instead of trying to directly code client MULTI_CONN efforts into qemu proper, it may be better to segregate things into more modular building blocks.
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' qemu-img convert -W -m 16 -p -f qcow2 $uri -O raw jammy-server-cloudimg-amd64.img.raw ' (100.00/100%)
real 3m29.412s user 0m17.856s sys 0m8.218s
Since we know from (1) above that copying could go at line speed when multi-conn is enabled, and also that qemu-img convert from the same local file takes only about 2 seconds, it seems like qemu-img convert above could go a lot faster if we allowed multi-conn.
Is there any work on adding multi-conn support to qemu's NBD client?
Not that I'm aware of at the moment, but we have proof that it may prove fruitful to have someone spend time on.
Kubernetes team are complaining that this particular case (stream qcow2 file from a website and convert it to raw without a temporary file) is slow. This is not a case that I've encountered before because it's not relevant to virt-v2v, but it does appear to be important. - * - * - I didn't want to complicate the original message with irrelevant stuff, but there's something else I want to mention now. If we just use qemu's curl driver (thus eliminating NBD & nbdkit from the mix), it gets even slower: $ time ~/d/qemu/build/qemu-img convert -p -W -f qcow2 'json:{ "file.readahead": 67108864, "file.driver": "http", "file.url": "http://oirase.annexia.org/tmp/jammy-server-cloudimg-amd64.qcow2", "file.timeout":2000 }' -O raw jammy-server-cloudimg-amd64.img.raw (100.00/100%) real 3m53.923s user 0m13.751s sys 0m15.346s Now this is not something I'm personally concerned about (since I've long been arguing we should deprecate the qemu curl driver and use nbdkit), but it's also very surprising. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On Thu, Feb 02, 2023 at 04:26:04PM +0000, Richard W.M. Jones wrote: ...
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '
Yay - I'm glad my multi-conn filter makes it easier to test things like this!
Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention that despite multi-conn-mode=unsafe being unsafe for a plugin that does not have consistency, it is useful for timing tests on a plugin where we suspect consistency is available in order to test timing to see if it actually makes a difference?
Yes, sounds good.
How about: diff --git i/filters/multi-conn/nbdkit-multi-conn-filter.pod w/filters/multi-conn/nbdkit-multi-conn-filter.pod index 87b31692..7f70ade9 100644 --- i/filters/multi-conn/nbdkit-multi-conn-filter.pod +++ w/filters/multi-conn/nbdkit-multi-conn-filter.pod @@ -118,7 +118,11 @@ passed on to the plugin. When B<unsafe> mode is chosen, this filter blindly advertises multi-conn to the client even if the plugin lacks support. This is dangerous, and risks data corruption if the client makes assumptions -about flush consistency that were not actually met. +about flush consistency that were not actually met. However, for a +plugin that does not yet advertise multi-conn, but where it is +suspected that the plugin behaves consistently, this is a great way to +run timing and accuracy tests to see enabling multi-conn in the plugin +will make a difference. =item B<multi-conn-track-dirty=fast>
Is there any work on adding multi-conn support to qemu's NBD client?
Not that I'm aware of at the moment, but we have proof that it may prove fruitful to have someone spend time on.
Kubernetes team are complaining that this particular case (stream qcow2 file from a website and convert it to raw without a temporary file) is slow. This is not a case that I've encountered before because it's not relevant to virt-v2v, but it does appear to be important.
- * - * -
I didn't want to complicate the original message with irrelevant stuff, but there's something else I want to mention now. If we just use qemu's curl driver (thus eliminating NBD & nbdkit from the mix), it gets even slower:
$ time ~/d/qemu/build/qemu-img convert -p -W -f qcow2 'json:{ "file.readahead": 67108864, "file.driver": "http", "file.url": "http://oirase.annexia.org/tmp/jammy-server-cloudimg-amd64.qcow2", "file.timeout":2000 }' -O raw jammy-server-cloudimg-amd64.img.raw (100.00/100%)
real 3m53.923s user 0m13.751s sys 0m15.346s
Now this is not something I'm personally concerned about (since I've long been arguing we should deprecate the qemu curl driver and use nbdkit), but it's also very surprising.
I'm also not surprised if qemu's curl driver is not efficient, because it does not seem to be a case frequently used. Offloading it to other paths, like nbdkit, is indeed a good goal. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Thu, Feb 02, 2023 at 12:54:16PM -0600, Eric Blake wrote:
On Thu, Feb 02, 2023 at 04:26:04PM +0000, Richard W.M. Jones wrote: ...
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '
Yay - I'm glad my multi-conn filter makes it easier to test things like this!
Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention that despite multi-conn-mode=unsafe being unsafe for a plugin that does not have consistency, it is useful for timing tests on a plugin where we suspect consistency is available in order to test timing to see if it actually makes a difference?
Yes, sounds good.
How about:
diff --git i/filters/multi-conn/nbdkit-multi-conn-filter.pod w/filters/multi-conn/nbdkit-multi-conn-filter.pod index 87b31692..7f70ade9 100644 --- i/filters/multi-conn/nbdkit-multi-conn-filter.pod +++ w/filters/multi-conn/nbdkit-multi-conn-filter.pod @@ -118,7 +118,11 @@ passed on to the plugin. When B<unsafe> mode is chosen, this filter blindly advertises multi-conn to the client even if the plugin lacks support. This is dangerous, and risks data corruption if the client makes assumptions -about flush consistency that were not actually met. +about flush consistency that were not actually met. However, for a +plugin that does not yet advertise multi-conn, but where it is +suspected that the plugin behaves consistently, this is a great way to +run timing and accuracy tests to see enabling multi-conn in the plugin
^^^ "if" (or "whether")
+will make a difference.
ACK
=item B<multi-conn-track-dirty=fast>
Is there any work on adding multi-conn support to qemu's NBD client?
Not that I'm aware of at the moment, but we have proof that it may prove fruitful to have someone spend time on.
Kubernetes team are complaining that this particular case (stream qcow2 file from a website and convert it to raw without a temporary file) is slow. This is not a case that I've encountered before because it's not relevant to virt-v2v, but it does appear to be important.
- * - * -
I didn't want to complicate the original message with irrelevant stuff, but there's something else I want to mention now. If we just use qemu's curl driver (thus eliminating NBD & nbdkit from the mix), it gets even slower:
$ time ~/d/qemu/build/qemu-img convert -p -W -f qcow2 'json:{ "file.readahead": 67108864, "file.driver": "http", "file.url": "http://oirase.annexia.org/tmp/jammy-server-cloudimg-amd64.qcow2", "file.timeout":2000 }' -O raw jammy-server-cloudimg-amd64.img.raw (100.00/100%)
real 3m53.923s user 0m13.751s sys 0m15.346s
Now this is not something I'm personally concerned about (since I've long been arguing we should deprecate the qemu curl driver and use nbdkit), but it's also very surprising.
I'm also not surprised if qemu's curl driver is not efficient, because it does not seem to be a case frequently used. Offloading it to other paths, like nbdkit, is indeed a good goal.
Yes it may just be that the curl driver is slow. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
participants (2)
-
Eric Blake -
Richard W.M. Jones