On Thu, Feb 02, 2023 at 04:26:04PM +0000, Richard W.M. Jones wrote: ...
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.im... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '
Yay - I'm glad my multi-conn filter makes it easier to test things like this!
Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention that despite multi-conn-mode=unsafe being unsafe for a plugin that does not have consistency, it is useful for timing tests on a plugin where we suspect consistency is available in order to test timing to see if it actually makes a difference?
Yes, sounds good.
How about: diff --git i/filters/multi-conn/nbdkit-multi-conn-filter.pod w/filters/multi-conn/nbdkit-multi-conn-filter.pod index 87b31692..7f70ade9 100644 --- i/filters/multi-conn/nbdkit-multi-conn-filter.pod +++ w/filters/multi-conn/nbdkit-multi-conn-filter.pod @@ -118,7 +118,11 @@ passed on to the plugin. When B<unsafe> mode is chosen, this filter blindly advertises multi-conn to the client even if the plugin lacks support. This is dangerous, and risks data corruption if the client makes assumptions -about flush consistency that were not actually met. +about flush consistency that were not actually met. However, for a +plugin that does not yet advertise multi-conn, but where it is +suspected that the plugin behaves consistently, this is a great way to +run timing and accuracy tests to see enabling multi-conn in the plugin +will make a difference. =item B<multi-conn-track-dirty=fast>
Is there any work on adding multi-conn support to qemu's NBD client?
Not that I'm aware of at the moment, but we have proof that it may prove fruitful to have someone spend time on.
Kubernetes team are complaining that this particular case (stream qcow2 file from a website and convert it to raw without a temporary file) is slow. This is not a case that I've encountered before because it's not relevant to virt-v2v, but it does appear to be important.
- * - * -
I didn't want to complicate the original message with irrelevant stuff, but there's something else I want to mention now. If we just use qemu's curl driver (thus eliminating NBD & nbdkit from the mix), it gets even slower:
$ time ~/d/qemu/build/qemu-img convert -p -W -f qcow2 'json:{ "file.readahead": 67108864, "file.driver": "http", "file.url": "http://oirase.annexia.org/tmp/jammy-server-cloudimg-amd64.qcow2", "file.timeout":2000 }' -O raw jammy-server-cloudimg-amd64.img.raw (100.00/100%)
real 3m53.923s user 0m13.751s sys 0m15.346s
Now this is not something I'm personally concerned about (since I've long been arguing we should deprecate the qemu curl driver and use nbdkit), but it's also very surprising.
I'm also not surprised if qemu's curl driver is not efficient, because it does not seem to be a case frequently used. Offloading it to other paths, like nbdkit, is indeed a good goal. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org