Re: [Libguestfs] Two multi-conn questions
On Thu, Feb 02, 2023 at 04:26:04PM +0000, Richard W.M. Jones wrote:
...
> > $ time nbdkit -r curl
https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64...
--filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents
-p $uri jammy-server-cloudimg-amd64.img '
>
> Yay - I'm glad my multi-conn filter makes it easier to test things
> like this!
>
> Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention
> that despite multi-conn-mode=unsafe being unsafe for a plugin that
> does not have consistency, it is useful for timing tests on a plugin
> where we suspect consistency is available in order to test timing to
> see if it actually makes a difference?
Yes, sounds good.
How about:
diff --git i/filters/multi-conn/nbdkit-multi-conn-filter.pod
w/filters/multi-conn/nbdkit-multi-conn-filter.pod
index 87b31692..7f70ade9 100644
--- i/filters/multi-conn/nbdkit-multi-conn-filter.pod
+++ w/filters/multi-conn/nbdkit-multi-conn-filter.pod
@@ -118,7 +118,11 @@ passed on to the plugin.
When B<unsafe> mode is chosen, this filter blindly advertises
multi-conn to the client even if the plugin lacks support. This is
dangerous, and risks data corruption if the client makes assumptions
-about flush consistency that were not actually met.
+about flush consistency that were not actually met. However, for a
+plugin that does not yet advertise multi-conn, but where it is
+suspected that the plugin behaves consistently, this is a great way to
+run timing and accuracy tests to see enabling multi-conn in the plugin
+will make a difference.
=item B<multi-conn-track-dirty=fast>
> > Is there any work on adding multi-conn support to
qemu's NBD client?
>
> Not that I'm aware of at the moment, but we have proof that it may
> prove fruitful to have someone spend time on.
Kubernetes team are complaining that this particular case (stream
qcow2 file from a website and convert it to raw without a temporary
file) is slow. This is not a case that I've encountered before
because it's not relevant to virt-v2v, but it does appear to be
important.
- * - * -
I didn't want to complicate the original message with irrelevant
stuff, but there's something else I want to mention now. If we just
use qemu's curl driver (thus eliminating NBD & nbdkit from the mix),
it gets even slower:
$ time ~/d/qemu/build/qemu-img convert -p -W -f qcow2 'json:{
"file.readahead": 67108864, "file.driver": "http",
"file.url":
"http://oirase.annexia.org/tmp/jammy-server-cloudimg-amd64.qcow2",
"file.timeout":2000 }' -O raw jammy-server-cloudimg-amd64.img.raw
(100.00/100%)
real 3m53.923s
user 0m13.751s
sys 0m15.346s
Now this is not something I'm personally concerned about (since I've
long been arguing we should deprecate the qemu curl driver and use
nbdkit), but it's also very surprising.
I'm also not surprised if qemu's curl driver is not efficient, because
it does not seem to be a case frequently used. Offloading it to other
paths, like nbdkit, is indeed a good goal.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org