Re: [Libguestfs] Two multi-conn questions

On Thu, Feb 02, 2023 at 04:26:04PM +0000, Richard W.M. Jones wrote: ... > > > $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' > > > > Yay - I'm glad my multi-conn filter makes it easier to test things > > like this! > > > > Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention > > that despite multi-conn-mode=unsafe being unsafe for a plugin that > > does not have consistency, it is useful for timing tests on a plugin > > where we suspect consistency is available in order to test timing to > > see if it actually makes a difference? > > Yes, sounds good. How about: diff --git i/filters/multi-conn/nbdkit-multi-conn-filter.pod w/filters/multi-conn/nbdkit-multi-conn-filter.pod index 87b31692..7f70ade9 100644 --- i/filters/multi-conn/nbdkit-multi-conn-filter.pod +++ w/filters/multi-conn/nbdkit-multi-conn-filter.pod @@ -118,7 +118,11 @@ passed on to the plugin. When B<unsafe> mode is chosen, this filter blindly advertises multi-conn to the client even if the plugin lacks support. This is dangerous, and risks data corruption if the client makes assumptions -about flush consistency that were not actually met. +about flush consistency that were not actually met. However, for a +plugin that does not yet advertise multi-conn, but where it is +suspected that the plugin behaves consistently, this is a great way to +run timing and accuracy tests to see enabling multi-conn in the plugin

+will make a difference.

=item B<multi-conn-track-dirty=fast> > > > Is there any work on adding multi-conn support to qemu's NBD client? > > > > Not that I'm aware of at the moment, but we have proof that it may > > prove fruitful to have someone spend time on. > > Kubernetes team are complaining that this particular case (stream > qcow2 file from a website and convert it to raw without a temporary > file) is slow. This is not a case that I've encountered before > because it's not relevant to virt-v2v, but it does appear to be > important. > > - * - * - > > I didn't want to complicate the original message with irrelevant > stuff, but there's something else I want to mention now. If we just > use qemu's curl driver (thus eliminating NBD & nbdkit from the mix), > it gets even slower: > > $ time ~/d/qemu/build/qemu-img convert -p -W -f qcow2 'json:{ "file.readahead": 67108864, "file.driver": "http", "file.url": "http://oirase.annexia.org/tmp/jammy-server-cloudimg-amd64.qcow2", "file.timeout":2000 }' -O raw jammy-server-cloudimg-amd64.img.raw > (100.00/100%) > > real 3m53.923s > user 0m13.751s > sys 0m15.346s > > Now this is not something I'm personally concerned about (since I've > long been arguing we should deprecate the qemu curl driver and use > nbdkit), but it's also very surprising. I'm also not surprised if qemu's curl driver is not efficient, because it does not seem to be a case frequently used. Offloading it to other paths, like nbdkit, is indeed a good goal.