Re: [Libguestfs] Two multi-conn questions

On Wed, Feb 01, 2023 at 12:40:45PM +0000, Richard W.M. Jones wrote: > Hi Eric, > > A couple of semi-related multi-conn questions. > > (1) nbdkit-curl-plugin does not currently advertise multi-conn. However > forcing multi-conn makes it faster, eg: > > $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64... timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' > █ 100% [****************************************] > > real 1m15.438s > user 0m2.513s > sys 0m4.063s I'm jealous of your connection. Even with bare wget (without nbdkit in the middle), for some reason my computer is failing to connect when trying IPv6. I then tried 'wget -4 ...', and didn't get the full file until 5m37s. And when I tried to reproduce your nbdkit line, I got: nbdkit: curl[1]: error: pread: curl_easy_perform: Failed sending data to the peer: Connection died, tried 5 times before giving up nbdcopy: read at offset 245104640 failed: Input/output error at 6 minutes in. So slow speeds and some connection flakiness is not helping me.

> $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' Yay - I'm glad my multi-conn filter makes it easier to test things like this! Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention that despite multi-conn-mode=unsafe being unsafe for a plugin that does not have consistency, it is useful for timing tests on a plugin where we suspect consistency is available in order to test timing to see if it actually makes a difference?

> █ 100% [****************************************] > > real 0m59.635s > user 0m3.010s > sys 0m4.538s > > (Note that this is line speed - wget also takes 59 seconds) > > I believe that multi-conn should be safe to enable in the curl plugin, > certainly if we're in read-only mode (which is the most common case > for this plugin). It's probably not safe when operating in read-write > mode, because HTTP doesn't have any concept of flushing, but almost no > web server supports writing. Agreed on both fronts; I see you have proposed this patch (and I acked it) in the meantime. > > (2) qemu's NBD client still doesn't seem to support multi-conn. This > makes qemu-img convert very slow when reading from nbdkit, as in the > example below. (Note the '.img' file on that website is actually a > qcow2 file.) Correct - qemu does NOT do multi-queue (or multipath) client connections. It is blindly using exactly one NBD client connection per block device, and ignoring the server's advertisement of multi-conn support. Teaching qemu to support a multipath paradigm is not a trivial task. I'm wondering if the effort might be easier by teaching libblkio how to do arbitrary multipath for any underlying block device, coupled with current efforts to teach qemu how to target any arbitrary libblkio target, so that qemu proper doesn't have to worry about the intricacies of multipath; from there, we'd need a way for libblkio to target NBD devices (probably by libnbd). In other words, instead of trying to directly code client MULTI_CONN efforts into qemu proper, it may be better to segregate things into more modular building blocks. > > $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' qemu-img convert -W -m 16 -p -f qcow2 $uri -O raw jammy-server-cloudimg-amd64.img.raw ' > (100.00/100%) > > real 3m29.412s > user 0m17.856s > sys 0m8.218s > > Since we know from (1) above that copying could go at line speed when > multi-conn is enabled, and also that qemu-img convert from the same > local file takes only about 2 seconds, it seems like qemu-img convert > above could go a lot faster if we allowed multi-conn. > > Is there any work on adding multi-conn support to qemu's NBD client? Not that I'm aware of at the moment, but we have proof that it may prove fruitful to have someone spend time on.