On Sat, Jan 23, 2021 at 06:38:10AM +0000, Richard W.M. Jones wrote:
On Sat, Jan 23, 2021 at 12:45:22AM +0200, Nir Soffer wrote:
Use multiple connections to imageio server to speed up the transfer.
Connections are managed via a thread safe queue. Threads remove a connection from the queue for every request, and put it back when at the end of the request. Only one thread can access the connection at the same time.
Threads are accessing existing values in the handle dict, like h["path"]. They may also modify h["failed"] on errors. These operations are thread safe and do not require additional locking.
Sending flush request is more tricky; on imageio side, we have one qemu-nbd server, with multiple connections. I'm not sure if sending one flush command on one of the connections is good enough to flush all commands, so we send flush command on all connections in the flush callback.
I know the answer to this! It depends if the NBD server advertises multi-conn or not. With libnbd you can find out by querying nbd_can_multi_conn on any of the connections (if the server is behaving, the answer should be identical for any connection with the same exportname). See:
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md
"bit 8, NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to the client. In the absence of this flag, clients SHOULD NOT multiplex their commands over more than one connection to the export."
For unclear reasons qemu-nbd only advertises multi-conn for r/o connections, assuming my reading of the code is correct. For nbdkit we went through the plugins a long time ago and made them advertise (or not) multi-conn as appropriate.
$ truncate -s 10M /tmp/disk.img $ rm -f /tmp/sock $ qemu-nbd -k /tmp/sock --format=raw /tmp/disk.img & pid=$! $ nbdinfo --json 'nbd+unix://?socket=/tmp/sock' | jq '.exports[0].can_multi_conn' false $ kill $pid Adding the --read-only option doesn't change it: false Adding --shared=2 --read-only: true For comparison: $ nbdkit file /tmp/disk.img --run 'nbdinfo --json "$uri"' | jq '.exports[0].can_multi_conn' true $ nbdkit memory 1G --run 'nbdinfo --json "$uri"' | jq '.exports[0].can_multi_conn' true etc. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/