[PATCH libnbd 0/3] Enable multiple connections - take 2
Fix the test using qemu-nbd to allow multiple connections and update the man page about the new semantics. Unfortunately we have no way to tell if a NBD server supports multiple connetions, so our default of using 4 connetions is fragile when using remote qemu-nbd. I think the default should change to use one connection so it always works. Nir Soffer (3): copy: copy-file-to-qcow2.sh: Enable multiple connections Revert "Revert "copy: Always allow multiple connections"" copy: Update nbdcopy(1) about multiple connections copy/copy-file-to-qcow2.sh | 2 +- copy/main.c | 25 ++++++++++++++++--------- copy/nbdcopy.pod | 25 +++++++------------------ 3 files changed, 24 insertions(+), 28 deletions(-) -- 2.26.3
With --shared=4 qemu-nbd allows 4 concurrent connections. Without this only the first connection succeed and the rest will block forever. Signed-off-by: Nir Soffer <nsoffer@redhat.com> --- copy/copy-file-to-qcow2.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/copy/copy-file-to-qcow2.sh b/copy/copy-file-to-qcow2.sh index 57c18b3..ae41ed0 100755 --- a/copy/copy-file-to-qcow2.sh +++ b/copy/copy-file-to-qcow2.sh @@ -54,7 +54,7 @@ qemu-img create -f qcow2 $qcow2 $size # Run qemu-nbd as a separate process so that we can copy to and from # the single process in two separate operations. -qemu-nbd -f qcow2 -t --socket=$sock --pid-file=$pidfile $qcow2 & +qemu-nbd -f qcow2 -t --shared=4 --socket=$sock --pid-file=$pidfile $qcow2 & cleanup_fn kill $! # Wait for qemu-nbd to start up. -- 2.26.3
Enable again multiple connections. When using qemu-nbd users need to use: qemu-nbd --shared=4 ... Otherwise nbdcopy will deadlock trying to open 4 connections to qemu-nbd. This reverts commit fb42056929e9177e0de2603e2f430a8111ae7a5f. Signed-off-by: Nir Soffer <nsoffer@redhat.com> --- copy/main.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/copy/main.c b/copy/main.c index b9dbe1d..70a4e11 100644 --- a/copy/main.c +++ b/copy/main.c @@ -321,10 +321,6 @@ main (int argc, char *argv[]) exit (EXIT_FAILURE); } - /* If multi-conn is not supported, force connections to 1. */ - if (! src->ops->can_multi_conn (src) || ! dst->ops->can_multi_conn (dst)) - connections = 1; - /* Calculate the number of threads from the number of connections. */ if (threads == 0) { long t; @@ -379,15 +375,26 @@ main (int argc, char *argv[]) synchronous ? "true" : "false"); } - /* If multi-conn is enabled on either side, then at this point we + /* If using multiple connections, then at this point we * need to ask the backend to open the extra connections. + * + * Using multiple connections when the server does not report + * can_multi_conn is not safe in general, but is safe for the special + * access pattern in nbdcopy. + * + * - We split the image to 128m segments, and every segment is written + * by single worker. + * + * - Writes do not overlap, and are never partial block that another + * worker may modify at the same time. + * + * - We never read from the destination so we don't have caching + * synchronization issue between clients. */ if (connections > 1) { assert (threads == connections); - if (src->ops->can_multi_conn (src)) - src->ops->start_multi_conn (src); - if (dst->ops->can_multi_conn (dst)) - dst->ops->start_multi_conn (dst); + src->ops->start_multi_conn (src); + dst->ops->start_multi_conn (dst); } /* If the source is NBD and we couldn't negotiate meta -- 2.26.3
Using multiple connections does not depend now on NBD multi-conn flag. Add a note about using --shared=4 in qemu-nbd to allow multiple connections. Signed-off-by: Nir Soffer <nsoffer@redhat.com> --- copy/nbdcopy.pod | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/copy/nbdcopy.pod b/copy/nbdcopy.pod index f710093..781bb10 100644 --- a/copy/nbdcopy.pod +++ b/copy/nbdcopy.pod @@ -103,10 +103,9 @@ output on the destination. =item B<--connections=>N -Set the maximum number of NBD connections ("multi-conn"). By default -nbdcopy will try to use multi-conn with up to 4 connections if the NBD -server supports it. If copying between NBD servers then nbdcopy -cannot use multi-conn if either of the servers does not support it. +Set the maximum number of NBD connections. By default nbdcopy will try +to use multiple connections with up to 4 connections. Some servers +like L<qemu-nbd(8)> support only one connection by default. =item B<--destination-is-zero> @@ -216,23 +215,13 @@ pipe, or a socket, or if you use the I<--synchronous> option, then nbdcopy works in synchronous mode with no parallelism, and nothing else in this section applies. -The I<--connections=N> option controls NBD multi-conn (see -L<libnbd(3)/Multi-conn>), opening up to N connections to the NBD +The I<--connections=N> option open up to N connections to the NBD server (or to both NBD servers if copying between NBD servers). This -defaults to 4. The NBD servers must support and advertise multi-conn. -For L<nbdkit(1)> availability of multi-conn can depend on the plugin. -You can use L<nbdinfo(1)> to find out if a particular NBD server is -advertising multi-conn. If the NBD server doesn’t advertise -multi-conn then only one connection will be opened regardless of the -I<--connections> flag. - -When copying between two NBD servers, the number of connections is -limited to the minimum multi-conn supported on both sides. For the -purposes of this calculation, you can consider local files and block -devices as supporting infinite multi-conn. +defaults to 4. When using L<qemu-nbd(8)> you need to enable multiple +conections using the I<--shared=4> command line option. When you run an NBD server as a subprocess (using the S<C<[ ... ]>> -syntax) multi-conn cannot be used. +syntax) multiple connections cannot be used. The I<--threads=N> option allows nbdcopy to start up to N threads (defaulting to the number of cores). However nbdcopy cannot use more -- 2.26.3
I want to see what Eric has to say about whether we can or should change qemu-nbd. There's a small typo in this patch:
@@ -216,23 +215,13 @@ pipe, or a socket, or if you use the I<--synchronous> option, then nbdcopy works in synchronous mode with no parallelism, and nothing else in this section applies.
-The I<--connections=N> option controls NBD multi-conn (see -L<libnbd(3)/Multi-conn>), opening up to N connections to the NBD +The I<--connections=N> option open up to N connections to the NBD
s/open/opens/ Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Thu, May 27, 2021 at 02:07:19PM +0100, Richard W.M. Jones wrote:
I want to see what Eric has to say about whether we can or should change qemu-nbd.
Changing the default to allow multiple writers may risky; I'm worried we may be stuck with always having to opt-in to it. On the other hand, if qemu-nbd advertises multi-conn only when --shared=N for N>1, we'd have a nice witness for when we can use multi-conn (but only after qemu 6.1 or downstream backports of this as-yet unwritten patch). Other ideas: Add an NBD_OPT extension to allow a client to inform the server of its intent to perform non-overlapping parallel I/O. If the server understands the new option, it can then proceed to allow additional connections (that is, make qemu-nbd dynamically turn on --shared=N based on a new-enough client requesting upload mode). Obviously wouldn't work until qemu 6.1 or downstream backports of the new feature. Add an NBD_OPT extension to allow a client to learn from the server how many connections it actually accepts (supporting a number > 1 even without multi-conn is viable). If the server answers, then use that many connections; if not, then stick to 1 connection unless multi-conn is set. Again, something that won't work out of the box until qemu 6.1 or downstream backports of the new feature. At any rate, looks like I need to accelerate my patches to have qemu-nbd advertise multi-conn for --shared=N writeable servers (where qemu itself should already be cache-consistent because it has proper serialization of any overlapping requests, regardless of which client is making the request). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
נשלח מה-iPhone שלי
ב-27 במאי 2021, בשעה 21:02, Eric Blake <eblake@redhat.com> כתב/ה:
On Thu, May 27, 2021 at 02:07:19PM +0100, Richard W.M. Jones wrote:
I want to see what Eric has to say about whether we can or should change qemu-nbd.
Changing the default to allow multiple writers may risky; I'm worried we may be stuck with always having to opt-in to it.
Maybe it is better to avoid promising any default behavior. For example we can say that “nbdcopy may use multiple connections”.
On the other hand, if qemu-nbd advertises multi-conn only when --shared=N for N>1, we'd have a nice witness for when we can use multi-conn (but only after qemu 6.1 or downstream backports of this as-yet unwritten patch).
This seems to be the best option. It seems that emu-nbd already ready for this.
Other ideas:
Add an NBD_OPT extension to allow a client to inform the server of its intent to perform non-overlapping parallel I/O. If the server understands the new option, it can then proceed to allow additional connections
This sound too complicated for the common user.
(that is, make qemu-nbd dynamically turn on --shared=N based on a new-enough client requesting upload mode). Obviously wouldn't work until qemu 6.1 or downstream backports of the new feature.
Add an NBD_OPT extension to allow a client to learn from the server how many connections it actually accepts (supporting a number > 1 even without multi-conn is viable).
I think this is needed anyway. Without this, if a remote qemu-nbd was started with —shared=2 nbdcopy will deadlock trying to connect 4 times. We can change nbdcopy to avoid the deadlock but using server limit would be much simpler and will help all nbd clients. If we return more detailed info about multiple connections, we can also report the “isolation” level supported by the server. For example qemu can report “full” since it does proper locking. Server without such locking can report that only non-overlapping writes are supported (“none”?). We probably need to take this to nbd mailing list.
If the server answers, then use that many connections; if not, then stick to 1 connection unless multi-conn is set. Again, something that won't work out of the box until qemu 6.1 or downstream backports of the new feature.
At any rate, looks like I need to accelerate my patches to have qemu-nbd advertise multi-conn for --shared=N writeable servers (where qemu itself should already be cache-consistent because it has proper serialization of any overlapping requests, regardless of which client is making the request).
Do you need any help to make this happen?
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
participants (4)
-
Eric Blake -
Nir Soffer -
Nir Soffer -
Richard W.M. Jones