7:34 a.m.
Fix the test using qemu-nbd to allow multiple connections and update the
man page about the new semantics.
Unfortunately we have no way to tell if a NBD server supports multiple
connetions, so our default of using 4 connetions is fragile when using
remote qemu-nbd. I think the default should change to use one connection
so it always works.
Nir Soffer (3):
copy: copy-file-to-qcow2.sh: Enable multiple connections
Revert "Revert "copy: Always allow multiple connections""
copy: Update nbdcopy(1) about multiple connections
copy/copy-file-to-qcow2.sh | 2 +-
copy/main.c | 25 ++++++++++++++++---------
copy/nbdcopy.pod | 25 +++++++------------------
3 files changed, 24 insertions(+), 28 deletions(-)
--
2.26.3
7:34 a.m.
New subject: [PATCH libnbd 1/3] copy: copy-file-to-qcow2.sh: Enable multiple connections
With --shared=4 qemu-nbd allows 4 concurrent connections. Without this
only the first connection succeed and the rest will block forever.
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
copy/copy-file-to-qcow2.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/copy/copy-file-to-qcow2.sh b/copy/copy-file-to-qcow2.sh
index 57c18b3..ae41ed0 100755
--- a/copy/copy-file-to-qcow2.sh
+++ b/copy/copy-file-to-qcow2.sh
@@ -54,7 +54,7 @@ qemu-img create -f qcow2 $qcow2 $size
# Run qemu-nbd as a separate process so that we can copy to and from
# the single process in two separate operations.
-qemu-nbd -f qcow2 -t --socket=$sock --pid-file=$pidfile $qcow2 &
+qemu-nbd -f qcow2 -t --shared=4 --socket=$sock --pid-file=$pidfile $qcow2 &
cleanup_fn kill $!
# Wait for qemu-nbd to start up.
--
2.26.3
7:34 a.m.
New subject: [PATCH libnbd 2/3] Revert "Revert "copy: Always allow multiple connections""
Enable again multiple connections. When using qemu-nbd users need to
use:
qemu-nbd --shared=4 ...
Otherwise nbdcopy will deadlock trying to open 4 connections to
qemu-nbd.
This reverts commit fb42056929e9177e0de2603e2f430a8111ae7a5f.
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
copy/main.c | 25 ++++++++++++++++---------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/copy/main.c b/copy/main.c
index b9dbe1d..70a4e11 100644
--- a/copy/main.c
+++ b/copy/main.c
@@ -321,10 +321,6 @@ main (int argc, char *argv[])
exit (EXIT_FAILURE);
}
- /* If multi-conn is not supported, force connections to 1. */
- if (! src->ops->can_multi_conn (src) || ! dst->ops->can_multi_conn (dst))
- connections = 1;
-
/* Calculate the number of threads from the number of connections. */
if (threads == 0) {
long t;
@@ -379,15 +375,26 @@ main (int argc, char *argv[])
synchronous ? "true" : "false");
}
- /* If multi-conn is enabled on either side, then at this point we
+ /* If using multiple connections, then at this point we
* need to ask the backend to open the extra connections.
+ *
+ * Using multiple connections when the server does not report
+ * can_multi_conn is not safe in general, but is safe for the special
+ * access pattern in nbdcopy.
+ *
+ * - We split the image to 128m segments, and every segment is written
+ * by single worker.
+ *
+ * - Writes do not overlap, and are never partial block that another
+ * worker may modify at the same time.
+ *
+ * - We never read from the destination so we don't have caching
+ * synchronization issue between clients.
*/
if (connections > 1) {
assert (threads == connections);
- if (src->ops->can_multi_conn (src))
- src->ops->start_multi_conn (src);
- if (dst->ops->can_multi_conn (dst))
- dst->ops->start_multi_conn (dst);
+ src->ops->start_multi_conn (src);
+ dst->ops->start_multi_conn (dst);
}
/* If the source is NBD and we couldn't negotiate meta
--
2.26.3
7:34 a.m.
New subject: [PATCH libnbd 3/3] copy: Update nbdcopy(1) about multiple connections
Using multiple connections does not depend now on NBD multi-conn flag.
Add a note about using --shared=4 in qemu-nbd to allow multiple
connections.
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
copy/nbdcopy.pod | 25 +++++++------------------
1 file changed, 7 insertions(+), 18 deletions(-)
diff --git a/copy/nbdcopy.pod b/copy/nbdcopy.pod
index f710093..781bb10 100644
--- a/copy/nbdcopy.pod
+++ b/copy/nbdcopy.pod
@@ -103,10 +103,9 @@ output on the destination.
=item B<--connections=>N
-Set the maximum number of NBD connections ("multi-conn"). By default
-nbdcopy will try to use multi-conn with up to 4 connections if the NBD
-server supports it. If copying between NBD servers then nbdcopy
-cannot use multi-conn if either of the servers does not support it.
+Set the maximum number of NBD connections. By default nbdcopy will try
+to use multiple connections with up to 4 connections. Some servers
+like L<qemu-nbd(8)> support only one connection by default.
=item B<--destination-is-zero>
@@ -216,23 +215,13 @@ pipe, or a socket, or if you use the I<--synchronous> option,
then
nbdcopy works in synchronous mode with no parallelism, and nothing
else in this section applies.
-The I<--connections=N> option controls NBD multi-conn (see
-L<libnbd(3)/Multi-conn>), opening up to N connections to the NBD
+The I<--connections=N> option open up to N connections to the NBD
server (or to both NBD servers if copying between NBD servers). This
-defaults to 4. The NBD servers must support and advertise multi-conn.
-For L<nbdkit(1)> availability of multi-conn can depend on the plugin.
-You can use L<nbdinfo(1)> to find out if a particular NBD server is
-advertising multi-conn. If the NBD server doesn’t advertise
-multi-conn then only one connection will be opened regardless of the
-I<--connections> flag.
-
-When copying between two NBD servers, the number of connections is
-limited to the minimum multi-conn supported on both sides. For the
-purposes of this calculation, you can consider local files and block
-devices as supporting infinite multi-conn.
+defaults to 4. When using L<qemu-nbd(8)> you need to enable multiple
+conections using the I<--shared=4> command line option.
When you run an NBD server as a subprocess (using the S<C<[ ... ]>>
-syntax) multi-conn cannot be used.
+syntax) multiple connections cannot be used.
The I<--threads=N> option allows nbdcopy to start up to N threads
(defaulting to the number of cores). However nbdcopy cannot use more
--
2.26.3
8:07 a.m.
New subject: [PATCH libnbd 3/3] copy: Update nbdcopy(1) about multiple connections
I want to see what Eric has to say about whether we can or should
change qemu-nbd.
There's a small typo in this patch:
@@ -216,23 +215,13 @@ pipe, or a socket, or if you use the
I<--synchronous> option, then
nbdcopy works in synchronous mode with no parallelism, and nothing
else in this section applies.
-The I<--connections=N> option controls NBD multi-conn (see
-L<libnbd(3)/Multi-conn>), opening up to N connections to the NBD
+The I<--connections=N> option open up to N connections to the NBD
s/open/opens/
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
1:02 p.m.
New subject: [PATCH libnbd 3/3] copy: Update nbdcopy(1) about multiple connections
On Thu, May 27, 2021 at 02:07:19PM +0100, Richard W.M. Jones wrote:
I want to see what Eric has to say about whether we can or should
change qemu-nbd.
Changing the default to allow multiple writers may risky; I'm worried
we may be stuck with always having to opt-in to it.
On the other hand, if qemu-nbd advertises multi-conn only when
--shared=N for N>1, we'd have a nice witness for when we can use
multi-conn (but only after qemu 6.1 or downstream backports of this
as-yet unwritten patch).
Other ideas:
Add an NBD_OPT extension to allow a client to inform the server of its
intent to perform non-overlapping parallel I/O. If the server
understands the new option, it can then proceed to allow additional
connections (that is, make qemu-nbd dynamically turn on --shared=N
based on a new-enough client requesting upload mode). Obviously
wouldn't work until qemu 6.1 or downstream backports of the new feature.
Add an NBD_OPT extension to allow a client to learn from the server
how many connections it actually accepts (supporting a number > 1 even
without multi-conn is viable). If the server answers, then use that
many connections; if not, then stick to 1 connection unless multi-conn
is set. Again, something that won't work out of the box until qemu
6.1 or downstream backports of the new feature.
At any rate, looks like I need to accelerate my patches to have
qemu-nbd advertise multi-conn for --shared=N writeable servers (where
qemu itself should already be cache-consistent because it has proper
serialization of any overlapping requests, regardless of which client
is making the request).
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
5:47 p.m.
New subject: [PATCH libnbd 3/3] copy: Update nbdcopy(1) about multiple connections
נשלח מה-iPhone שלי
ב-27 במאי 2021, בשעה 21:02, Eric Blake <eblake(a)redhat.com>
כתב/ה:
On Thu, May 27, 2021 at 02:07:19PM +0100, Richard W.M. Jones wrote:
> I want to see what Eric has to say about whether we can or should
> change qemu-nbd.
Changing the default to allow multiple writers may risky; I'm worried
we may be stuck with always having to opt-in to it.
Maybe it is better to avoid promising any default behavior. For example we can say that
“nbdcopy may use multiple connections”.
On the other hand, if qemu-nbd advertises multi-conn only when
--shared=N for N>1, we'd have a nice witness for when we can use
multi-conn (but only after qemu 6.1 or downstream backports of this
as-yet unwritten patch).
This seems to be the best option. It seems that emu-nbd already ready for this.
Other ideas:
Add an NBD_OPT extension to allow a client to inform the server of its
intent to perform non-overlapping parallel I/O. If the server
understands the new option, it can then proceed to allow additional
connections
This sound too complicated for the common user.
(that is, make qemu-nbd dynamically turn on --shared=N
based on a new-enough client requesting upload mode). Obviously
wouldn't work until qemu 6.1 or downstream backports of the new feature.
Add an NBD_OPT extension to allow a client to learn from the server
how many connections it actually accepts (supporting a number > 1 even
without multi-conn is viable).
I think this is needed anyway. Without this, if a remote qemu-nbd was started with
—shared=2 nbdcopy will deadlock trying to connect 4 times.
We can change nbdcopy to avoid the deadlock but using server limit would be much simpler
and will help all nbd clients.
If we return more detailed info about multiple connections, we can also report the
“isolation” level supported by the server. For example qemu can report “full” since it
does proper locking. Server without such locking can report that only non-overlapping
writes are supported (“none”?).
We probably need to take this to nbd mailing list.
If the server answers, then use that
many connections; if not, then stick to 1 connection unless multi-conn
is set. Again, something that won't work out of the box until qemu
6.1 or downstream backports of the new feature.
At any rate, looks like I need to accelerate my patches to have
qemu-nbd advertise multi-conn for --shared=N writeable servers (where
qemu itself should already be cache-consistent because it has proper
serialization of any overlapping requests, regardless of which client
is making the request).
Do you need any help to make this happen?
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
1087
days inactive
1089
days old
6 comments
4 participants
participants (4)
-
Eric Blake -
Nir Soffer -
Nir Soffer -
Richard W.M. Jones