Two multi-conn questions
by Richard W.M. Jones
Hi Eric,
A couple of semi-related multi-conn questions.
(1) nbdkit-curl-plugin does not currently advertise multi-conn. However
forcing multi-conn makes it faster, eg:
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64... timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '
█ 100% [****************************************]
real 1m15.438s
user 0m2.513s
sys 0m4.063s
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img '
█ 100% [****************************************]
real 0m59.635s
user 0m3.010s
sys 0m4.538s
(Note that this is line speed - wget also takes 59 seconds)
I believe that multi-conn should be safe to enable in the curl plugin,
certainly if we're in read-only mode (which is the most common case
for this plugin). It's probably not safe when operating in read-write
mode, because HTTP doesn't have any concept of flushing, but almost no
web server supports writing.
(2) qemu's NBD client still doesn't seem to support multi-conn. This
makes qemu-img convert very slow when reading from nbdkit, as in the
example below. (Note the '.img' file on that website is actually a
qcow2 file.)
$ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64... --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' qemu-img convert -W -m 16 -p -f qcow2 $uri -O raw jammy-server-cloudimg-amd64.img.raw '
(100.00/100%)
real 3m29.412s
user 0m17.856s
sys 0m8.218s
Since we know from (1) above that copying could go at line speed when
multi-conn is enabled, and also that qemu-img convert from the same
local file takes only about 2 seconds, it seems like qemu-img convert
above could go a lot faster if we allowed multi-conn.
Is there any work on adding multi-conn support to qemu's NBD client?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
1 year, 3 months
[p2v PATCH v2 00/11] Expose virt-v2v's "-oo"; re-enable openstack
by Laszlo Ersek
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1792141
V1 was at
<https://listman.redhat.com/archives/libguestfs/2023-January/030539.html>
(Message-Id: <20230130142228.108135-1-lersek(a)redhat.com>).
In v2, I've picked up Rich's R-b's for all patches except #9 and #10.
Those have been updated; please see the Notes sections on them.
Laszlo
Alban Lecorps (1):
Introduce "p2v.output.misc" for passing "-oo" options to virt-v2v
Laszlo Ersek (10):
test-virt-p2v-cmdline: turn option list into a shell array
guestfs-utils: import guestfs_int_join_strings()
gui: factor out entry_text_dup()
gui: factor out tgl_btn_is_act()
gui: flatten get_phys_topo_from_conv_dlg()
gui: wrap overlong function declaration lines
gui: wrap string literals
gui: expose "p2v.output.misc" (-oo)
Reenable the OpenStack output mode
make-kickstart: add URLs for RHEL-9
generate-p2v-config.pl | 10 ++
libguestfs/guestfs-utils.h | 1 +
conversion.c | 9 ++
gui.c | 170 ++++++++++++++++----
libguestfs/guestfs-utils.c | 35 ++++
ssh.c | 5 +-
test-virt-p2v-cmdline.sh | 24 ++-
virt-p2v-make-kickstart.in | 13 ++
virt-p2v.pod | 11 ++
9 files changed, 241 insertions(+), 37 deletions(-)
1 year, 3 months
[PATCH nbdkit] curl: Enable multi-conn for read-only connections
by Richard W.M. Jones
Multi-conn is enabled only when we know the connection is read-only:
$ ./nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn
can_multi_conn: true
$ ./nbdkit curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn
can_multi_conn: false
This improves performance.
Comparing before and after this commit shows approximate doubling of
performance:
Benchmark 1: nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:"
Time (mean ± σ): 943.8 ms ± 18.8 ms [User: 316.2 ms, System: 1029.7 ms]
Range (min … max): 923.7 ms … 989.2 ms 10 runs
Benchmark 2: ~/d/nbdkit/nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:"
Time (mean ± σ): 455.0 ms ± 6.2 ms [User: 542.2 ms, System: 1824.7 ms]
Range (min … max): 449.1 ms … 471.6 ms 10 runs
Summary
' ~/d/nbdkit/nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" ' ran
2.07 ± 0.05 times faster than ' nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" '
See also:
https://listman.redhat.com/archives/libguestfs/2023-February/030581.html
---
plugins/curl/curldefs.h | 1 +
plugins/curl/curl.c | 14 ++++++++++++++
2 files changed, 15 insertions(+)
diff --git a/plugins/curl/curldefs.h b/plugins/curl/curldefs.h
index a2c521c50..16b6071cc 100644
--- a/plugins/curl/curldefs.h
+++ b/plugins/curl/curldefs.h
@@ -68,6 +68,7 @@ extern const char *user_agent;
/* The per-connection handle. */
struct curl_handle {
CURL *c;
+ int readonly;
bool accept_range;
int64_t exportsize;
char errbuf[CURL_ERROR_SIZE];
diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c
index 42aba4216..2f34d1dc1 100644
--- a/plugins/curl/curl.c
+++ b/plugins/curl/curl.c
@@ -465,6 +465,7 @@ curl_open (int readonly)
nbdkit_error ("calloc: %m");
return NULL;
}
+ h->readonly = readonly;
h->c = curl_easy_init ();
if (h->c == NULL) {
@@ -781,6 +782,18 @@ curl_get_size (void *handle)
return h->exportsize;
}
+/* Multi-conn is safe for read-only connections, but HTTP does not
+ * have any concept of flushing so we cannot use it for read-write
+ * connections.
+ */
+static int
+curl_can_multi_conn (void *handle)
+{
+ struct curl_handle *h = handle;
+
+ return !! h->readonly;
+}
+
/* NB: The terminology used by libcurl is confusing!
*
* WRITEFUNCTION / write_cb is used when reading from the remote server
@@ -924,6 +937,7 @@ static struct nbdkit_plugin plugin = {
.open = curl_open,
.close = curl_close,
.get_size = curl_get_size,
+ .can_multi_conn = curl_can_multi_conn,
.pread = curl_pread,
.pwrite = curl_pwrite,
};
--
2.39.0
1 year, 3 months
[p2v PATCH 00/11] Expose virt-v2v's "-oo"; re-enable openstack
by Laszlo Ersek
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1792141
Let the user pass "-oo" options from the kernel cmdline and from the GUI
to virt-v2v. This is primarily useful with the OpenStack output mode,
so reenable that mode.
Cc: Alban Lecorps <alban.lecorps(a)ubisoft.com>
Laszlo
Alban Lecorps (1):
Introduce "p2v.output.misc" for passing "-oo" options to virt-v2v
Laszlo Ersek (10):
test-virt-p2v-cmdline: turn option list into a shell array
guestfs-utils: import guestfs_int_join_strings()
gui: factor out entry_text_dup()
gui: factor out tgl_btn_is_act()
gui: flatten get_phys_topo_from_conv_dlg()
gui: wrap overlong function declaration lines
gui: wrap string literals
gui: expose "p2v.output.misc" (-oo)
Reenable the OpenStack output mode
make-kickstart: add URLs for RHEL-9
conversion.c | 9 ++
generate-p2v-config.pl | 10 ++
gui.c | 168 ++++++++++++++++----
libguestfs/guestfs-utils.c | 35 ++++
libguestfs/guestfs-utils.h | 1 +
ssh.c | 5 +-
test-virt-p2v-cmdline.sh | 24 ++-
virt-p2v-make-kickstart.in | 13 ++
virt-p2v.pod | 2 +
9 files changed, 230 insertions(+), 37 deletions(-)
1 year, 3 months