February 2023 - Libguestfs - libguestfs List Archives

Two multi-conn questions
by Richard W.M. Jones 02 Feb '23

02 Feb '23

Hi Eric, A couple of semi-related multi-conn questions. (1) nbdkit-curl-plugin does not currently advertise multi-conn. However forcing multi-conn makes it faster, eg: $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.i… timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' █ 100% [****************************************] real 1m15.438s user 0m2.513s sys 0m4.063s $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.i… --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy --no-extents -p $uri jammy-server-cloudimg-amd64.img ' █ 100% [****************************************] real 0m59.635s user 0m3.010s sys 0m4.538s (Note that this is line speed - wget also takes 59 seconds) I believe that multi-conn should be safe to enable in the curl plugin, certainly if we're in read-only mode (which is the most common case for this plugin). It's probably not safe when operating in read-write mode, because HTTP doesn't have any concept of flushing, but almost no web server supports writing. (2) qemu's NBD client still doesn't seem to support multi-conn. This makes qemu-img convert very slow when reading from nbdkit, as in the example below. (Note the '.img' file on that website is actually a qcow2 file.) $ time nbdkit -r curl https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.i… --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' qemu-img convert -W -m 16 -p -f qcow2 $uri -O raw jammy-server-cloudimg-amd64.img.raw ' (100.00/100%) real 3m29.412s user 0m17.856s sys 0m8.218s Since we know from (1) above that copying could go at line speed when multi-conn is enabled, and also that qemu-img convert from the same local file takes only about 2 seconds, it seems like qemu-img convert above could go a lot faster if we allowed multi-conn. Is there any work on adding multi-conn support to qemu's NBD client? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

2 4

[p2v PATCH v2 00/11] Expose virt-v2v's "-oo"; re-enable openstack
by Laszlo Ersek 02 Feb '23

02 Feb '23

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1792141 V1 was at <https://listman.redhat.com/archives/libguestfs/2023-January/030539.html> (Message-Id: <20230130142228.108135-1-lersek(a)redhat.com>). In v2, I've picked up Rich's R-b's for all patches except #9 and #10. Those have been updated; please see the Notes sections on them. Laszlo Alban Lecorps (1): Introduce "p2v.output.misc" for passing "-oo" options to virt-v2v Laszlo Ersek (10): test-virt-p2v-cmdline: turn option list into a shell array guestfs-utils: import guestfs_int_join_strings() gui: factor out entry_text_dup() gui: factor out tgl_btn_is_act() gui: flatten get_phys_topo_from_conv_dlg() gui: wrap overlong function declaration lines gui: wrap string literals gui: expose "p2v.output.misc" (-oo) Reenable the OpenStack output mode make-kickstart: add URLs for RHEL-9 generate-p2v-config.pl | 10 ++ libguestfs/guestfs-utils.h | 1 + conversion.c | 9 ++ gui.c | 170 ++++++++++++++++---- libguestfs/guestfs-utils.c | 35 ++++ ssh.c | 5 +- test-virt-p2v-cmdline.sh | 24 ++- virt-p2v-make-kickstart.in | 13 ++ virt-p2v.pod | 11 ++ 9 files changed, 241 insertions(+), 37 deletions(-)

2 14

[PATCH nbdkit] curl: Enable multi-conn for read-only connections
by Richard W.M. Jones 02 Feb '23

02 Feb '23

Multi-conn is enabled only when we know the connection is read-only: $ ./nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: true $ ./nbdkit curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: false This improves performance. Comparing before and after this commit shows approximate doubling of performance: Benchmark 1: nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" Time (mean ± σ): 943.8 ms ± 18.8 ms [User: 316.2 ms, System: 1029.7 ms] Range (min … max): 923.7 ms … 989.2 ms 10 runs Benchmark 2: ~/d/nbdkit/nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" Time (mean ± σ): 455.0 ms ± 6.2 ms [User: 542.2 ms, System: 1824.7 ms] Range (min … max): 449.1 ms … 471.6 ms 10 runs Summary ' ~/d/nbdkit/nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" ' ran 2.07 ± 0.05 times faster than ' nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" ' See also: https://listman.redhat.com/archives/libguestfs/2023-February/030581.html --- plugins/curl/curldefs.h | 1 + plugins/curl/curl.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+) diff --git a/plugins/curl/curldefs.h b/plugins/curl/curldefs.h index a2c521c50..16b6071cc 100644 --- a/plugins/curl/curldefs.h +++ b/plugins/curl/curldefs.h @@ -68,6 +68,7 @@ extern const char *user_agent; /* The per-connection handle. */ struct curl_handle { CURL *c; + int readonly; bool accept_range; int64_t exportsize; char errbuf[CURL_ERROR_SIZE]; diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c index 42aba4216..2f34d1dc1 100644 --- a/plugins/curl/curl.c +++ b/plugins/curl/curl.c @@ -465,6 +465,7 @@ curl_open (int readonly) nbdkit_error ("calloc: %m"); return NULL; } + h->readonly = readonly; h->c = curl_easy_init (); if (h->c == NULL) { @@ -781,6 +782,18 @@ curl_get_size (void *handle) return h->exportsize; } +/* Multi-conn is safe for read-only connections, but HTTP does not + * have any concept of flushing so we cannot use it for read-write + * connections. + */ +static int +curl_can_multi_conn (void *handle) +{ + struct curl_handle *h = handle; + + return !! h->readonly; +} + /* NB: The terminology used by libcurl is confusing! * * WRITEFUNCTION / write_cb is used when reading from the remote server @@ -924,6 +937,7 @@ static struct nbdkit_plugin plugin = { .open = curl_open, .close = curl_close, .get_size = curl_get_size, + .can_multi_conn = curl_can_multi_conn, .pread = curl_pread, .pwrite = curl_pwrite, }; -- 2.39.0

2 2

[p2v PATCH 00/11] Expose virt-v2v's "-oo"; re-enable openstack
by Laszlo Ersek 02 Feb '23

02 Feb '23

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1792141 Let the user pass "-oo" options from the kernel cmdline and from the GUI to virt-v2v. This is primarily useful with the OpenStack output mode, so reenable that mode. Cc: Alban Lecorps <alban.lecorps(a)ubisoft.com> Laszlo Alban Lecorps (1): Introduce "p2v.output.misc" for passing "-oo" options to virt-v2v Laszlo Ersek (10): test-virt-p2v-cmdline: turn option list into a shell array guestfs-utils: import guestfs_int_join_strings() gui: factor out entry_text_dup() gui: factor out tgl_btn_is_act() gui: flatten get_phys_topo_from_conv_dlg() gui: wrap overlong function declaration lines gui: wrap string literals gui: expose "p2v.output.misc" (-oo) Reenable the OpenStack output mode make-kickstart: add URLs for RHEL-9 conversion.c | 9 ++ generate-p2v-config.pl | 10 ++ gui.c | 168 ++++++++++++++++---- libguestfs/guestfs-utils.c | 35 ++++ libguestfs/guestfs-utils.h | 1 + ssh.c | 5 +- test-virt-p2v-cmdline.sh | 24 ++- virt-p2v-make-kickstart.in | 13 ++ virt-p2v.pod | 2 + 9 files changed, 230 insertions(+), 37 deletions(-)

2 17

[PATCH libnbd v2 0/4] Pass LISTEN_FDNAMES with systemd socket activation
by Richard W.M. Jones 01 Feb '23

01 Feb '23

This is an alternative approach to https://listman.redhat.com/archives/libguestfs/2023-January/030535.html After discussing this with Dan Berrange we came to the conclusion that you really might want to set LISTEN_FDNAMES to arbitrary short strings (or not set it). Especially when talking to qemu-storage-daemon which would allow you to use these names on the command line. Rich.

3 15