[PATCH nbdkit] curl: Enable multi-conn for read-only connections
Multi-conn is enabled only when we know the connection is read-only: $ ./nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: true $ ./nbdkit curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: false This improves performance. Comparing before and after this commit shows approximate doubling of performance: Benchmark 1: nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" Time (mean ± σ): 943.8 ms ± 18.8 ms [User: 316.2 ms, System: 1029.7 ms] Range (min … max): 923.7 ms … 989.2 ms 10 runs Benchmark 2: ~/d/nbdkit/nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" Time (mean ± σ): 455.0 ms ± 6.2 ms [User: 542.2 ms, System: 1824.7 ms] Range (min … max): 449.1 ms … 471.6 ms 10 runs Summary ' ~/d/nbdkit/nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" ' ran 2.07 ± 0.05 times faster than ' nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run "nbdcopy -p \$uri null:" ' See also: https://listman.redhat.com/archives/libguestfs/2023-February/030581.html --- plugins/curl/curldefs.h | 1 + plugins/curl/curl.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+) diff --git a/plugins/curl/curldefs.h b/plugins/curl/curldefs.h index a2c521c50..16b6071cc 100644 --- a/plugins/curl/curldefs.h +++ b/plugins/curl/curldefs.h @@ -68,6 +68,7 @@ extern const char *user_agent; /* The per-connection handle. */ struct curl_handle { CURL *c; + int readonly; bool accept_range; int64_t exportsize; char errbuf[CURL_ERROR_SIZE]; diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c index 42aba4216..2f34d1dc1 100644 --- a/plugins/curl/curl.c +++ b/plugins/curl/curl.c @@ -465,6 +465,7 @@ curl_open (int readonly) nbdkit_error ("calloc: %m"); return NULL; } + h->readonly = readonly; h->c = curl_easy_init (); if (h->c == NULL) { @@ -781,6 +782,18 @@ curl_get_size (void *handle) return h->exportsize; } +/* Multi-conn is safe for read-only connections, but HTTP does not + * have any concept of flushing so we cannot use it for read-write + * connections. + */ +static int +curl_can_multi_conn (void *handle) +{ + struct curl_handle *h = handle; + + return !! h->readonly; +} + /* NB: The terminology used by libcurl is confusing! * * WRITEFUNCTION / write_cb is used when reading from the remote server @@ -924,6 +937,7 @@ static struct nbdkit_plugin plugin = { .open = curl_open, .close = curl_close, .get_size = curl_get_size, + .can_multi_conn = curl_can_multi_conn, .pread = curl_pread, .pwrite = curl_pwrite, }; -- 2.39.0
On Thu, Feb 02, 2023 at 03:06:30PM +0000, Richard W.M. Jones wrote:
Multi-conn is enabled only when we know the connection is read-only:
$ ./nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: true $ ./nbdkit curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: false
This improves performance.
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Thu, Feb 02, 2023 at 09:18:34AM -0600, Eric Blake wrote:
On Thu, Feb 02, 2023 at 03:06:30PM +0000, Richard W.M. Jones wrote:
Multi-conn is enabled only when we know the connection is read-only:
$ ./nbdkit -r curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: true $ ./nbdkit curl file:/var/tmp/jammy-server-cloudimg-amd64.raw --run ' nbdinfo $uri ' | grep can_multi_conn can_multi_conn: false
This improves performance.
Reviewed-by: Eric Blake <eblake@redhat.com>
Thanks, upstream in bb0f93ad7 I am going to propose this for RHEL 9.2: https://bugzilla.redhat.com/show_bug.cgi?id=2166686 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
participants (2)
-
Eric Blake -
Richard W.M. Jones