This rather complex feature solves a problem for certain web services that require a cookie or token for access, especially one which must be periodically renewed. For motivation for this feature see the included documentation, and item (1)(b) here: https://www.redhat.com/archives/libguestfs/2020-July/msg00069.html --- plugins/curl/nbdkit-curl-plugin.pod | 142 ++++++++++++ plugins/curl/Makefile.am | 2 + tests/Makefile.am | 47 ++++ plugins/curl/curldefs.h | 76 +++++++ plugins/curl/curl.c | 116 +++++++--- plugins/curl/scripts.c | 330 ++++++++++++++++++++++++++++ tests/test-curl-cookie-script.c | 143 ++++++++++++ tests/test-curl-header-script.c | 165 ++++++++++++++ .gitignore | 2 + 9 files changed, 991 insertions(+), 32 deletions(-) diff --git a/plugins/curl/nbdkit-curl-plugin.pod b/plugins/curl/nbdkit-curl-plugin.pod index 22c07f39..52875988 100644 --- a/plugins/curl/nbdkit-curl-plugin.pod +++ b/plugins/curl/nbdkit-curl-plugin.pod @@ -81,6 +81,14 @@ command line is not secure on shared machines. Use the alternate C<+FILENAME> syntax to pass it in a file, C<-> to read the cookie interactively, or C<-FD> to read it from a file descriptor. +=item B<cookie-script=>SCRIPT + +=item B<cookie-script-renew=>SECS + +Run C<SCRIPT> (a command or shell script fragment) to generate the +HTTP/HTTPS cookies. C<cookie-script> cannot be used with C<cookie>. +See L</HEADER AND COOKIE SCRIPTS> below. + =item B<header=>HEADER (nbdkit E<ge> 1.22) @@ -106,6 +114,14 @@ requests, even when following a redirect, which can cause headers (eg. containing sensitive authorization information) to be sent to hosts other than the one originally requested. +=item B<header-script=>SCRIPT + +=item B<header-script-renew=>SECS + +Run C<SCRIPT> (a command or shell script fragment) to generate the +HTTP/HTTPS headers. C<header-script> cannot be used with C<header>. +See L</HEADER AND COOKIE SCRIPTS> below. + =item B<password=>PASSWORD Set the password to use when connecting to the remote server. @@ -226,6 +242,132 @@ user-agent header. =back +=head1 HEADER AND COOKIE SCRIPTS + +While the C<header> and C<cookie> parameters can be used to specify +static headers and cookies which are used in every HTTP/HTTPS request, +the alternate C<header-script> and C<cookie-script> parameters can be +used to run an external script or program to generate headers and/or +cookies. This is particularly useful to access services which require +an authorization token. In addition the C<header-script-renew> and +C<cookie-script-renew> parameters allow you to renew the authorization +token by rerunning the script periodically. + +C<header-script> is incompatible with C<header>, and C<cookie-script> +is incompatible with C<cookie>. + +=head2 Header script + +The header script should print zero or more HTTP headers, each line of +output in the same format as the C<header> parameter. The headers +printed by the script are passed to L<CURLOPT_HTTPHEADER(3)>. + +In the following example, an imaginary web service requires +authentication using a token fetched from a separate login server. +The token expires after 60 seconds, so we also tell the plugin that it +must renew the token (by re-running the script) if more than 45 +seconds have elapsed since the last request: + + nbdkit curl https://service.example.com/disk.img \ + header-script=' + echo -n "Authorization: Bearer " + curl -s -X POST https://auth.example.com/login | + jq -r .token + ' \ + header-script-renew=50 + +=head2 Cookie script + +The cookie script should print a single line in the same format as the +C<cookie> parameter. This is passed to L<CURLOPT_COOKIE(3)>. + +=head2 Header and cookie script shell variables + +Within the C<header-script> and C<cookie-script> the following shell +variables are available: + +=over 4 + +=item C<$iteration> + +The number of times that the script has been called. The first time +the script is called this contains C<0>. + +=item C<$url> + +The URL as passed to the plugin. + +=back + +=head2 Example: VMware ESXi cookies + +VMware ESXi’s web server can expose both VMDK and raw format disk +images, but requires you to log in using HTTP Basic Authentication. +While you can use the C<user> and C<password> parameters to send HTTP +Basic Authentication headers in every request, tests have shown that +it is faster to accept the cookie which the server returns and send +that instead. (It is not clear why it is faster, but one theory is +that VMware has to do a more expensive username and password check +each time.) + +The web server can be accessed as below. Since the cookie expires +after a certain period of time, we use C<cookie-script-renew>, and +because the server uses a self-signed certificate we must use +I<--insecure> and C<sslverify=false>. + + SERVER=esx.example.com + DCPATH=data + DS=datastore1 + GUEST=guest-name + URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS" + + nbdkit curl "$URL" \ + cookie-script=' + curl --head -s --insecure -u root:password "$url" | + sed -ne '{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }' + ' \ + cookie-script-renew=500 \ + sslverify=false + +=head2 Example: Docker Hub authorization tokens + +Accessing objects like container layers from Docker Hub requires that +you first fetch an authorization token, even for anonymous access. +These tokens expire after about 5 minutes (300 seconds) so must be +periodically renewed. + +You will need this authorization script (F</tmp/auth.sh>): + + #!/bin/sh - + IMAGE=library/fedora + curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" | + jq -r .token + +You will also need this script to get the blobSum of the layer +(F</tmp/blobsum.sh>): + + #!/bin/sh - + TOKEN=`/tmp/auth.sh` + IMAGE=library/fedora + curl -s -X GET -H "Authorization: Bearer $TOKEN" \ + "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" | + jq -r '.fsLayers[0].blobSum' + +Both scripts must be executable, and both can be run on their own to +check they are working. To run nbdkit: + + IMAGE=library/fedora + BLOBSUM=`/tmp/blobsum.sh` + URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM" + + nbdkit curl "$URL" \ + header-script=' echo -n "Authorization: Bearer "; /tmp/auth.sh ' \ + header-script-renew=200 \ + --filter=gzip + +Note that this exposes a tar file over NBD. See also +L<nbdkit-tar-filter(1)>. + =head1 DEBUG FLAG =over 4 diff --git a/plugins/curl/Makefile.am b/plugins/curl/Makefile.am index ddf1a215..0dd78199 100644 --- a/plugins/curl/Makefile.am +++ b/plugins/curl/Makefile.am @@ -38,6 +38,8 @@ if HAVE_CURL plugin_LTLIBRARIES = nbdkit-curl-plugin.la nbdkit_curl_plugin_la_SOURCES = \ + curldefs.h \ + scripts.c \ curl.c \ $(top_srcdir)/include/nbdkit-plugin.h \ $(NULL) diff --git a/tests/Makefile.am b/tests/Makefile.am index b830d80e..2641910b 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -502,6 +502,10 @@ if HAVE_CURL TESTS += test-curl-file.sh EXTRA_DIST += test-curl-file.sh LIBGUESTFS_TESTS += test-curl +LIBNBD_TESTS += \ + test-curl-header-script \ + test-curl-cookie-script \ + $(NULL) test_curl_SOURCES = \ test-curl.c \ @@ -524,6 +528,49 @@ test_curl_LDADD = \ libtest.la \ $(LIBGUESTFS_LIBS) \ $(NULL) + +test_curl_header_script_SOURCES = \ + test-curl-header-script.c \ + web-server.c \ + web-server.h \ + $(NULL) +test_curl_header_script_CPPFLAGS = \ + -I$(top_srcdir)/common/utils \ + $(NULL) +test_curl_header_script_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(LIBNBD_CFLAGS) \ + $(PTHREAD_CFLAGS) \ + $(NULL) +test_curl_header_script_LDFLAGS = \ + $(top_builddir)/common/utils/libutils.la \ + $(PTHREAD_LIBS) \ + $(NULL) +test_curl_header_script_LDADD = \ + $(LIBNBD_LIBS) \ + $(NULL) + +test_curl_cookie_script_SOURCES = \ + test-curl-header-script.c \ + web-server.c \ + web-server.h \ + $(NULL) +test_curl_cookie_script_CPPFLAGS = \ + -I$(top_srcdir)/common/utils \ + $(NULL) +test_curl_cookie_script_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(LIBNBD_CFLAGS) \ + $(PTHREAD_CFLAGS) \ + $(NULL) +test_curl_cookie_script_LDFLAGS = \ + $(top_builddir)/common/utils/libutils.la \ + $(PTHREAD_LIBS) \ + $(NULL) +test_curl_cookie_script_LDADD = \ + $(LIBNBD_LIBS) \ + $(NULL) + endif HAVE_CURL endif HAVE_MKE2FS_WITH_D diff --git a/plugins/curl/curldefs.h b/plugins/curl/curldefs.h new file mode 100644 index 00000000..5ec03231 --- /dev/null +++ b/plugins/curl/curldefs.h @@ -0,0 +1,76 @@ +/* nbdkit + * Copyright (C) 2014-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#ifndef NBDKIT_CURLDEFS_H +#define NBDKIT_CURLDEFS_H + +extern const char *url; + +extern const char *cainfo; +extern const char *capath; +extern char *cookie; +extern const char *cookie_script; +extern unsigned cookie_script_renew; +extern struct curl_slist *headers; +extern const char *header_script; +extern unsigned header_script_renew; +extern char *password; +extern long protocols; +extern const char *proxy; +extern char *proxy_password; +extern const char *proxy_user; +extern bool sslverify; +extern bool tcp_keepalive; +extern bool tcp_nodelay; +extern uint32_t timeout; +extern const char *unix_socket_path; +extern const char *user; +extern const char *user_agent; + +/* The per-connection handle. */ +struct curl_handle { + CURL *c; + bool accept_range; + int64_t exportsize; + char errbuf[CURL_ERROR_SIZE]; + char *write_buf; + uint32_t write_count; + const char *read_buf; + uint32_t read_count; + struct curl_slist *headers_copy; +}; + +/* scripts.c */ +extern int do_scripts (struct curl_handle *h); +extern void scripts_unload (void); + +#endif /* NBDKIT_CURLDEFS_H */ diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c index 50eef1a8..8731a506 100644 --- a/plugins/curl/curl.c +++ b/plugins/curl/curl.c @@ -48,9 +48,11 @@ #include <nbdkit-plugin.h> -#include "cleanup.h" #include "ascii-ctype.h" #include "ascii-string.h" +#include "cleanup.h" + +#include "curldefs.h" /* Macro CURL_AT_LEAST_VERSION was added in 2015 (Curl 7.43) so if the * macro isn't present then Curl is very old. @@ -61,24 +63,29 @@ #endif #endif -static const char *url = NULL; /* required */ +/* Plugin configuration. */ +const char *url = NULL; /* required */ -static const char *cainfo = NULL; -static const char *capath = NULL; -static char *cookie = NULL; -static struct curl_slist *headers = NULL; -static char *password = NULL; -static long protocols = CURLPROTO_ALL; -static const char *proxy = NULL; -static char *proxy_password = NULL; -static const char *proxy_user = NULL; -static bool sslverify = true; -static bool tcp_keepalive = false; -static bool tcp_nodelay = true; -static uint32_t timeout = 0; -static const char *unix_socket_path = NULL; -static const char *user = NULL; -static const char *user_agent = NULL; +const char *cainfo = NULL; +const char *capath = NULL; +char *cookie = NULL; +const char *cookie_script = NULL; +unsigned cookie_script_renew = 0; +struct curl_slist *headers = NULL; +const char *header_script = NULL; +unsigned header_script_renew = 0; +char *password = NULL; +long protocols = CURLPROTO_ALL; +const char *proxy = NULL; +char *proxy_password = NULL; +const char *proxy_user = NULL; +bool sslverify = true; +bool tcp_keepalive = false; +bool tcp_nodelay = true; +uint32_t timeout = 0; +const char *unix_socket_path = NULL; +const char *user = NULL; +const char *user_agent = NULL; /* Use '-D curl.verbose=1' to set. */ int curl_debug_verbose = 0; @@ -98,11 +105,12 @@ curl_load (void) static void curl_unload (void) { - free (password); - free (proxy_password); free (cookie); if (headers) curl_slist_free_all (headers); + free (password); + free (proxy_password); + scripts_unload (); curl_global_cleanup (); } @@ -202,6 +210,16 @@ curl_config (const char *key, const char *value) return -1; } + else if (strcmp (key, "cookie-script") == 0) { + cookie_script = value; + } + + else if (strcmp (key, "cookie-script-renew") == 0) { + if (nbdkit_parse_unsigned ("cookie-script-renew", value, + &cookie_script_renew) == -1) + return -1; + } + else if (strcmp (key, "header") == 0) { headers = curl_slist_append (headers, value); if (headers == NULL) { @@ -210,6 +228,16 @@ curl_config (const char *key, const char *value) } } + else if (strcmp (key, "header-script") == 0) { + header_script = value; + } + + else if (strcmp (key, "header-script-renew") == 0) { + if (nbdkit_parse_unsigned ("header-script-renew", value, + &header_script_renew) == -1) + return -1; + } + else if (strcmp (key, "password") == 0) { free (password); if (nbdkit_read_password (value, &password) == -1) @@ -300,6 +328,26 @@ curl_config_complete (void) return -1; } + if (headers && header_script) { + nbdkit_error ("header and header-script cannot be used at the same time"); + return -1; + } + + if (!header_script && header_script_renew) { + nbdkit_error ("header-script-renew cannot be used without header-script"); + return -1; + } + + if (cookie && cookie_script) { + nbdkit_error ("cookie and cookie-script cannot be used at the same time"); + return -1; + } + + if (!cookie_script && cookie_script_renew) { + nbdkit_error ("cookie-script-renew cannot be used without cookie-script"); + return -1; + } + return 0; } @@ -307,7 +355,11 @@ curl_config_complete (void) "cainfo=<CAINFO> Path to Certificate Authority file.\n" \ "capath=<CAPATH> Path to directory with CA certificates.\n" \ "cookie=<COOKIE> Set HTTP/HTTPS cookies.\n" \ + "cookie-script=<SCRIPT> Script to set HTTP/HTTPS cookies.\n" \ + "cookie-script-renew=<SECS> Time to renew HTTP/HTTPS cookies.\n" \ "header=<HEADER> Set HTTP/HTTPS header.\n" \ + "header-script=<SCRIPT> Script to set HTTP/HTTPS headers.\n" \ + "header-script-renew=<SECS> Time to renew HTTP/HTTPS headers.\n" \ "password=<PASSWORD> The password for the user account.\n" \ "protocols=PROTO,PROTO,.. Limit protocols allowed.\n" \ "proxy=<PROXY> Set proxy URL.\n" \ @@ -322,18 +374,6 @@ curl_config_complete (void) "user=<USER> The user to log in as.\n" \ "user-agent=<USER-AGENT> Send user-agent header for HTTP/HTTPS." -/* The per-connection handle. */ -struct curl_handle { - CURL *c; - bool accept_range; - int64_t exportsize; - char errbuf[CURL_ERROR_SIZE]; - char *write_buf; - uint32_t write_count; - const char *read_buf; - uint32_t read_count; -}; - /* Translate CURLcode to nbdkit_error. */ #define display_curl_error(h, r, fs, ...) \ do { \ @@ -450,7 +490,11 @@ curl_open (int readonly) /* Get the file size and also whether the remote HTTP server * supports byte ranges. + * + * We must run the scripts if necessary and set headers in the + * handle. */ + if (do_scripts (h) == -1) goto err; h->accept_range = false; curl_easy_setopt (h->c, CURLOPT_NOBODY, 1); /* No Body, not nobody! */ curl_easy_setopt (h->c, CURLOPT_HEADERFUNCTION, header_cb); @@ -608,6 +652,8 @@ curl_close (void *handle) struct curl_handle *h = handle; curl_easy_cleanup (h->c); + if (h->headers_copy) + curl_slist_free_all (h->headers_copy); free (h); } @@ -638,6 +684,9 @@ curl_pread (void *handle, void *buf, uint32_t count, uint64_t offset) CURLcode r; char range[128]; + /* Run the scripts if necessary and set headers in the handle. */ + if (do_scripts (h) == -1) return -1; + /* Tell the write_cb where we want the data to be written. write_cb * will update this if the data comes in multiple sections. */ @@ -699,6 +748,9 @@ curl_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset) CURLcode r; char range[128]; + /* Run the scripts if necessary and set headers in the handle. */ + if (do_scripts (h) == -1) return -1; + /* Tell the read_cb where we want the data to be read from. read_cb * will update this if the data comes in multiple sections. */ diff --git a/plugins/curl/scripts.c b/plugins/curl/scripts.c new file mode 100644 index 00000000..5d961391 --- /dev/null +++ b/plugins/curl/scripts.c @@ -0,0 +1,330 @@ +/* nbdkit + * Copyright (C) 2014-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Header and cookie scripts. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <string.h> +#include <unistd.h> +#include <time.h> +#include <assert.h> +#include <pthread.h> + +#include <curl/curl.h> + +#include <nbdkit-plugin.h> + +#include "ascii-ctype.h" +#include "cleanup.h" +#include "utils.h" + +#include "curldefs.h" + +/* This lock protects internal state in this file. */ +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; + +/* Last time header-script or cookie-script was run. */ +static time_t header_last = 0; +static time_t cookie_last = 0; +static bool header_script_has_run = false; +static bool cookie_script_has_run = false; +static unsigned header_iteration = 0; +static unsigned cookie_iteration = 0; + +/* Last set of headers and cookies generated by the scripts. */ +static struct curl_slist *headers_from_script = NULL; +static char *cookies_from_script = NULL; + +void +scripts_unload (void) +{ + curl_slist_free_all (headers_from_script); + free (cookies_from_script); +} + +static int run_header_script (struct curl_handle *); +static int run_cookie_script (struct curl_handle *); + +/* This is called from any thread just before we make a curl request. + * + * Because the thread model is NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS + * we can be assured of exclusive access to curl_handle here. + */ +int +do_scripts (struct curl_handle *h) +{ + time_t now; + struct curl_slist *p; + + /* Return quickly without acquiring the lock if this feature is not + * being used. + */ + if (!header_script && !cookie_script) + return 0; + + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + + /* Run or re-run header-script if we need to. */ + if (header_script) { + time (&now); + if (!header_script_has_run || + (header_script_renew > 0 && now - header_last >= header_script_renew)) { + if (run_header_script (h) == -1) + return -1; + header_last = now; + header_script_has_run = true; + } + } + + /* Run or re-run cookie-script if we need to. */ + if (cookie_script) { + time (&now); + if (!cookie_script_has_run || + (cookie_script_renew > 0 && now - cookie_last >= cookie_script_renew)) { + if (run_cookie_script (h) == -1) + return -1; + cookie_last = now; + cookie_script_has_run = true; + } + } + + /* Set headers and cookies in the handle. + * + * When calling CURLOPT_HTTPHEADER we have to keep the list around + * because unfortunately curl doesn't take a copy. Since we don't + * know which other threads might be using it, we must make a copy + * of the global list (headers_from_script) per handle + * (h->headers_copy). For CURLOPT_COOKIE, curl internally takes a + * copy so we don't need to do this. + */ + if (h->headers_copy) { + curl_easy_setopt (h->c, CURLOPT_HTTPHEADER, NULL); + curl_slist_free_all (h->headers_copy); + h->headers_copy = NULL; + } + for (p = headers_from_script; p != NULL; p = p->next) { + h->headers_copy = curl_slist_append (h->headers_copy, p->data); + if (h->headers_copy == NULL) { + nbdkit_error ("curl_slist_append: %m"); + return -1; + } + } + curl_easy_setopt (h->c, CURLOPT_HTTPHEADER, h->headers_copy); + + curl_easy_setopt (h->c, CURLOPT_COOKIE, cookies_from_script); + + return 0; +} + +/* This is called with the lock held when we must run or re-run the + * header-script. + */ +static int +run_header_script (struct curl_handle *h) +{ + int fd; + char tmpfile[] = "/tmp/errorsXXXXXX"; + FILE *fp; + CLEANUP_FREE char *cmd = NULL, *line = NULL; + size_t len = 0, linelen = 0, nr_headers = 0; + + assert (header_script != NULL); /* checked by caller */ + + /* Reset the list of headers. */ + curl_slist_free_all (headers_from_script); + headers_from_script = NULL; + + /* Create a temporary file for the errors so we can redirect them + * into nbdkit_error. + */ + fd = mkstemp (tmpfile); + if (fd == -1) { + nbdkit_error ("mkstemp"); + return -1; + } + close (fd); + + /* Generate the full script with the local $url variable. */ + fp = open_memstream (&cmd, &len); + if (fp == NULL) { + nbdkit_error ("open_memstream: %m"); + return -1; + } + fprintf (fp, "exec </dev/null\n"); /* Avoid stdin leaking (nbdkit -s). */ + fprintf (fp, "exec 2>%s\n", tmpfile); /* Catch errors to a temporary file. */ + fprintf (fp, "url="); /* Set the shell variables. */ + shell_quote (url, fp); + putc ('\n', fp); + fprintf (fp, "iteration=%u\n", header_iteration++); + putc ('\n', fp); + fprintf (fp, "%s", header_script); /* The script or command. */ + if (fclose (fp) == EOF) { + nbdkit_error ("memstream failed"); + return -1; + } + + /* Run the script and read the headers. */ + nbdkit_debug ("curl: running header-script"); + fp = popen (cmd, "r"); + if (fp == NULL) { + nbdkit_error ("popen: %m"); + return -1; + } + while ((len = getline (&line, &linelen, fp)) != -1) { + /* Remove trailing \n and whitespace. */ + while (len > 0 && ascii_isspace (line[len-1])) + line[--len] = '\0'; + if (len == 0) + continue; + + headers_from_script = curl_slist_append (headers_from_script, line); + if (headers_from_script == NULL) { + nbdkit_error ("curl_slist_append: %m"); + pclose (fp); + return -1; + } + nr_headers++; + } + + /* If the command failed, this should return EOF and the error + * message should be in the temporary file (but we only read the + * first line). + */ + if (pclose (fp) == EOF) { + fp = fopen (tmpfile, "r"); + if ((len = getline (&line, &linelen, fp)) >= 0) { + if (len > 0 && line[len-1] == '\n') + line[len-1] = '\0'; + nbdkit_error ("header-script failed: %s", line); + } + else + nbdkit_error ("header-script failed"); + return -1; + } + + nbdkit_debug ("header-script returned %zu header(s)", nr_headers); + return 0; +} + +/* This is called with the lock held when we must run or re-run the + * cookie-script. + */ +static int +run_cookie_script (struct curl_handle *h) +{ + int fd; + char tmpfile[] = "/tmp/errorsXXXXXX"; + FILE *fp; + CLEANUP_FREE char *cmd = NULL, *line = NULL; + size_t len = 0, linelen = 0; + + assert (cookie_script != NULL); /* checked by caller */ + + /* Reset the cookies. */ + free (cookies_from_script); + cookies_from_script = NULL; + + /* Create a temporary file for the errors so we can redirect them + * into nbdkit_error. + */ + fd = mkstemp (tmpfile); + if (fd == -1) { + nbdkit_error ("mkstemp"); + return -1; + } + close (fd); + + /* Generate the full script with the local $url variable. */ + fp = open_memstream (&cmd, &len); + if (fp == NULL) { + nbdkit_error ("open_memstream: %m"); + return -1; + } + fprintf (fp, "exec </dev/null\n"); /* Avoid stdin leaking (nbdkit -s). */ + fprintf (fp, "exec 2>%s\n", tmpfile); /* Catch errors to a temporary file. */ + fprintf (fp, "url="); /* Set the shell variable. */ + shell_quote (url, fp); + putc ('\n', fp); + fprintf (fp, "iteration=%u\n", cookie_iteration++); + putc ('\n', fp); + fprintf (fp, "%s", cookie_script); /* The script or command. */ + if (fclose (fp) == EOF) { + nbdkit_error ("memstream failed"); + return -1; + } + + /* Run the script and read the cookies. */ + nbdkit_debug ("curl: running cookie-script"); + fp = popen (cmd, "r"); + if (fp == NULL) { + nbdkit_error ("popen: %m"); + return -1; + } + len = getline (&line, &linelen, fp); + if (len > 0) { + /* Remove trailing \n and whitespace. */ + while (len > 0 && ascii_isspace (line[len-1])) + line[--len] = '\0'; + if (len > 0) { + cookies_from_script = strdup (line); + if (cookies_from_script == NULL) { + nbdkit_error ("strdup"); + pclose (fp); + return -1; + } + } + } + + /* If the command failed, this should return EOF and the error + * message should be in the temporary file (but we only read the + * first line). + */ + if (pclose (fp) == EOF) { + fp = fopen (tmpfile, "r"); + if ((len = getline (&line, &linelen, fp)) >= 0) { + if (len > 0 && line[len-1] == '\n') + line[len-1] = '\0'; + nbdkit_error ("cookie-script failed: %s", line); + } + else + nbdkit_error ("cookie-script failed"); + return -1; + } + + nbdkit_debug ("cookie-script returned %scookies", + cookies_from_script ? "" : "no "); + return 0; +} diff --git a/tests/test-curl-cookie-script.c b/tests/test-curl-cookie-script.c new file mode 100644 index 00000000..481207b5 --- /dev/null +++ b/tests/test-curl-cookie-script.c @@ -0,0 +1,143 @@ +/* nbdkit + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> + +#include <libnbd.h> + +#include "cleanup.h" +#include "web-server.h" + +#include "test.h" + +static int iteration; + +#define SCRIPT \ + "echo iteration=$iteration" + +static void +check_request (const char *request) +{ + char expected[64]; + + /* Check the Cookie header. */ + snprintf (expected, sizeof expected, + "\r\nCookie: iteration=%u\r\n", iteration); + if (strcasestr (request, expected) == NULL) { + fprintf (stderr, "%s: no/incorrect iteration cookie in request\n", + program_name); + exit (EXIT_FAILURE); + } +} + +static char buf[512]; + +int +main (int argc, char *argv[]) +{ + const char *sockpath; + struct nbd_handle *nbd; + CLEANUP_FREE char *usp_param = NULL; + +#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH + fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n", + program_name); + exit (77); +#endif + + sockpath = web_server ("disk", check_request); + if (sockpath == NULL) { + fprintf (stderr, "%s: could not start web server thread\n", program_name); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* We expect that connecting will cause a HEAD request (to find the + * size). $iteration will be 0. + */ + iteration = 0; + + /* Start nbdkit. */ + if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + char *args[] = { + "nbdkit", "-s", "--exit-with-parent", "-v", + "curl", + "-D", "curl.verbose=1", + "http://localhost/disk", + "cookie-script=" SCRIPT, + "cookie-script-renew=1", + usp_param, /* unix-socket-path=... */ + NULL + }; + if (nbd_connect_command (nbd, args) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Sleep the script will be called again. $iteration will be 1. */ + sleep (2); + iteration = 1; + + /* Make a request. */ + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Sleep again and make another request. $iteration will be 2. */ + sleep (2); + iteration = 2; + + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +} diff --git a/tests/test-curl-header-script.c b/tests/test-curl-header-script.c new file mode 100644 index 00000000..a151af05 --- /dev/null +++ b/tests/test-curl-header-script.c @@ -0,0 +1,165 @@ +/* nbdkit + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> + +#include <libnbd.h> + +#include "cleanup.h" +#include "web-server.h" + +#include "test.h" + +static int iteration; + +#define SCRIPT \ + "if [ $iteration -eq 0 ]; then echo X-Test: hello; fi\n" \ + "echo X-Iteration: $iteration\n" \ + "echo 'X-Empty;'\n" + +static void +check_request (const char *request) +{ + char expected[64]; + + /* Check the iteration header. */ + snprintf (expected, sizeof expected, "\r\nX-Iteration: %u\r\n", iteration); + if (strcasestr (request, expected) == NULL) { + fprintf (stderr, "%s: no/incorrect X-Iteration header in request\n", + program_name); + exit (EXIT_FAILURE); + } + + /* Check the test header, only sent when $iteration = 0. */ + if (iteration == 0) { + if (strcasestr (request, "\r\nX-Test: hello\r\n") == NULL) { + fprintf (stderr, "%s: no X-Test header in request\n", program_name); + exit (EXIT_FAILURE); + } + } + else { + if (strcasestr (request, "\r\nX-Test:") != NULL) { + fprintf (stderr, "%s: X-Test header sent but not expected\n", + program_name); + exit (EXIT_FAILURE); + } + } + + /* Check the empty header. */ + if (strcasestr (request, "\r\nX-Empty:\r\n") == NULL) { + fprintf (stderr, "%s: no X-Empty header in request\n", program_name); + exit (EXIT_FAILURE); + } +} + +static char buf[512]; + +int +main (int argc, char *argv[]) +{ + const char *sockpath; + struct nbd_handle *nbd; + CLEANUP_FREE char *usp_param = NULL; + +#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH + fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n", + program_name); + exit (77); +#endif + + sockpath = web_server ("disk", check_request); + if (sockpath == NULL) { + fprintf (stderr, "%s: could not start web server thread\n", program_name); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* We expect that connecting will cause a HEAD request (to find the + * size). $iteration will be 0. + */ + iteration = 0; + + /* Start nbdkit. */ + if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + char *args[] = { + "nbdkit", "-s", "--exit-with-parent", "-v", + "curl", + "-D", "curl.verbose=1", + "http://localhost/disk", + "header-script=" SCRIPT, + "header-script-renew=1", + usp_param, /* unix-socket-path=... */ + NULL + }; + if (nbd_connect_command (nbd, args) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Sleep the script will be called again. $iteration will be 1. */ + sleep (2); + iteration = 1; + + /* Make a request. */ + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Sleep again and make another request. $iteration will be 2. */ + sleep (2); + iteration = 2; + + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +} diff --git a/.gitignore b/.gitignore index a39aa675..255a97a5 100644 --- a/.gitignore +++ b/.gitignore @@ -113,6 +113,8 @@ plugins/*/*.3 /tests/stamp-ssh-user-key /tests/test-connect /tests/test-curl +/tests/test-curl-cookie-script +/tests/test-curl-header-script /tests/test-data /tests/test-delay /tests/test-exit-with-parent -- 2.27.0

On Mon, Jul 20, 2020 at 10:20:48AM -0500, Eric Blake wrote: Yes you guessed right. There is still the opportunity for someone to set the renew time to be too low (or for it to be set correctly but the remote taking excessively long to renew) and they will end up wasting time, but I'm not sure there's much we can do about it. ... I'll push small updates to fix this thanks. I thought I checked these ... will check again and amend as required. Unfortunately it's hard to add automated tests of this stuff since we can't rely on having a real server. Will fix. Assuming it costs nothing, I think it's sometimes worth self-documenting by having these unnecessary initializations. Of course if GCC is doing anything stupid like moving them out of the BSS then that would be a problem. I attempted that but it's annoyingly difficult. They are a little bit too different :-( Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/