[Libguestfs] [PATCH nbdkit v2] curl: Implement header and cookie scripts.

This rather complex feature solves a problem for certain web services that require a cookie or token for access, especially one which must be periodically renewed. For motivation for this feature see the included documentation, and item (1)(b) here: https://www.redhat.com/archives/libguestfs/2020-July/msg00069.html --- plugins/curl/nbdkit-curl-plugin.pod | 142 ++++++++++++ plugins/curl/Makefile.am | 2 + tests/Makefile.am | 47 ++++ plugins/curl/curldefs.h | 76 +++++++ plugins/curl/curl.c | 116 +++++++--- plugins/curl/scripts.c | 330 ++++++++++++++++++++++++++++ tests/test-curl-cookie-script.c | 143 ++++++++++++ tests/test-curl-header-script.c | 165 ++++++++++++++ .gitignore | 2 + 9 files changed, 991 insertions(+), 32 deletions(-) diff --git a/plugins/curl/nbdkit-curl-plugin.pod b/plugins/curl/nbdkit-curl-plugin.pod index 22c07f39..52875988 100644 --- a/plugins/curl/nbdkit-curl-plugin.pod +++ b/plugins/curl/nbdkit-curl-plugin.pod @@ -81,6 +81,14 @@ command line is not secure on shared machines. Use the alternate C<+FILENAME> syntax to pass it in a file, C<-> to read the cookie interactively, or C<-FD> to read it from a file descriptor. +=item B<cookie-script=>SCRIPT + +=item B<cookie-script-renew=>SECS + +Run C<SCRIPT> (a command or shell script fragment) to generate the +HTTP/HTTPS cookies. C<cookie-script> cannot be used with C<cookie>. +See L</HEADER AND COOKIE SCRIPTS> below. + =item B<header=>HEADER (nbdkit E<ge> 1.22) @@ -106,6 +114,14 @@ requests, even when following a redirect, which can cause headers (eg. containing sensitive authorization information) to be sent to hosts other than the one originally requested. +=item B<header-script=>SCRIPT + +=item B<header-script-renew=>SECS + +Run C<SCRIPT> (a command or shell script fragment) to generate the +HTTP/HTTPS headers. C<header-script> cannot be used with C<header>. +See L</HEADER AND COOKIE SCRIPTS> below. + =item B<password=>PASSWORD Set the password to use when connecting to the remote server. @@ -226,6 +242,132 @@ user-agent header. =back +=head1 HEADER AND COOKIE SCRIPTS + +While the C<header> and C<cookie> parameters can be used to specify +static headers and cookies which are used in every HTTP/HTTPS request, +the alternate C<header-script> and C<cookie-script> parameters can be +used to run an external script or program to generate headers and/or +cookies. This is particularly useful to access services which require +an authorization token. In addition the C<header-script-renew> and +C<cookie-script-renew> parameters allow you to renew the authorization +token by rerunning the script periodically. + +C<header-script> is incompatible with C<header>, and C<cookie-script> +is incompatible with C<cookie>. + +=head2 Header script + +The header script should print zero or more HTTP headers, each line of +output in the same format as the C<header> parameter. The headers +printed by the script are passed to L<CURLOPT_HTTPHEADER(3)>. + +In the following example, an imaginary web service requires +authentication using a token fetched from a separate login server. +The token expires after 60 seconds, so we also tell the plugin that it +must renew the token (by re-running the script) if more than 45 +seconds have elapsed since the last request: + + nbdkit curl https://service.example.com/disk.img \ + header-script=' + echo -n "Authorization: Bearer " + curl -s -X POST https://auth.example.com/login | + jq -r .token + ' \ + header-script-renew=50 + +=head2 Cookie script + +The cookie script should print a single line in the same format as the +C<cookie> parameter. This is passed to L<CURLOPT_COOKIE(3)>. + +=head2 Header and cookie script shell variables + +Within the C<header-script> and C<cookie-script> the following shell +variables are available: + +=over 4 + +=item C<$iteration> + +The number of times that the script has been called. The first time +the script is called this contains C<0>. + +=item C<$url> + +The URL as passed to the plugin. + +=back + +=head2 Example: VMware ESXi cookies + +VMware ESXi’s web server can expose both VMDK and raw format disk +images, but requires you to log in using HTTP Basic Authentication. +While you can use the C<user> and C<password> parameters to send HTTP +Basic Authentication headers in every request, tests have shown that +it is faster to accept the cookie which the server returns and send +that instead. (It is not clear why it is faster, but one theory is +that VMware has to do a more expensive username and password check +each time.) + +The web server can be accessed as below. Since the cookie expires +after a certain period of time, we use C<cookie-script-renew>, and +because the server uses a self-signed certificate we must use +I<--insecure> and C<sslverify=false>. + + SERVER=esx.example.com + DCPATH=data + DS=datastore1 + GUEST=guest-name + URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS" + + nbdkit curl "$URL" \ + cookie-script=' + curl --head -s --insecure -u root:password "$url" | + sed -ne '{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }' + ' \ + cookie-script-renew=500 \ + sslverify=false + +=head2 Example: Docker Hub authorization tokens + +Accessing objects like container layers from Docker Hub requires that +you first fetch an authorization token, even for anonymous access. +These tokens expire after about 5 minutes (300 seconds) so must be +periodically renewed. + +You will need this authorization script (F</tmp/auth.sh>): + + #!/bin/sh - + IMAGE=library/fedora + curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" | + jq -r .token + +You will also need this script to get the blobSum of the layer +(F</tmp/blobsum.sh>): + + #!/bin/sh - + TOKEN=`/tmp/auth.sh` + IMAGE=library/fedora + curl -s -X GET -H "Authorization: Bearer $TOKEN" \ + "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" | + jq -r '.fsLayers[0].blobSum' + +Both scripts must be executable, and both can be run on their own to +check they are working. To run nbdkit: + + IMAGE=library/fedora + BLOBSUM=`/tmp/blobsum.sh` + URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM" + + nbdkit curl "$URL" \ + header-script=' echo -n "Authorization: Bearer "; /tmp/auth.sh ' \ + header-script-renew=200 \ + --filter=gzip + +Note that this exposes a tar file over NBD. See also +L<nbdkit-tar-filter(1)>. + =head1 DEBUG FLAG =over 4 diff --git a/plugins/curl/Makefile.am b/plugins/curl/Makefile.am index ddf1a215..0dd78199 100644 --- a/plugins/curl/Makefile.am +++ b/plugins/curl/Makefile.am @@ -38,6 +38,8 @@ if HAVE_CURL plugin_LTLIBRARIES = nbdkit-curl-plugin.la nbdkit_curl_plugin_la_SOURCES = \ + curldefs.h \ + scripts.c \ curl.c \ $(top_srcdir)/include/nbdkit-plugin.h \ $(NULL) diff --git a/tests/Makefile.am b/tests/Makefile.am index b830d80e..2641910b 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -502,6 +502,10 @@ if HAVE_CURL TESTS += test-curl-file.sh EXTRA_DIST += test-curl-file.sh LIBGUESTFS_TESTS += test-curl +LIBNBD_TESTS += \ + test-curl-header-script \ + test-curl-cookie-script \ + $(NULL) test_curl_SOURCES = \ test-curl.c \ @@ -524,6 +528,49 @@ test_curl_LDADD = \ libtest.la \ $(LIBGUESTFS_LIBS) \ $(NULL) + +test_curl_header_script_SOURCES = \ + test-curl-header-script.c \ + web-server.c \ + web-server.h \ + $(NULL) +test_curl_header_script_CPPFLAGS = \ + -I$(top_srcdir)/common/utils \ + $(NULL) +test_curl_header_script_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(LIBNBD_CFLAGS) \ + $(PTHREAD_CFLAGS) \ + $(NULL) +test_curl_header_script_LDFLAGS = \ + $(top_builddir)/common/utils/libutils.la \ + $(PTHREAD_LIBS) \ + $(NULL) +test_curl_header_script_LDADD = \ + $(LIBNBD_LIBS) \ + $(NULL) + +test_curl_cookie_script_SOURCES = \ + test-curl-header-script.c \ + web-server.c \ + web-server.h \ + $(NULL) +test_curl_cookie_script_CPPFLAGS = \ + -I$(top_srcdir)/common/utils \ + $(NULL) +test_curl_cookie_script_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(LIBNBD_CFLAGS) \ + $(PTHREAD_CFLAGS) \ + $(NULL) +test_curl_cookie_script_LDFLAGS = \ + $(top_builddir)/common/utils/libutils.la \ + $(PTHREAD_LIBS) \ + $(NULL) +test_curl_cookie_script_LDADD = \ + $(LIBNBD_LIBS) \ + $(NULL) + endif HAVE_CURL endif HAVE_MKE2FS_WITH_D diff --git a/plugins/curl/curldefs.h b/plugins/curl/curldefs.h new file mode 100644 index 00000000..5ec03231 --- /dev/null +++ b/plugins/curl/curldefs.h @@ -0,0 +1,76 @@ +/* nbdkit + * Copyright (C) 2014-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#ifndef NBDKIT_CURLDEFS_H +#define NBDKIT_CURLDEFS_H + +extern const char *url; + +extern const char *cainfo; +extern const char *capath; +extern char *cookie; +extern const char *cookie_script; +extern unsigned cookie_script_renew; +extern struct curl_slist *headers; +extern const char *header_script; +extern unsigned header_script_renew; +extern char *password; +extern long protocols; +extern const char *proxy; +extern char *proxy_password; +extern const char *proxy_user; +extern bool sslverify; +extern bool tcp_keepalive; +extern bool tcp_nodelay; +extern uint32_t timeout; +extern const char *unix_socket_path; +extern const char *user; +extern const char *user_agent; + +/* The per-connection handle. */ +struct curl_handle { + CURL *c; + bool accept_range; + int64_t exportsize; + char errbuf[CURL_ERROR_SIZE]; + char *write_buf; + uint32_t write_count; + const char *read_buf; + uint32_t read_count; + struct curl_slist *headers_copy; +}; + +/* scripts.c */ +extern int do_scripts (struct curl_handle *h); +extern void scripts_unload (void); + +#endif /* NBDKIT_CURLDEFS_H */ diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c index 50eef1a8..8731a506 100644 --- a/plugins/curl/curl.c +++ b/plugins/curl/curl.c @@ -48,9 +48,11 @@ #include <nbdkit-plugin.h> -#include "cleanup.h" #include "ascii-ctype.h" #include "ascii-string.h" +#include "cleanup.h" + +#include "curldefs.h" /* Macro CURL_AT_LEAST_VERSION was added in 2015 (Curl 7.43) so if the * macro isn't present then Curl is very old. @@ -61,24 +63,29 @@ #endif #endif -static const char *url = NULL; /* required */ +/* Plugin configuration. */ +const char *url = NULL; /* required */ -static const char *cainfo = NULL; -static const char *capath = NULL; -static char *cookie = NULL; -static struct curl_slist *headers = NULL; -static char *password = NULL; -static long protocols = CURLPROTO_ALL; -static const char *proxy = NULL; -static char *proxy_password = NULL; -static const char *proxy_user = NULL; -static bool sslverify = true; -static bool tcp_keepalive = false; -static bool tcp_nodelay = true; -static uint32_t timeout = 0; -static const char *unix_socket_path = NULL; -static const char *user = NULL; -static const char *user_agent = NULL; +const char *cainfo = NULL; +const char *capath = NULL; +char *cookie = NULL; +const char *cookie_script = NULL; +unsigned cookie_script_renew = 0; +struct curl_slist *headers = NULL; +const char *header_script = NULL; +unsigned header_script_renew = 0; +char *password = NULL; +long protocols = CURLPROTO_ALL; +const char *proxy = NULL; +char *proxy_password = NULL; +const char *proxy_user = NULL; +bool sslverify = true; +bool tcp_keepalive = false; +bool tcp_nodelay = true; +uint32_t timeout = 0; +const char *unix_socket_path = NULL; +const char *user = NULL; +const char *user_agent = NULL; /* Use '-D curl.verbose=1' to set. */ int curl_debug_verbose = 0; @@ -98,11 +105,12 @@ curl_load (void) static void curl_unload (void) { - free (password); - free (proxy_password); free (cookie); if (headers) curl_slist_free_all (headers); + free (password); + free (proxy_password); + scripts_unload (); curl_global_cleanup (); } @@ -202,6 +210,16 @@ curl_config (const char *key, const char *value) return -1; } + else if (strcmp (key, "cookie-script") == 0) { + cookie_script = value; + } + + else if (strcmp (key, "cookie-script-renew") == 0) { + if (nbdkit_parse_unsigned ("cookie-script-renew", value, + &cookie_script_renew) == -1) + return -1; + } + else if (strcmp (key, "header") == 0) { headers = curl_slist_append (headers, value); if (headers == NULL) { @@ -210,6 +228,16 @@ curl_config (const char *key, const char *value) } } + else if (strcmp (key, "header-script") == 0) { + header_script = value; + } + + else if (strcmp (key, "header-script-renew") == 0) { + if (nbdkit_parse_unsigned ("header-script-renew", value, + &header_script_renew) == -1) + return -1; + } + else if (strcmp (key, "password") == 0) { free (password); if (nbdkit_read_password (value, &password) == -1) @@ -300,6 +328,26 @@ curl_config_complete (void) return -1; } + if (headers && header_script) { + nbdkit_error ("header and header-script cannot be used at the same time"); + return -1; + } + + if (!header_script && header_script_renew) { + nbdkit_error ("header-script-renew cannot be used without header-script"); + return -1; + } + + if (cookie && cookie_script) { + nbdkit_error ("cookie and cookie-script cannot be used at the same time"); + return -1; + } + + if (!cookie_script && cookie_script_renew) { + nbdkit_error ("cookie-script-renew cannot be used without cookie-script"); + return -1; + } + return 0; } @@ -307,7 +355,11 @@ curl_config_complete (void) "cainfo=<CAINFO> Path to Certificate Authority file.\n" \ "capath=<CAPATH> Path to directory with CA certificates.\n" \ "cookie=<COOKIE> Set HTTP/HTTPS cookies.\n" \ + "cookie-script=<SCRIPT> Script to set HTTP/HTTPS cookies.\n" \ + "cookie-script-renew=<SECS> Time to renew HTTP/HTTPS cookies.\n" \ "header=<HEADER> Set HTTP/HTTPS header.\n" \ + "header-script=<SCRIPT> Script to set HTTP/HTTPS headers.\n" \ + "header-script-renew=<SECS> Time to renew HTTP/HTTPS headers.\n" \ "password=<PASSWORD> The password for the user account.\n" \ "protocols=PROTO,PROTO,.. Limit protocols allowed.\n" \ "proxy=<PROXY> Set proxy URL.\n" \ @@ -322,18 +374,6 @@ curl_config_complete (void) "user=<USER> The user to log in as.\n" \ "user-agent=<USER-AGENT> Send user-agent header for HTTP/HTTPS." -/* The per-connection handle. */ -struct curl_handle { - CURL *c; - bool accept_range; - int64_t exportsize; - char errbuf[CURL_ERROR_SIZE]; - char *write_buf; - uint32_t write_count; - const char *read_buf; - uint32_t read_count; -}; - /* Translate CURLcode to nbdkit_error. */ #define display_curl_error(h, r, fs, ...) \ do { \ @@ -450,7 +490,11 @@ curl_open (int readonly) /* Get the file size and also whether the remote HTTP server * supports byte ranges. + * + * We must run the scripts if necessary and set headers in the + * handle. */ + if (do_scripts (h) == -1) goto err; h->accept_range = false; curl_easy_setopt (h->c, CURLOPT_NOBODY, 1); /* No Body, not nobody! */ curl_easy_setopt (h->c, CURLOPT_HEADERFUNCTION, header_cb); @@ -608,6 +652,8 @@ curl_close (void *handle) struct curl_handle *h = handle; curl_easy_cleanup (h->c); + if (h->headers_copy) + curl_slist_free_all (h->headers_copy); free (h); } @@ -638,6 +684,9 @@ curl_pread (void *handle, void *buf, uint32_t count, uint64_t offset) CURLcode r; char range[128]; + /* Run the scripts if necessary and set headers in the handle. */ + if (do_scripts (h) == -1) return -1; + /* Tell the write_cb where we want the data to be written. write_cb * will update this if the data comes in multiple sections. */ @@ -699,6 +748,9 @@ curl_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset) CURLcode r; char range[128]; + /* Run the scripts if necessary and set headers in the handle. */ + if (do_scripts (h) == -1) return -1; + /* Tell the read_cb where we want the data to be read from. read_cb * will update this if the data comes in multiple sections. */ diff --git a/plugins/curl/scripts.c b/plugins/curl/scripts.c new file mode 100644 index 00000000..5d961391 --- /dev/null +++ b/plugins/curl/scripts.c @@ -0,0 +1,330 @@ +/* nbdkit + * Copyright (C) 2014-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Header and cookie scripts. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <string.h> +#include <unistd.h> +#include <time.h> +#include <assert.h> +#include <pthread.h> + +#include <curl/curl.h> + +#include <nbdkit-plugin.h> + +#include "ascii-ctype.h" +#include "cleanup.h" +#include "utils.h" + +#include "curldefs.h" + +/* This lock protects internal state in this file. */ +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; + +/* Last time header-script or cookie-script was run. */ +static time_t header_last = 0; +static time_t cookie_last = 0; +static bool header_script_has_run = false; +static bool cookie_script_has_run = false; +static unsigned header_iteration = 0; +static unsigned cookie_iteration = 0; + +/* Last set of headers and cookies generated by the scripts. */ +static struct curl_slist *headers_from_script = NULL; +static char *cookies_from_script = NULL; + +void +scripts_unload (void) +{ + curl_slist_free_all (headers_from_script); + free (cookies_from_script); +} + +static int run_header_script (struct curl_handle *); +static int run_cookie_script (struct curl_handle *); + +/* This is called from any thread just before we make a curl request. + * + * Because the thread model is NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS + * we can be assured of exclusive access to curl_handle here. + */ +int +do_scripts (struct curl_handle *h) +{ + time_t now; + struct curl_slist *p; + + /* Return quickly without acquiring the lock if this feature is not + * being used. + */ + if (!header_script && !cookie_script) + return 0; + + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + + /* Run or re-run header-script if we need to. */ + if (header_script) { + time (&now); + if (!header_script_has_run || + (header_script_renew > 0 && now - header_last >= header_script_renew)) { + if (run_header_script (h) == -1) + return -1; + header_last = now; + header_script_has_run = true; + } + } + + /* Run or re-run cookie-script if we need to. */ + if (cookie_script) { + time (&now); + if (!cookie_script_has_run || + (cookie_script_renew > 0 && now - cookie_last >= cookie_script_renew)) { + if (run_cookie_script (h) == -1) + return -1; + cookie_last = now; + cookie_script_has_run = true; + } + } + + /* Set headers and cookies in the handle. + * + * When calling CURLOPT_HTTPHEADER we have to keep the list around + * because unfortunately curl doesn't take a copy. Since we don't + * know which other threads might be using it, we must make a copy + * of the global list (headers_from_script) per handle + * (h->headers_copy). For CURLOPT_COOKIE, curl internally takes a + * copy so we don't need to do this. + */ + if (h->headers_copy) { + curl_easy_setopt (h->c, CURLOPT_HTTPHEADER, NULL); + curl_slist_free_all (h->headers_copy); + h->headers_copy = NULL; + } + for (p = headers_from_script; p != NULL; p = p->next) { + h->headers_copy = curl_slist_append (h->headers_copy, p->data); + if (h->headers_copy == NULL) { + nbdkit_error ("curl_slist_append: %m"); + return -1; + } + } + curl_easy_setopt (h->c, CURLOPT_HTTPHEADER, h->headers_copy); + + curl_easy_setopt (h->c, CURLOPT_COOKIE, cookies_from_script); + + return 0; +} + +/* This is called with the lock held when we must run or re-run the + * header-script. + */ +static int +run_header_script (struct curl_handle *h) +{ + int fd; + char tmpfile[] = "/tmp/errorsXXXXXX"; + FILE *fp; + CLEANUP_FREE char *cmd = NULL, *line = NULL; + size_t len = 0, linelen = 0, nr_headers = 0; + + assert (header_script != NULL); /* checked by caller */ + + /* Reset the list of headers. */ + curl_slist_free_all (headers_from_script); + headers_from_script = NULL; + + /* Create a temporary file for the errors so we can redirect them + * into nbdkit_error. + */ + fd = mkstemp (tmpfile); + if (fd == -1) { + nbdkit_error ("mkstemp"); + return -1; + } + close (fd); + + /* Generate the full script with the local $url variable. */ + fp = open_memstream (&cmd, &len); + if (fp == NULL) { + nbdkit_error ("open_memstream: %m"); + return -1; + } + fprintf (fp, "exec </dev/null\n"); /* Avoid stdin leaking (nbdkit -s). */ + fprintf (fp, "exec 2>%s\n", tmpfile); /* Catch errors to a temporary file. */ + fprintf (fp, "url="); /* Set the shell variables. */ + shell_quote (url, fp); + putc ('\n', fp); + fprintf (fp, "iteration=%u\n", header_iteration++); + putc ('\n', fp); + fprintf (fp, "%s", header_script); /* The script or command. */ + if (fclose (fp) == EOF) { + nbdkit_error ("memstream failed"); + return -1; + } + + /* Run the script and read the headers. */ + nbdkit_debug ("curl: running header-script"); + fp = popen (cmd, "r"); + if (fp == NULL) { + nbdkit_error ("popen: %m"); + return -1; + } + while ((len = getline (&line, &linelen, fp)) != -1) { + /* Remove trailing \n and whitespace. */ + while (len > 0 && ascii_isspace (line[len-1])) + line[--len] = '\0'; + if (len == 0) + continue; + + headers_from_script = curl_slist_append (headers_from_script, line); + if (headers_from_script == NULL) { + nbdkit_error ("curl_slist_append: %m"); + pclose (fp); + return -1; + } + nr_headers++; + } + + /* If the command failed, this should return EOF and the error + * message should be in the temporary file (but we only read the + * first line). + */ + if (pclose (fp) == EOF) { + fp = fopen (tmpfile, "r"); + if ((len = getline (&line, &linelen, fp)) >= 0) { + if (len > 0 && line[len-1] == '\n') + line[len-1] = '\0'; + nbdkit_error ("header-script failed: %s", line); + } + else + nbdkit_error ("header-script failed"); + return -1; + } + + nbdkit_debug ("header-script returned %zu header(s)", nr_headers); + return 0; +} + +/* This is called with the lock held when we must run or re-run the + * cookie-script. + */ +static int +run_cookie_script (struct curl_handle *h) +{ + int fd; + char tmpfile[] = "/tmp/errorsXXXXXX"; + FILE *fp; + CLEANUP_FREE char *cmd = NULL, *line = NULL; + size_t len = 0, linelen = 0; + + assert (cookie_script != NULL); /* checked by caller */ + + /* Reset the cookies. */ + free (cookies_from_script); + cookies_from_script = NULL; + + /* Create a temporary file for the errors so we can redirect them + * into nbdkit_error. + */ + fd = mkstemp (tmpfile); + if (fd == -1) { + nbdkit_error ("mkstemp"); + return -1; + } + close (fd); + + /* Generate the full script with the local $url variable. */ + fp = open_memstream (&cmd, &len); + if (fp == NULL) { + nbdkit_error ("open_memstream: %m"); + return -1; + } + fprintf (fp, "exec </dev/null\n"); /* Avoid stdin leaking (nbdkit -s). */ + fprintf (fp, "exec 2>%s\n", tmpfile); /* Catch errors to a temporary file. */ + fprintf (fp, "url="); /* Set the shell variable. */ + shell_quote (url, fp); + putc ('\n', fp); + fprintf (fp, "iteration=%u\n", cookie_iteration++); + putc ('\n', fp); + fprintf (fp, "%s", cookie_script); /* The script or command. */ + if (fclose (fp) == EOF) { + nbdkit_error ("memstream failed"); + return -1; + } + + /* Run the script and read the cookies. */ + nbdkit_debug ("curl: running cookie-script"); + fp = popen (cmd, "r"); + if (fp == NULL) { + nbdkit_error ("popen: %m"); + return -1; + } + len = getline (&line, &linelen, fp); + if (len > 0) { + /* Remove trailing \n and whitespace. */ + while (len > 0 && ascii_isspace (line[len-1])) + line[--len] = '\0'; + if (len > 0) { + cookies_from_script = strdup (line); + if (cookies_from_script == NULL) { + nbdkit_error ("strdup"); + pclose (fp); + return -1; + } + } + } + + /* If the command failed, this should return EOF and the error + * message should be in the temporary file (but we only read the + * first line). + */ + if (pclose (fp) == EOF) { + fp = fopen (tmpfile, "r"); + if ((len = getline (&line, &linelen, fp)) >= 0) { + if (len > 0 && line[len-1] == '\n') + line[len-1] = '\0'; + nbdkit_error ("cookie-script failed: %s", line); + } + else + nbdkit_error ("cookie-script failed"); + return -1; + } + + nbdkit_debug ("cookie-script returned %scookies", + cookies_from_script ? "" : "no "); + return 0; +} diff --git a/tests/test-curl-cookie-script.c b/tests/test-curl-cookie-script.c new file mode 100644 index 00000000..481207b5 --- /dev/null +++ b/tests/test-curl-cookie-script.c @@ -0,0 +1,143 @@ +/* nbdkit + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> + +#include <libnbd.h> + +#include "cleanup.h" +#include "web-server.h" + +#include "test.h" + +static int iteration; + +#define SCRIPT \ + "echo iteration=$iteration" + +static void +check_request (const char *request) +{ + char expected[64]; + + /* Check the Cookie header. */ + snprintf (expected, sizeof expected, + "\r\nCookie: iteration=%u\r\n", iteration); + if (strcasestr (request, expected) == NULL) { + fprintf (stderr, "%s: no/incorrect iteration cookie in request\n", + program_name); + exit (EXIT_FAILURE); + } +} + +static char buf[512]; + +int +main (int argc, char *argv[]) +{ + const char *sockpath; + struct nbd_handle *nbd; + CLEANUP_FREE char *usp_param = NULL; + +#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH + fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n", + program_name); + exit (77); +#endif + + sockpath = web_server ("disk", check_request); + if (sockpath == NULL) { + fprintf (stderr, "%s: could not start web server thread\n", program_name); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* We expect that connecting will cause a HEAD request (to find the + * size). $iteration will be 0. + */ + iteration = 0; + + /* Start nbdkit. */ + if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + char *args[] = { + "nbdkit", "-s", "--exit-with-parent", "-v", + "curl", + "-D", "curl.verbose=1", + "http://localhost/disk", + "cookie-script=" SCRIPT, + "cookie-script-renew=1", + usp_param, /* unix-socket-path=... */ + NULL + }; + if (nbd_connect_command (nbd, args) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Sleep the script will be called again. $iteration will be 1. */ + sleep (2); + iteration = 1; + + /* Make a request. */ + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Sleep again and make another request. $iteration will be 2. */ + sleep (2); + iteration = 2; + + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +} diff --git a/tests/test-curl-header-script.c b/tests/test-curl-header-script.c new file mode 100644 index 00000000..a151af05 --- /dev/null +++ b/tests/test-curl-header-script.c @@ -0,0 +1,165 @@ +/* nbdkit + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> + +#include <libnbd.h> + +#include "cleanup.h" +#include "web-server.h" + +#include "test.h" + +static int iteration; + +#define SCRIPT \ + "if [ $iteration -eq 0 ]; then echo X-Test: hello; fi\n" \ + "echo X-Iteration: $iteration\n" \ + "echo 'X-Empty;'\n" + +static void +check_request (const char *request) +{ + char expected[64]; + + /* Check the iteration header. */ + snprintf (expected, sizeof expected, "\r\nX-Iteration: %u\r\n", iteration); + if (strcasestr (request, expected) == NULL) { + fprintf (stderr, "%s: no/incorrect X-Iteration header in request\n", + program_name); + exit (EXIT_FAILURE); + } + + /* Check the test header, only sent when $iteration = 0. */ + if (iteration == 0) { + if (strcasestr (request, "\r\nX-Test: hello\r\n") == NULL) { + fprintf (stderr, "%s: no X-Test header in request\n", program_name); + exit (EXIT_FAILURE); + } + } + else { + if (strcasestr (request, "\r\nX-Test:") != NULL) { + fprintf (stderr, "%s: X-Test header sent but not expected\n", + program_name); + exit (EXIT_FAILURE); + } + } + + /* Check the empty header. */ + if (strcasestr (request, "\r\nX-Empty:\r\n") == NULL) { + fprintf (stderr, "%s: no X-Empty header in request\n", program_name); + exit (EXIT_FAILURE); + } +} + +static char buf[512]; + +int +main (int argc, char *argv[]) +{ + const char *sockpath; + struct nbd_handle *nbd; + CLEANUP_FREE char *usp_param = NULL; + +#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH + fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n", + program_name); + exit (77); +#endif + + sockpath = web_server ("disk", check_request); + if (sockpath == NULL) { + fprintf (stderr, "%s: could not start web server thread\n", program_name); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* We expect that connecting will cause a HEAD request (to find the + * size). $iteration will be 0. + */ + iteration = 0; + + /* Start nbdkit. */ + if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + char *args[] = { + "nbdkit", "-s", "--exit-with-parent", "-v", + "curl", + "-D", "curl.verbose=1", + "http://localhost/disk", + "header-script=" SCRIPT, + "header-script-renew=1", + usp_param, /* unix-socket-path=... */ + NULL + }; + if (nbd_connect_command (nbd, args) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Sleep the script will be called again. $iteration will be 1. */ + sleep (2); + iteration = 1; + + /* Make a request. */ + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Sleep again and make another request. $iteration will be 2. */ + sleep (2); + iteration = 2; + + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +} diff --git a/.gitignore b/.gitignore index a39aa675..255a97a5 100644 --- a/.gitignore +++ b/.gitignore @@ -113,6 +113,8 @@ plugins/*/*.3 /tests/stamp-ssh-user-key /tests/test-connect /tests/test-curl +/tests/test-curl-cookie-script +/tests/test-curl-header-script /tests/test-data /tests/test-delay /tests/test-exit-with-parent -- 2.27.0