[PATCH nbdkit 0/2] curl: Fallback to GET if HEAD not supported
AWS S3 servers are mad. You can do an unclear "pre-signing" operation on them, but only for a single method (eg. just GET). This means that the way we get the size of the export, by first querying the headers with HEAD and then using GET for partial data transfers, does not work. In theory we can get around this by falling back to using GET if we notice that the HEAD operation failed, and then dropping the connection when the server starts sending data. I restricted the fallback behaviour so it only happens when we see the expected 403 Forbidden error from AWS. I haven't been able to test it against a real server, but I modified our internal test web server so it can behave like this and added a test. Fixes: https://github.com/kubevirt/containerized-data-importer/issues/2737 Rich.
If the client (curl plugin) disconnects early then the whole test can fail with SIGPIPE, and it can be unclear why the test failed -- you just get the mysterious error "FAIL test (exit status: 141)". We always check the return code from write(2) so just report EPIPE failures through that. In addition, a future extension to this test will allow writes to fail with EPIPE in some circumstances. --- tests/web-server.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tests/web-server.c b/tests/web-server.c index 9b37326c0..dbc8bc845 100644 --- a/tests/web-server.c +++ b/tests/web-server.c @@ -43,6 +43,7 @@ #include <fcntl.h> #include <unistd.h> #include <errno.h> +#include <signal.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/socket.h> @@ -90,6 +91,13 @@ cleanup (void) rmdir (tmpdir); } +static void +ignore_sigpipe (void) +{ + struct sigaction sa = { .sa_flags = SA_RESTART, .sa_handler = SIG_IGN }; + sigaction (SIGPIPE, &sa, NULL); +} + const char * web_server (const char *filename, check_request_t _check_request) { @@ -97,6 +105,8 @@ web_server (const char *filename, check_request_t _check_request) pthread_t thread; int err; + ignore_sigpipe (); + check_request = _check_request; /* Open the file. */ -- 2.40.1
On 6/6/23 13:22, Richard W.M. Jones wrote:
If the client (curl plugin) disconnects early then the whole test can fail with SIGPIPE, and it can be unclear why the test failed -- you just get the mysterious error "FAIL test (exit status: 141)". We always check the return code from write(2) so just report EPIPE failures through that.
In addition, a future extension to this test will allow writes to fail with EPIPE in some circumstances. --- tests/web-server.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/tests/web-server.c b/tests/web-server.c index 9b37326c0..dbc8bc845 100644 --- a/tests/web-server.c +++ b/tests/web-server.c @@ -43,6 +43,7 @@ #include <fcntl.h> #include <unistd.h> #include <errno.h> +#include <signal.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/socket.h> @@ -90,6 +91,13 @@ cleanup (void) rmdir (tmpdir); }
+static void +ignore_sigpipe (void) +{ + struct sigaction sa = { .sa_flags = SA_RESTART, .sa_handler = SIG_IGN };
SA_RESTART makes no difference when the disposition is SIG_IGN; I'd rather remove SA_RESTART lest we imply otherwise. With that: Reviewed-by: Laszlo Ersek <lersek@redhat.com>
+ sigaction (SIGPIPE, &sa, NULL); +} + const char * web_server (const char *filename, check_request_t _check_request) { @@ -97,6 +105,8 @@ web_server (const char *filename, check_request_t _check_request) pthread_t thread; int err;
+ ignore_sigpipe (); + check_request = _check_request;
/* Open the file. */
Some servers do not support HEAD for requesting the headers. If the HEAD request fails, fallback to using the GET method, abandoning the transfer as soon as possible after the headers have been received. Fixes: https://github.com/kubevirt/containerized-data-importer/issues/2737 --- tests/Makefile.am | 23 +++++ plugins/curl/pool.c | 49 ++++++++++- tests/test-curl-cookie-script.c | 2 +- tests/test-curl-head-forbidden.c | 134 ++++++++++++++++++++++++++++++ tests/test-curl-header-script.c | 2 +- tests/test-curl.c | 2 +- tests/test-gzip-curl.c | 2 +- tests/test-retry-request-mirror.c | 2 +- tests/test-tar-gzip-curl.c | 2 +- tests/test-tar-xz-curl.c | 2 +- tests/test-xz-curl.c | 2 +- tests/web-server.c | 51 +++++++++++- tests/web-server.h | 9 +- .gitignore | 1 + 14 files changed, 271 insertions(+), 12 deletions(-) diff --git a/tests/Makefile.am b/tests/Makefile.am index 165d61aff..6694e409e 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -655,6 +655,7 @@ EXTRA_DIST += \ $(NULL) LIBGUESTFS_TESTS += test-curl LIBNBD_TESTS += \ + test-curl-head-forbidden \ test-curl-header-script \ test-curl-cookie-script \ $(NULL) @@ -683,6 +684,28 @@ test_curl_LDADD = \ $(LIBGUESTFS_LIBS) \ $(NULL) +test_curl_head_forbidden_SOURCES = \ + test-curl-head-forbidden.c \ + web-server.c \ + web-server.h \ + $(NULL) +test_curl_head_forbidden_CPPFLAGS = \ + -I$(top_srcdir)/common/include \ + -I$(top_srcdir)/common/utils \ + $(NULL) +test_curl_head_forbidden_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(LIBNBD_CFLAGS) \ + $(PTHREAD_CFLAGS) \ + $(NULL) +test_curl_head_forbidden_LDFLAGS = \ + $(top_builddir)/common/utils/libutils.la \ + $(PTHREAD_LIBS) \ + $(NULL) +test_curl_head_forbidden_LDADD = \ + $(LIBNBD_LIBS) \ + $(NULL) + test_curl_header_script_SOURCES = \ test-curl-header-script.c \ web-server.c \ diff --git a/plugins/curl/pool.c b/plugins/curl/pool.c index 2af5461a2..c73d52293 100644 --- a/plugins/curl/pool.c +++ b/plugins/curl/pool.c @@ -76,7 +76,9 @@ static int debug_cb (CURL *handle, curl_infotype type, static size_t write_cb (char *ptr, size_t size, size_t nmemb, void *opaque); static size_t read_cb (void *ptr, size_t size, size_t nmemb, void *opaque); static int get_content_length_accept_range (struct curl_handle *ch); +static bool try_fallback_GET_method (struct curl_handle *ch); static size_t header_cb (void *ptr, size_t size, size_t nmemb, void *opaque); +static size_t error_cb (char *ptr, size_t size, size_t nmemb, void *opaque); /* This lock protects access to the curl_handles vector below. */ static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; @@ -443,6 +445,7 @@ static int get_content_length_accept_range (struct curl_handle *ch) { CURLcode r; + long code; #ifdef HAVE_CURLINFO_CONTENT_LENGTH_DOWNLOAD_T curl_off_t o; #else @@ -469,7 +472,20 @@ get_content_length_accept_range (struct curl_handle *ch) display_curl_error (ch, r, "problem doing HEAD request to fetch size of URL [%s]", url); - return -1; + + /* Get the HTTP status code, if available. */ + r = curl_easy_getinfo (ch->c, CURLINFO_RESPONSE_CODE, &code); + if (r == CURLE_OK) + nbdkit_debug ("HTTP status code: %ld", code); + else + code = -1; + + /* S3 servers can return 403 Forbidden for HEAD but still respond + * to GET, so we give it a second chance in that case. + * https://github.com/kubevirt/containerized-data-importer/issues/2737#issuecom... + */ + if (code != 403 || !try_fallback_GET_method (ch)) + return -1; } /* Get the content length. */ @@ -520,6 +536,27 @@ get_content_length_accept_range (struct curl_handle *ch) return 0; } +static bool +try_fallback_GET_method (struct curl_handle *ch) +{ + CURLcode r; + + nbdkit_debug ("attempting to fetch headers using GET method"); + + curl_easy_setopt (ch->c, CURLOPT_HTTPGET, 1L); + curl_easy_setopt (ch->c, CURLOPT_HEADERFUNCTION, header_cb); + curl_easy_setopt (ch->c, CURLOPT_HEADERDATA, ch); + curl_easy_setopt (ch->c, CURLOPT_WRITEFUNCTION, error_cb); + curl_easy_setopt (ch->c, CURLOPT_WRITEDATA, ch); + r = curl_easy_perform (ch->c); + + /* We expect CURLE_WRITE_ERROR here, but CURLE_OK is possible too + * (eg if the remote has zero length). Other errors might happen + * but we ignore them since it is a fallback path. + */ + return r == CURLE_OK || r == CURLE_WRITE_ERROR; +} + static size_t header_cb (void *ptr, size_t size, size_t nmemb, void *opaque) { @@ -552,3 +589,13 @@ header_cb (void *ptr, size_t size, size_t nmemb, void *opaque) return realsize; } + +static size_t +error_cb (char *ptr, size_t size, size_t nmemb, void *opaque) +{ +#ifdef CURL_WRITEFUNC_ERROR + return CURL_WRITEFUNC_ERROR; +#else + return 0; /* in older curl, any size < requested will also be an error */ +#endif +} diff --git a/tests/test-curl-cookie-script.c b/tests/test-curl-cookie-script.c index 005e73ed0..95b602849 100644 --- a/tests/test-curl-cookie-script.c +++ b/tests/test-curl-cookie-script.c @@ -82,7 +82,7 @@ main (int argc, char *argv[]) exit (77); #endif - sockpath = web_server ("disk", check_request); + sockpath = web_server ("disk", check_request, false); if (sockpath == NULL) { fprintf (stderr, "%s: could not start web server thread\n", program_name); exit (EXIT_FAILURE); diff --git a/tests/test-curl-head-forbidden.c b/tests/test-curl-head-forbidden.c new file mode 100644 index 000000000..16b1f0533 --- /dev/null +++ b/tests/test-curl-head-forbidden.c @@ -0,0 +1,134 @@ +/* nbdkit + * Copyright Red Hat + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Test curl plugin against a simulated webserver which responds with + * 403 Forbidden to HEAD requests, but allows the GET method. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> +#include <errno.h> +#include <sys/stat.h> + +#include <libnbd.h> + +#include "cleanup.h" +#include "web-server.h" + +#include "test.h" + +static char buf[1024]; + +int +main (int argc, char *argv[]) +{ + struct stat statbuf; + const char *sockpath; + struct nbd_handle *nbd; + CLEANUP_FREE char *usp_param = NULL; + int64_t size; + +#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH + fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n", + program_name); + exit (77); +#endif + + if (stat ("disk", &statbuf) == -1) { + if (errno == ENOENT) { + fprintf (stderr, "%s: test skipped because \"disk\" is missing\n", + program_name); + exit (77); + } + perror ("disk"); + exit (EXIT_FAILURE); + } + + sockpath = web_server ("disk", NULL, true); + if (sockpath == NULL) { + fprintf (stderr, "%s: could not start web server thread\n", program_name); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Start nbdkit. */ + if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + if (nbd_connect_command (nbd, + (char *[]) { + "nbdkit", "-s", "--exit-with-parent", "-v", + "curl", + "-D", "curl.verbose=1", + "http://localhost/disk", + usp_param, /* unix-socket-path=... */ + NULL }) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Check the size is expected. */ + size = nbd_get_size (nbd); + if (size == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (size != statbuf.st_size) { + fprintf (stderr, "%s: incorrect export size, " + "expected: %" PRIu64 " actual: %" PRIi64 "\n", + program_name, + (uint64_t) statbuf.st_size, size); + exit (EXIT_FAILURE); + } + + /* Make a request. */ + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +} diff --git a/tests/test-curl-header-script.c b/tests/test-curl-header-script.c index 335c96494..e1eb4dacb 100644 --- a/tests/test-curl-header-script.c +++ b/tests/test-curl-header-script.c @@ -104,7 +104,7 @@ main (int argc, char *argv[]) exit (77); #endif - sockpath = web_server ("disk", check_request); + sockpath = web_server ("disk", check_request, false); if (sockpath == NULL) { fprintf (stderr, "%s: could not start web server thread\n", program_name); exit (EXIT_FAILURE); diff --git a/tests/test-curl.c b/tests/test-curl.c index ac267a175..e6134d78d 100644 --- a/tests/test-curl.c +++ b/tests/test-curl.c @@ -82,7 +82,7 @@ main (int argc, char *argv[]) exit (77); #endif - sockpath = web_server ("disk", check_request); + sockpath = web_server ("disk", check_request, false); if (sockpath == NULL) { fprintf (stderr, "%s: could not start web server thread\n", program_name); exit (EXIT_FAILURE); diff --git a/tests/test-gzip-curl.c b/tests/test-gzip-curl.c index a4133aee5..a3fcb88bf 100644 --- a/tests/test-gzip-curl.c +++ b/tests/test-gzip-curl.c @@ -68,7 +68,7 @@ main (int argc, char *argv[]) exit (77); #endif - sockpath = web_server (disk, NULL); + sockpath = web_server (disk, NULL, false); if (sockpath == NULL) { fprintf (stderr, "test-gzip-curl: could not start web server thread\n"); exit (EXIT_FAILURE); diff --git a/tests/test-retry-request-mirror.c b/tests/test-retry-request-mirror.c index 276460b1f..cf42c5964 100644 --- a/tests/test-retry-request-mirror.c +++ b/tests/test-retry-request-mirror.c @@ -72,7 +72,7 @@ main (int argc, char *argv[]) exit (77); } - sockpath = web_server ("disk" /* not used but must be set */, NULL); + sockpath = web_server ("disk" /* not used but must be set */, NULL, false); if (sockpath == NULL) { fprintf (stderr, "%s: could not start web server thread\n", argv[0]); exit (EXIT_FAILURE); diff --git a/tests/test-tar-gzip-curl.c b/tests/test-tar-gzip-curl.c index ec0035b80..bbfa02f93 100644 --- a/tests/test-tar-gzip-curl.c +++ b/tests/test-tar-gzip-curl.c @@ -68,7 +68,7 @@ main (int argc, char *argv[]) exit (77); #endif - sockpath = web_server (disk, NULL); + sockpath = web_server (disk, NULL, false); if (sockpath == NULL) { fprintf (stderr, "test-tar-gzip-curl: could not start web server thread\n"); exit (EXIT_FAILURE); diff --git a/tests/test-tar-xz-curl.c b/tests/test-tar-xz-curl.c index ff92dda28..01d548bb5 100644 --- a/tests/test-tar-xz-curl.c +++ b/tests/test-tar-xz-curl.c @@ -68,7 +68,7 @@ main (int argc, char *argv[]) exit (77); #endif - sockpath = web_server (disk, NULL); + sockpath = web_server (disk, NULL, false); if (sockpath == NULL) { fprintf (stderr, "test-tar-xz-curl: could not start web server thread\n"); exit (EXIT_FAILURE); diff --git a/tests/test-xz-curl.c b/tests/test-xz-curl.c index 52b37a0c7..af81cbac7 100644 --- a/tests/test-xz-curl.c +++ b/tests/test-xz-curl.c @@ -68,7 +68,7 @@ main (int argc, char *argv[]) exit (77); #endif - sockpath = web_server (disk, NULL); + sockpath = web_server (disk, NULL, false); if (sockpath == NULL) { fprintf (stderr, "test-xz-curl: could not start web server thread\n"); exit (EXIT_FAILURE); diff --git a/tests/web-server.c b/tests/web-server.c index dbc8bc845..ba85e3205 100644 --- a/tests/web-server.c +++ b/tests/web-server.c @@ -67,16 +67,19 @@ static int fd = -1; static struct stat statbuf; static char request[16384]; static check_request_t check_request; +static bool head_fails_with_403 = false; static void *start_web_server (void *arg); static void handle_requests (int s); static void handle_file_request (int s, enum method method); static void handle_mirror_redirect_request (int s); static void handle_mirror_data_request (int s, enum method method, char byte); +static void send_403_forbidden (int s); static void send_404_not_found (int s); static void send_405_method_not_allowed (int s); static void send_500_internal_server_error (int s); static void xwrite (int s, const char *buf, size_t len); +static void xwrite_allow_epipe (int s, const char *buf, size_t len); static void xpread (char *buf, size_t count, off_t offset); static void @@ -99,7 +102,8 @@ ignore_sigpipe (void) } const char * -web_server (const char *filename, check_request_t _check_request) +web_server (const char *filename, check_request_t _check_request, + bool _head_fails_with_403) { struct sockaddr_un addr; pthread_t thread; @@ -108,6 +112,7 @@ web_server (const char *filename, check_request_t _check_request) ignore_sigpipe (); check_request = _check_request; + head_fails_with_403 = _head_fails_with_403; /* Open the file. */ fd = open (filename, O_RDONLY|O_CLOEXEC); @@ -250,6 +255,11 @@ handle_requests (int s) } memcpy (path, &request[5], n); path[n] = '\0'; + if (head_fails_with_403) { + send_403_forbidden (s); + eof = true; + break; + } } else if (strncmp (request, "GET ", 4) == 0) { method = GET; @@ -361,7 +371,14 @@ handle_file_request (int s, enum method method) } xpread (data, length, offset); - xwrite (s, data, length); + if (!head_fails_with_403 || offset != 0) + xwrite (s, data, length); + else + /* In the special case where we are testing the fallback from HEAD + * request case, the curl plugin will issue a GET for the whole + * data, but not read it all. Ignore EPIPE errors here. + */ + xwrite_allow_epipe (s, data, length); free (data); } @@ -449,6 +466,16 @@ handle_mirror_data_request (int s, enum method method, char byte) free (data); } +static void +send_403_forbidden (int s) +{ + const char response[] = + "HTTP/1.1 403 Forbidden\r\n" + "Connection: close\r\n" + "\r\n"; + xwrite (s, response, strlen (response)); +} + static void send_404_not_found (int s) { @@ -498,6 +525,26 @@ xwrite (int s, const char *buf, size_t len) } } +static void +xwrite_allow_epipe (int s, const char *buf, size_t len) +{ + ssize_t r; + + while (len > 0) { + r = write (s, buf, len); + if (r == -1) { + if (errno != EPIPE) { + perror ("write"); + exit (EXIT_FAILURE); + } + else + return; + } + buf += r; + len -= r; + } +} + static void xpread (char *buf, size_t count, off_t offset) { diff --git a/tests/web-server.h b/tests/web-server.h index 36f0ea1e1..0961f4c5d 100644 --- a/tests/web-server.h +++ b/tests/web-server.h @@ -43,6 +43,8 @@ #ifndef NBDKIT_WEB_SERVER_H #define NBDKIT_WEB_SERVER_H +#include <stdbool.h> + /* Starts a web server in a background thread. The web server will * serve 'filename' (only) - the URL in requests is ignored. * @@ -57,10 +59,15 @@ * The optional check_request function is called when the request is * received (note: not in the main thread) and can be used to perform * checks for example that particular headers were sent. + * + * If head_fails_with_403 == true then we simulate a server that + * responds to GET but fails HEAD with 403 errors, see: + * https://github.com/kubevirt/containerized-data-importer/issues/2737#issuecom... */ typedef void (*check_request_t) (const char *request); extern const char *web_server (const char *filename, - check_request_t check_request) + check_request_t check_request, + bool head_fails_with_403) __attribute__ ((__nonnull__ (1))); #endif /* NBDKIT_WEB_SERVER_H */ diff --git a/.gitignore b/.gitignore index e62c23d8b..49af3998f 100644 --- a/.gitignore +++ b/.gitignore @@ -122,6 +122,7 @@ plugins/*/*.3 /tests/test-curl /tests/test-curl-cookie-script /tests/test-curl-header-script +/tests/test-curl-head-forbidden /tests/test-data /tests/test-delay /tests/test-exit-with-parent -- 2.40.1
only superficial comments: On 6/6/23 13:22, Richard W.M. Jones wrote:
Some servers do not support HEAD for requesting the headers. If the HEAD request fails, fallback to using the GET method, abandoning the transfer as soon as possible after the headers have been received.
Fixes: https://github.com/kubevirt/containerized-data-importer/issues/2737 --- tests/Makefile.am | 23 +++++ plugins/curl/pool.c | 49 ++++++++++- tests/test-curl-cookie-script.c | 2 +- tests/test-curl-head-forbidden.c | 134 ++++++++++++++++++++++++++++++ tests/test-curl-header-script.c | 2 +- tests/test-curl.c | 2 +- tests/test-gzip-curl.c | 2 +- tests/test-retry-request-mirror.c | 2 +- tests/test-tar-gzip-curl.c | 2 +- tests/test-tar-xz-curl.c | 2 +- tests/test-xz-curl.c | 2 +- tests/web-server.c | 51 +++++++++++- tests/web-server.h | 9 +- .gitignore | 1 + 14 files changed, 271 insertions(+), 12 deletions(-)
diff --git a/tests/Makefile.am b/tests/Makefile.am index 165d61aff..6694e409e 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -655,6 +655,7 @@ EXTRA_DIST += \ $(NULL) LIBGUESTFS_TESTS += test-curl LIBNBD_TESTS += \ + test-curl-head-forbidden \ test-curl-header-script \ test-curl-cookie-script \ $(NULL) @@ -683,6 +684,28 @@ test_curl_LDADD = \ $(LIBGUESTFS_LIBS) \ $(NULL)
+test_curl_head_forbidden_SOURCES = \ + test-curl-head-forbidden.c \ + web-server.c \ + web-server.h \ + $(NULL) +test_curl_head_forbidden_CPPFLAGS = \ + -I$(top_srcdir)/common/include \ + -I$(top_srcdir)/common/utils \ + $(NULL) +test_curl_head_forbidden_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(LIBNBD_CFLAGS) \ + $(PTHREAD_CFLAGS) \ + $(NULL) +test_curl_head_forbidden_LDFLAGS = \ + $(top_builddir)/common/utils/libutils.la \ + $(PTHREAD_LIBS) \ + $(NULL) +test_curl_head_forbidden_LDADD = \ + $(LIBNBD_LIBS) \ + $(NULL) + test_curl_header_script_SOURCES = \ test-curl-header-script.c \ web-server.c \ diff --git a/plugins/curl/pool.c b/plugins/curl/pool.c index 2af5461a2..c73d52293 100644 --- a/plugins/curl/pool.c +++ b/plugins/curl/pool.c @@ -76,7 +76,9 @@ static int debug_cb (CURL *handle, curl_infotype type, static size_t write_cb (char *ptr, size_t size, size_t nmemb, void *opaque); static size_t read_cb (void *ptr, size_t size, size_t nmemb, void *opaque); static int get_content_length_accept_range (struct curl_handle *ch); +static bool try_fallback_GET_method (struct curl_handle *ch); static size_t header_cb (void *ptr, size_t size, size_t nmemb, void *opaque); +static size_t error_cb (char *ptr, size_t size, size_t nmemb, void *opaque);
/* This lock protects access to the curl_handles vector below. */ static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; @@ -443,6 +445,7 @@ static int get_content_length_accept_range (struct curl_handle *ch) { CURLcode r; + long code; #ifdef HAVE_CURLINFO_CONTENT_LENGTH_DOWNLOAD_T curl_off_t o; #else @@ -469,7 +472,20 @@ get_content_length_accept_range (struct curl_handle *ch) display_curl_error (ch, r, "problem doing HEAD request to fetch size of URL [%s]", url); - return -1; + + /* Get the HTTP status code, if available. */ + r = curl_easy_getinfo (ch->c, CURLINFO_RESPONSE_CODE, &code); + if (r == CURLE_OK) + nbdkit_debug ("HTTP status code: %ld", code); + else + code = -1; + + /* S3 servers can return 403 Forbidden for HEAD but still respond + * to GET, so we give it a second chance in that case. + * https://github.com/kubevirt/containerized-data-importer/issues/2737#issuecom... + */ + if (code != 403 || !try_fallback_GET_method (ch)) + return -1; }
/* Get the content length. */ @@ -520,6 +536,27 @@ get_content_length_accept_range (struct curl_handle *ch) return 0; }
+static bool +try_fallback_GET_method (struct curl_handle *ch) +{ + CURLcode r; + + nbdkit_debug ("attempting to fetch headers using GET method"); + + curl_easy_setopt (ch->c, CURLOPT_HTTPGET, 1L); + curl_easy_setopt (ch->c, CURLOPT_HEADERFUNCTION, header_cb); + curl_easy_setopt (ch->c, CURLOPT_HEADERDATA, ch); + curl_easy_setopt (ch->c, CURLOPT_WRITEFUNCTION, error_cb); + curl_easy_setopt (ch->c, CURLOPT_WRITEDATA, ch); + r = curl_easy_perform (ch->c); + + /* We expect CURLE_WRITE_ERROR here, but CURLE_OK is possible too + * (eg if the remote has zero length). Other errors might happen + * but we ignore them since it is a fallback path. + */ + return r == CURLE_OK || r == CURLE_WRITE_ERROR; +} + static size_t header_cb (void *ptr, size_t size, size_t nmemb, void *opaque) { @@ -552,3 +589,13 @@ header_cb (void *ptr, size_t size, size_t nmemb, void *opaque)
return realsize; } + +static size_t +error_cb (char *ptr, size_t size, size_t nmemb, void *opaque) +{ +#ifdef CURL_WRITEFUNC_ERROR + return CURL_WRITEFUNC_ERROR; +#else + return 0; /* in older curl, any size < requested will also be an error */ +#endif +} diff --git a/tests/test-curl-cookie-script.c b/tests/test-curl-cookie-script.c index 005e73ed0..95b602849 100644 --- a/tests/test-curl-cookie-script.c +++ b/tests/test-curl-cookie-script.c @@ -82,7 +82,7 @@ main (int argc, char *argv[]) exit (77); #endif
- sockpath = web_server ("disk", check_request); + sockpath = web_server ("disk", check_request, false); if (sockpath == NULL) { fprintf (stderr, "%s: could not start web server thread\n", program_name); exit (EXIT_FAILURE); diff --git a/tests/test-curl-head-forbidden.c b/tests/test-curl-head-forbidden.c new file mode 100644 index 000000000..16b1f0533 --- /dev/null +++ b/tests/test-curl-head-forbidden.c @@ -0,0 +1,134 @@ +/* nbdkit + * Copyright Red Hat + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Test curl plugin against a simulated webserver which responds with + * 403 Forbidden to HEAD requests, but allows the GET method. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> +#include <errno.h> +#include <sys/stat.h> + +#include <libnbd.h> + +#include "cleanup.h" +#include "web-server.h" + +#include "test.h" + +static char buf[1024]; + +int +main (int argc, char *argv[]) +{ + struct stat statbuf; + const char *sockpath; + struct nbd_handle *nbd; + CLEANUP_FREE char *usp_param = NULL; + int64_t size; + +#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH + fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n", + program_name); + exit (77); +#endif + + if (stat ("disk", &statbuf) == -1) { + if (errno == ENOENT) { + fprintf (stderr, "%s: test skipped because \"disk\" is missing\n", + program_name); + exit (77); + } + perror ("disk"); + exit (EXIT_FAILURE); + } + + sockpath = web_server ("disk", NULL, true); + if (sockpath == NULL) { + fprintf (stderr, "%s: could not start web server thread\n", program_name); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Start nbdkit. */ + if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + if (nbd_connect_command (nbd, + (char *[]) { + "nbdkit", "-s", "--exit-with-parent", "-v", + "curl", + "-D", "curl.verbose=1", + "http://localhost/disk", + usp_param, /* unix-socket-path=... */ + NULL }) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Check the size is expected. */ + size = nbd_get_size (nbd); + if (size == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (size != statbuf.st_size) { + fprintf (stderr, "%s: incorrect export size, " + "expected: %" PRIu64 " actual: %" PRIi64 "\n", + program_name, + (uint64_t) statbuf.st_size, size); + exit (EXIT_FAILURE); + } + + /* Make a request. */ + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +}
I take it this is a small customization (more precisely, lightly modified copy) of another (existent) test. The --find-copies-harder git formatting option is good for presenting new code in such terms, i.e. as a copy of another file + diffs on top. I figure the only relevant difference would be the last, "true", argument, for web_server().
diff --git a/tests/test-curl-header-script.c b/tests/test-curl-header-script.c index 335c96494..e1eb4dacb 100644 --- a/tests/test-curl-header-script.c +++ b/tests/test-curl-header-script.c @@ -104,7 +104,7 @@ main (int argc, char *argv[]) exit (77); #endif
- sockpath = web_server ("disk", check_request); + sockpath = web_server ("disk", check_request, false); if (sockpath == NULL) { fprintf (stderr, "%s: could not start web server thread\n", program_name); exit (EXIT_FAILURE); diff --git a/tests/test-curl.c b/tests/test-curl.c index ac267a175..e6134d78d 100644 --- a/tests/test-curl.c +++ b/tests/test-curl.c @@ -82,7 +82,7 @@ main (int argc, char *argv[]) exit (77); #endif
- sockpath = web_server ("disk", check_request); + sockpath = web_server ("disk", check_request, false); if (sockpath == NULL) { fprintf (stderr, "%s: could not start web server thread\n", program_name); exit (EXIT_FAILURE); diff --git a/tests/test-gzip-curl.c b/tests/test-gzip-curl.c index a4133aee5..a3fcb88bf 100644 --- a/tests/test-gzip-curl.c +++ b/tests/test-gzip-curl.c @@ -68,7 +68,7 @@ main (int argc, char *argv[]) exit (77); #endif
- sockpath = web_server (disk, NULL); + sockpath = web_server (disk, NULL, false); if (sockpath == NULL) { fprintf (stderr, "test-gzip-curl: could not start web server thread\n"); exit (EXIT_FAILURE); diff --git a/tests/test-retry-request-mirror.c b/tests/test-retry-request-mirror.c index 276460b1f..cf42c5964 100644 --- a/tests/test-retry-request-mirror.c +++ b/tests/test-retry-request-mirror.c @@ -72,7 +72,7 @@ main (int argc, char *argv[]) exit (77); }
- sockpath = web_server ("disk" /* not used but must be set */, NULL); + sockpath = web_server ("disk" /* not used but must be set */, NULL, false); if (sockpath == NULL) { fprintf (stderr, "%s: could not start web server thread\n", argv[0]); exit (EXIT_FAILURE); diff --git a/tests/test-tar-gzip-curl.c b/tests/test-tar-gzip-curl.c index ec0035b80..bbfa02f93 100644 --- a/tests/test-tar-gzip-curl.c +++ b/tests/test-tar-gzip-curl.c @@ -68,7 +68,7 @@ main (int argc, char *argv[]) exit (77); #endif
- sockpath = web_server (disk, NULL); + sockpath = web_server (disk, NULL, false); if (sockpath == NULL) { fprintf (stderr, "test-tar-gzip-curl: could not start web server thread\n"); exit (EXIT_FAILURE); diff --git a/tests/test-tar-xz-curl.c b/tests/test-tar-xz-curl.c index ff92dda28..01d548bb5 100644 --- a/tests/test-tar-xz-curl.c +++ b/tests/test-tar-xz-curl.c @@ -68,7 +68,7 @@ main (int argc, char *argv[]) exit (77); #endif
- sockpath = web_server (disk, NULL); + sockpath = web_server (disk, NULL, false); if (sockpath == NULL) { fprintf (stderr, "test-tar-xz-curl: could not start web server thread\n"); exit (EXIT_FAILURE); diff --git a/tests/test-xz-curl.c b/tests/test-xz-curl.c index 52b37a0c7..af81cbac7 100644 --- a/tests/test-xz-curl.c +++ b/tests/test-xz-curl.c @@ -68,7 +68,7 @@ main (int argc, char *argv[]) exit (77); #endif
- sockpath = web_server (disk, NULL); + sockpath = web_server (disk, NULL, false); if (sockpath == NULL) { fprintf (stderr, "test-xz-curl: could not start web server thread\n"); exit (EXIT_FAILURE); diff --git a/tests/web-server.c b/tests/web-server.c index dbc8bc845..ba85e3205 100644 --- a/tests/web-server.c +++ b/tests/web-server.c @@ -67,16 +67,19 @@ static int fd = -1; static struct stat statbuf; static char request[16384]; static check_request_t check_request; +static bool head_fails_with_403 = false;
static void *start_web_server (void *arg); static void handle_requests (int s); static void handle_file_request (int s, enum method method); static void handle_mirror_redirect_request (int s); static void handle_mirror_data_request (int s, enum method method, char byte); +static void send_403_forbidden (int s); static void send_404_not_found (int s); static void send_405_method_not_allowed (int s); static void send_500_internal_server_error (int s); static void xwrite (int s, const char *buf, size_t len); +static void xwrite_allow_epipe (int s, const char *buf, size_t len); static void xpread (char *buf, size_t count, off_t offset);
static void @@ -99,7 +102,8 @@ ignore_sigpipe (void) }
const char * -web_server (const char *filename, check_request_t _check_request) +web_server (const char *filename, check_request_t _check_request, + bool _head_fails_with_403)
We shouldn't start identifiers with an underscore.
{ struct sockaddr_un addr; pthread_t thread; @@ -108,6 +112,7 @@ web_server (const char *filename, check_request_t _check_request) ignore_sigpipe ();
check_request = _check_request; + head_fails_with_403 = _head_fails_with_403;
/* Open the file. */ fd = open (filename, O_RDONLY|O_CLOEXEC); @@ -250,6 +255,11 @@ handle_requests (int s) } memcpy (path, &request[5], n); path[n] = '\0'; + if (head_fails_with_403) { + send_403_forbidden (s); + eof = true; + break; + }
I'm not a fan of the pre-existent pattern where we both set "eof = true" and break out of the loop. It would have to be cleaned up first, in a different patch, but I'm not sure if we care enough.
} else if (strncmp (request, "GET ", 4) == 0) { method = GET; @@ -361,7 +371,14 @@ handle_file_request (int s, enum method method) }
xpread (data, length, offset); - xwrite (s, data, length); + if (!head_fails_with_403 || offset != 0) + xwrite (s, data, length); + else + /* In the special case where we are testing the fallback from HEAD + * request case, the curl plugin will issue a GET for the whole + * data, but not read it all. Ignore EPIPE errors here. + */ + xwrite_allow_epipe (s, data, length);
free (data); } @@ -449,6 +466,16 @@ handle_mirror_data_request (int s, enum method method, char byte) free (data); }
+static void +send_403_forbidden (int s) +{ + const char response[] = + "HTTP/1.1 403 Forbidden\r\n" + "Connection: close\r\n" + "\r\n"; + xwrite (s, response, strlen (response)); +} + static void send_404_not_found (int s) { @@ -498,6 +525,26 @@ xwrite (int s, const char *buf, size_t len) } }
+static void +xwrite_allow_epipe (int s, const char *buf, size_t len) +{ + ssize_t r; + + while (len > 0) { + r = write (s, buf, len); + if (r == -1) { + if (errno != EPIPE) { + perror ("write"); + exit (EXIT_FAILURE); + } + else + return; + }
exit() doesn't return, so I suggest removing the "else" (i.e., unnesting the "return").
+ buf += r; + len -= r; + } +} + static void xpread (char *buf, size_t count, off_t offset) { diff --git a/tests/web-server.h b/tests/web-server.h index 36f0ea1e1..0961f4c5d 100644 --- a/tests/web-server.h +++ b/tests/web-server.h @@ -43,6 +43,8 @@ #ifndef NBDKIT_WEB_SERVER_H #define NBDKIT_WEB_SERVER_H
+#include <stdbool.h> + /* Starts a web server in a background thread. The web server will * serve 'filename' (only) - the URL in requests is ignored. * @@ -57,10 +59,15 @@ * The optional check_request function is called when the request is * received (note: not in the main thread) and can be used to perform * checks for example that particular headers were sent. + * + * If head_fails_with_403 == true then we simulate a server that + * responds to GET but fails HEAD with 403 errors, see: + * https://github.com/kubevirt/containerized-data-importer/issues/2737#issuecom... */ typedef void (*check_request_t) (const char *request); extern const char *web_server (const char *filename, - check_request_t check_request) + check_request_t check_request, + bool head_fails_with_403) __attribute__ ((__nonnull__ (1)));
#endif /* NBDKIT_WEB_SERVER_H */
I'd prefer the introduction of this new parameter to be a separate patch. That would isolate much the boilerplate from the relevant feature addition.
diff --git a/.gitignore b/.gitignore index e62c23d8b..49af3998f 100644 --- a/.gitignore +++ b/.gitignore @@ -122,6 +122,7 @@ plugins/*/*.3 /tests/test-curl /tests/test-curl-cookie-script /tests/test-curl-header-script +/tests/test-curl-head-forbidden /tests/test-data /tests/test-delay /tests/test-exit-with-parent
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
On Tue, Jun 06, 2023 at 06:09:09PM +0200, Laszlo Ersek wrote:
only superficial comments:
On 6/6/23 13:22, Richard W.M. Jones wrote: [...]
diff --git a/tests/test-curl-head-forbidden.c b/tests/test-curl-head-forbidden.c new file mode 100644 index 000000000..16b1f0533 --- /dev/null +++ b/tests/test-curl-head-forbidden.c @@ -0,0 +1,134 @@ +/* nbdkit + * Copyright Red Hat + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Test curl plugin against a simulated webserver which responds with + * 403 Forbidden to HEAD requests, but allows the GET method. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> +#include <errno.h> +#include <sys/stat.h> + +#include <libnbd.h> + +#include "cleanup.h" +#include "web-server.h" + +#include "test.h" + +static char buf[1024]; + +int +main (int argc, char *argv[]) +{ + struct stat statbuf; + const char *sockpath; + struct nbd_handle *nbd; + CLEANUP_FREE char *usp_param = NULL; + int64_t size; + +#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH + fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n", + program_name); + exit (77); +#endif + + if (stat ("disk", &statbuf) == -1) { + if (errno == ENOENT) { + fprintf (stderr, "%s: test skipped because \"disk\" is missing\n", + program_name); + exit (77); + } + perror ("disk"); + exit (EXIT_FAILURE); + } + + sockpath = web_server ("disk", NULL, true); + if (sockpath == NULL) { + fprintf (stderr, "%s: could not start web server thread\n", program_name); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Start nbdkit. */ + if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + if (nbd_connect_command (nbd, + (char *[]) { + "nbdkit", "-s", "--exit-with-parent", "-v", + "curl", + "-D", "curl.verbose=1", + "http://localhost/disk", + usp_param, /* unix-socket-path=... */ + NULL }) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Check the size is expected. */ + size = nbd_get_size (nbd); + if (size == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (size != statbuf.st_size) { + fprintf (stderr, "%s: incorrect export size, " + "expected: %" PRIu64 " actual: %" PRIi64 "\n", + program_name, + (uint64_t) statbuf.st_size, size); + exit (EXIT_FAILURE); + } + + /* Make a request. */ + if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +}
I take it this is a small customization (more precisely, lightly modified copy) of another (existent) test. The --find-copies-harder git formatting option is good for presenting new code in such terms, i.e. as a copy of another file + diffs on top. I figure the only relevant difference would be the last, "true", argument, for web_server().
It's indeed a copy of test-curl-header-script.c. --find-copies-harder thinks it's: --- a/tests/test-curl-cookie-script.c +++ b/tests/test-curl-head-forbidden.c which makes sense as test-curl-header-script.c and test-curl-cookie-script.c were from a common source originally.
@@ -108,6 +112,7 @@ web_server (const char *filename, check_request_t _check_request) ignore_sigpipe ();
check_request = _check_request; + head_fails_with_403 = _head_fails_with_403;
/* Open the file. */ fd = open (filename, O_RDONLY|O_CLOEXEC); @@ -250,6 +255,11 @@ handle_requests (int s) } memcpy (path, &request[5], n); path[n] = '\0'; + if (head_fails_with_403) { + send_403_forbidden (s); + eof = true; + break; + }
I'm not a fan of the pre-existent pattern where we both set "eof = true" and break out of the loop. It would have to be cleaned up first, in a different patch, but I'm not sure if we care enough.
It's a test :-) However what do you suggest? Both statements are necessary because we have to break out of the inner for(;;) loop, run a tiny bit of common cleanup code (just a fprintf), and then break from the while(!eof) loop.
+static void +xwrite_allow_epipe (int s, const char *buf, size_t len) +{ + ssize_t r; + + while (len > 0) { + r = write (s, buf, len); + if (r == -1) { + if (errno != EPIPE) { + perror ("write"); + exit (EXIT_FAILURE); + } + else + return; + }
exit() doesn't return, so I suggest removing the "else" (i.e., unnesting the "return").
OK.
+ buf += r; + len -= r; + } +} + static void xpread (char *buf, size_t count, off_t offset) { diff --git a/tests/web-server.h b/tests/web-server.h index 36f0ea1e1..0961f4c5d 100644 --- a/tests/web-server.h +++ b/tests/web-server.h @@ -43,6 +43,8 @@ #ifndef NBDKIT_WEB_SERVER_H #define NBDKIT_WEB_SERVER_H
+#include <stdbool.h> + /* Starts a web server in a background thread. The web server will * serve 'filename' (only) - the URL in requests is ignored. * @@ -57,10 +59,15 @@ * The optional check_request function is called when the request is * received (note: not in the main thread) and can be used to perform * checks for example that particular headers were sent. + * + * If head_fails_with_403 == true then we simulate a server that + * responds to GET but fails HEAD with 403 errors, see: + * https://github.com/kubevirt/containerized-data-importer/issues/2737#issuecom... */ typedef void (*check_request_t) (const char *request); extern const char *web_server (const char *filename, - check_request_t check_request) + check_request_t check_request, + bool head_fails_with_403) __attribute__ ((__nonnull__ (1)));
#endif /* NBDKIT_WEB_SERVER_H */
I'd prefer the introduction of this new parameter to be a separate patch. That would isolate much the boilerplate from the relevant feature addition.
OK.
diff --git a/.gitignore b/.gitignore index e62c23d8b..49af3998f 100644 --- a/.gitignore +++ b/.gitignore @@ -122,6 +122,7 @@ plugins/*/*.3 /tests/test-curl /tests/test-curl-cookie-script /tests/test-curl-header-script +/tests/test-curl-head-forbidden /tests/test-data /tests/test-delay /tests/test-exit-with-parent
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 6/6/23 18:38, Richard W.M. Jones wrote:
On Tue, Jun 06, 2023 at 06:09:09PM +0200, Laszlo Ersek wrote:
On 6/6/23 13:22, Richard W.M. Jones wrote:
@@ -250,6 +255,11 @@ handle_requests (int s) } memcpy (path, &request[5], n); path[n] = '\0'; + if (head_fails_with_403) { + send_403_forbidden (s); + eof = true; + break; + }
I'm not a fan of the pre-existent pattern where we both set "eof = true" and break out of the loop. It would have to be cleaned up first, in a different patch, but I'm not sure if we care enough.
It's a test :-) However what do you suggest? Both statements are necessary because we have to break out of the inner for(;;) loop, run a tiny bit of common cleanup code (just a fprintf), and then break from the while(!eof) loop.
That argument is valid in case we are inside the inner ("for") loop, but here we are not (handle_requests() [tests/web-server.c]), as far as I can tell. The context with memcpy (path, &request[5], n); path[n] = '\0'; is outside (after) the "for" loop. The pre-patch code is misleading in this regard (it uses the same pattern of both eof=true plus break in three places). One option would be to use "eof=true" plus *continue*, rather than "break". That would: - prevent the rest of the *outer* (while) loop's body from running, just as the break does, - set "eof=true" consistently with the outer loop's design; IOW, rely on "eof" solely for exiting the outer loop, plus, after the "while" loop, "eof" would be true. Consider the following (pre-patch) cleanup (git diff --function-context):
diff --git a/tests/web-server.c b/tests/web-server.c index 9b37326c0540..7f9a15760a22 100644 --- a/tests/web-server.c +++ b/tests/web-server.c @@ -190,107 +190,107 @@ static void handle_requests (int s) { bool eof = false;
fprintf (stderr, "web server: accepted connection\n");
while (!eof) { size_t r, n, sz; enum method method; char path[128];
/* Read request until we see "\r\n\r\n" (end of headers) or EOF. */ n = 0; for (;;) { if (n >= sizeof request - 1 /* allow one byte for \0 */) { fprintf (stderr, "web server: request too long\n"); exit (EXIT_FAILURE); } sz = sizeof request - n - 1; r = read (s, &request[n], sz); if (r == -1) { perror ("read"); exit (EXIT_FAILURE); } if (r == 0) { eof = true; break; } n += r; request[n] = '\0'; if (strstr (request, "\r\n\r\n")) break; }
if (n == 0) continue;
fprintf (stderr, "web server: request:\n%s", request);
/* Call the optional user function to check the request. */ if (check_request) check_request (request);
/* Get the method and path fields from the first line. */ if (strncmp (request, "HEAD ", 5) == 0) { method = HEAD; n = strcspn (&request[5], " \n\t"); if (n >= sizeof path) { send_500_internal_server_error (s); eof = true; - break; + continue; } memcpy (path, &request[5], n); path[n] = '\0'; } else if (strncmp (request, "GET ", 4) == 0) { method = GET; n = strcspn (&request[4], " \n\t"); if (n >= sizeof path) { send_500_internal_server_error (s); eof = true; - break; + continue; } memcpy (path, &request[4], n); path[n] = '\0'; } else { send_405_method_not_allowed (s); eof = true; - break; + continue; }
fprintf (stderr, "web server: requested path: %s\n", path);
/* For testing retry-request + curl: * /mirror redirects round-robin to /mirror1, /mirror2, /mirror3 * /mirror1 returns a file of \x01 bytes * /mirror2 returns a file of \x02 bytes * /mirror3 returns 404 errors * Anything else returns a 500 error */ if (strcmp (path, "/mirror") == 0) handle_mirror_redirect_request (s); else if (strcmp (path, "/mirror1") == 0) handle_mirror_data_request (s, method, 1); else if (strcmp (path, "/mirror2") == 0) handle_mirror_data_request (s, method, 2); else if (strcmp (path, "/mirror3") == 0) { send_404_not_found (s); eof = true; } else if (strncmp (path, "/mirror", 7) == 0) { send_500_internal_server_error (s); eof = true; }
/* Otherwise it's a regular file request. 'path' is ignored, we * only serve a single file passed to web_server(). */ else handle_file_request (s, method);
fprintf (stderr, "web server: completed request\n"); }
fprintf (stderr, "web server: closing socket\n"); close (s); }
This would prevent the "web server: completed request" message from being logged, but that's already the case with: - the one continue statement we have in place anyway, - all the break statements the apove patch replaces. Those too prevent the "web server: completed request" message, so the new "continue" statements make no difference in that regard. If we wanted to print "web server: completed request" in any case, then we have at least two options. The ugly one (git diff --function-context --ignore-space-change):
diff --git a/tests/web-server.c b/tests/web-server.c index 9b37326c0540..7f7bd4424cd8 100644 --- a/tests/web-server.c +++ b/tests/web-server.c @@ -190,107 +190,109 @@ static void handle_requests (int s) { bool eof = false;
fprintf (stderr, "web server: accepted connection\n");
while (!eof) { size_t r, n, sz; enum method method; char path[128];
/* Read request until we see "\r\n\r\n" (end of headers) or EOF. */ n = 0; for (;;) { if (n >= sizeof request - 1 /* allow one byte for \0 */) { fprintf (stderr, "web server: request too long\n"); exit (EXIT_FAILURE); } sz = sizeof request - n - 1; r = read (s, &request[n], sz); if (r == -1) { perror ("read"); exit (EXIT_FAILURE); } if (r == 0) { eof = true; break; } n += r; request[n] = '\0'; if (strstr (request, "\r\n\r\n")) break; }
if (n == 0) continue;
fprintf (stderr, "web server: request:\n%s", request);
/* Call the optional user function to check the request. */ if (check_request) check_request (request);
/* Get the method and path fields from the first line. */ if (strncmp (request, "HEAD ", 5) == 0) { method = HEAD; n = strcspn (&request[5], " \n\t"); if (n >= sizeof path) { send_500_internal_server_error (s); eof = true; - break; } + else { memcpy (path, &request[5], n); path[n] = '\0'; } + } else if (strncmp (request, "GET ", 4) == 0) { method = GET; n = strcspn (&request[4], " \n\t"); if (n >= sizeof path) { send_500_internal_server_error (s); eof = true; - break; } + else { memcpy (path, &request[4], n); path[n] = '\0'; } + } else { send_405_method_not_allowed (s); eof = true; - break; }
+ if (!eof) { fprintf (stderr, "web server: requested path: %s\n", path);
/* For testing retry-request + curl: * /mirror redirects round-robin to /mirror1, /mirror2, /mirror3 * /mirror1 returns a file of \x01 bytes * /mirror2 returns a file of \x02 bytes * /mirror3 returns 404 errors * Anything else returns a 500 error */ if (strcmp (path, "/mirror") == 0) handle_mirror_redirect_request (s); else if (strcmp (path, "/mirror1") == 0) handle_mirror_data_request (s, method, 1); else if (strcmp (path, "/mirror2") == 0) handle_mirror_data_request (s, method, 2); else if (strcmp (path, "/mirror3") == 0) { send_404_not_found (s); eof = true; } else if (strncmp (path, "/mirror", 7) == 0) { send_500_internal_server_error (s); eof = true; }
/* Otherwise it's a regular file request. 'path' is ignored, we * only serve a single file passed to web_server(). */ else handle_file_request (s, method); - + } fprintf (stderr, "web server: completed request\n"); }
fprintf (stderr, "web server: closing socket\n"); close (s); }
And the nice one (git diff --function-context --ignore-space-change):
diff --git a/tests/web-server.c b/tests/web-server.c index 9b37326c0540..3ea785f749af 100644 --- a/tests/web-server.c +++ b/tests/web-server.c @@ -186,111 +186,121 @@ start_web_server (void *arg) } }
-static void -handle_requests (int s) +/* Returns "true" iff we should continue serving requests. */ +static bool +handle_request (int s) { - bool eof = false; - - fprintf (stderr, "web server: accepted connection\n"); - - while (!eof) { - size_t r, n, sz; enum method method; + size_t n; char path[128];
- /* Read request until we see "\r\n\r\n" (end of headers) or EOF. */ - n = 0; - for (;;) { - if (n >= sizeof request - 1 /* allow one byte for \0 */) { - fprintf (stderr, "web server: request too long\n"); - exit (EXIT_FAILURE); - } - sz = sizeof request - n - 1; - r = read (s, &request[n], sz); - if (r == -1) { - perror ("read"); - exit (EXIT_FAILURE); - } - if (r == 0) { - eof = true; - break; - } - n += r; - request[n] = '\0'; - if (strstr (request, "\r\n\r\n")) - break; - } - - if (n == 0) - continue; - - fprintf (stderr, "web server: request:\n%s", request); - /* Call the optional user function to check the request. */ if (check_request) check_request (request);
/* Get the method and path fields from the first line. */ if (strncmp (request, "HEAD ", 5) == 0) { method = HEAD; n = strcspn (&request[5], " \n\t"); if (n >= sizeof path) { send_500_internal_server_error (s); - eof = true; - break; + return false; } memcpy (path, &request[5], n); path[n] = '\0'; } else if (strncmp (request, "GET ", 4) == 0) { method = GET; n = strcspn (&request[4], " \n\t"); if (n >= sizeof path) { send_500_internal_server_error (s); - eof = true; - break; + return false; } memcpy (path, &request[4], n); path[n] = '\0'; } else { send_405_method_not_allowed (s); - eof = true; - break; + return false; }
fprintf (stderr, "web server: requested path: %s\n", path);
/* For testing retry-request + curl: * /mirror redirects round-robin to /mirror1, /mirror2, /mirror3 * /mirror1 returns a file of \x01 bytes * /mirror2 returns a file of \x02 bytes * /mirror3 returns 404 errors * Anything else returns a 500 error */ - if (strcmp (path, "/mirror") == 0) + if (strcmp (path, "/mirror") == 0) { handle_mirror_redirect_request (s); - else if (strcmp (path, "/mirror1") == 0) + return true; + } + if (strcmp (path, "/mirror1") == 0) { handle_mirror_data_request (s, method, 1); - else if (strcmp (path, "/mirror2") == 0) + return true; + } + if (strcmp (path, "/mirror2") == 0) { handle_mirror_data_request (s, method, 2); - else if (strcmp (path, "/mirror3") == 0) { + return true; + } + if (strcmp (path, "/mirror3") == 0) { send_404_not_found (s); - eof = true; + return false; } - else if (strncmp (path, "/mirror", 7) == 0) { + if (strncmp (path, "/mirror", 7) == 0) { send_500_internal_server_error (s); - eof = true; + return false; }
/* Otherwise it's a regular file request. 'path' is ignored, we * only serve a single file passed to web_server(). */ - else handle_file_request (s, method); + return true; +}
+static void +handle_requests (int s) +{ + bool eof = false; + + fprintf (stderr, "web server: accepted connection\n"); + + while (!eof) { + size_t r, n, sz; + + /* Read request until we see "\r\n\r\n" (end of headers) or EOF. */ + n = 0; + for (;;) { + if (n >= sizeof request - 1 /* allow one byte for \0 */) { + fprintf (stderr, "web server: request too long\n"); + exit (EXIT_FAILURE); + } + sz = sizeof request - n - 1; + r = read (s, &request[n], sz); + if (r == -1) { + perror ("read"); + exit (EXIT_FAILURE); + } + if (r == 0) { + eof = true; + break; + } + n += r; + request[n] = '\0'; + if (strstr (request, "\r\n\r\n")) + break; + } + + if (n == 0) + continue; + + fprintf (stderr, "web server: request:\n%s", request); + eof = !handle_request (s); fprintf (stderr, "web server: completed request\n"); }
fprintf (stderr, "web server: closing socket\n"); close (s); }
(I've built and tested the last one.) But this is probably too much churn for a test. Laszlo
Michael Henriksen pointed out an issue with this approach. If the web server is actually generating the content on the fly then it may send it as chunked encoding, and in HTTP/1.1 it's not required that the Content-Length field is present (since it may not be known when the server begins sending the content): https://www.rfc-editor.org/rfc/rfc2616#section-4.4 The only way to know the true length would be to download all the content, which we definitely don't want to do. I'm not totally clear if Content-Length, if present, must be valid. Curl source code seems to imply not: https://github.com/curl/curl/blob/6e4fedeef7ff2aefaa4841a4710d6b8a37c6ae7c/l... but maybe they are just being over-cautious? The RFC is a bit confusing. AWS itself _does_ send Content-Length and it appears to be valid ... So one approach might be to assume it is valid, which I believe is what the current patch series does. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Tue, Jun 06, 2023 at 08:06:50PM +0100, Richard W.M. Jones wrote:
Michael Henriksen pointed out an issue with this approach.
If the web server is actually generating the content on the fly then it may send it as chunked encoding, and in HTTP/1.1 it's not required that the Content-Length field is present (since it may not be known when the server begins sending the content):
https://www.rfc-editor.org/rfc/rfc2616#section-4.4
The only way to know the true length would be to download all the content, which we definitely don't want to do.
I'm not totally clear if Content-Length, if present, must be valid. Curl source code seems to imply not:
https://github.com/curl/curl/blob/6e4fedeef7ff2aefaa4841a4710d6b8a37c6ae7c/l...
but maybe they are just being over-cautious? The RFC is a bit confusing.
AWS itself _does_ send Content-Length and it appears to be valid ... So one approach might be to assume it is valid, which I believe is what the current patch series does.
After looking at the code this morning I'm fairly sure this is a non-problem, or to be more specific it is an existing (but not serious) problem in nbdkit-curl-plugin. Curl already deals with the case where a web server sends the Transfer-encoding: chunked header with a Content-Length header (which is apparently wrong) by setting the size to -1: https://github.com/curl/curl/blob/cd18e5c4645e3a3f8ca0feecefe5b1150405c8ff/l... which in curl-speak means the size is unknown: https://curl.se/libcurl/c/CURLINFO_CONTENT_LENGTH_DOWNLOAD_T.html and we already deal with that. Since it is already possible for a web server to set headers like this in a HEAD request, we're already dealing with this case (by rejecting the URL in the plugin). Adding this GET fallback method, which uses the common code path for fetching the size, does not make this any worse or better. I added a comment to this effect to the code, and split up the patch a lot more, and have pushed it in commits 51b86cff3..4b111535c Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On 6/7/23 14:59, Richard W.M. Jones wrote:
On Tue, Jun 06, 2023 at 08:06:50PM +0100, Richard W.M. Jones wrote:
Michael Henriksen pointed out an issue with this approach.
If the web server is actually generating the content on the fly then it may send it as chunked encoding, and in HTTP/1.1 it's not required that the Content-Length field is present (since it may not be known when the server begins sending the content):
https://www.rfc-editor.org/rfc/rfc2616#section-4.4
The only way to know the true length would be to download all the content, which we definitely don't want to do.
I'm not totally clear if Content-Length, if present, must be valid. Curl source code seems to imply not:
https://github.com/curl/curl/blob/6e4fedeef7ff2aefaa4841a4710d6b8a37c6ae7c/l...
but maybe they are just being over-cautious? The RFC is a bit confusing.
AWS itself _does_ send Content-Length and it appears to be valid ... So one approach might be to assume it is valid, which I believe is what the current patch series does.
After looking at the code this morning I'm fairly sure this is a non-problem, or to be more specific it is an existing (but not serious) problem in nbdkit-curl-plugin.
Curl already deals with the case where a web server sends the Transfer-encoding: chunked header with a Content-Length header (which is apparently wrong) by setting the size to -1:
https://github.com/curl/curl/blob/cd18e5c4645e3a3f8ca0feecefe5b1150405c8ff/l...
which in curl-speak means the size is unknown:
https://curl.se/libcurl/c/CURLINFO_CONTENT_LENGTH_DOWNLOAD_T.html
and we already deal with that. Since it is already possible for a web server to set headers like this in a HEAD request, we're already dealing with this case (by rejecting the URL in the plugin).
Sounds good!
Adding this GET fallback method, which uses the common code path for fetching the size, does not make this any worse or better.
I added a comment to this effect to the code, and split up the patch a lot more, and have pushed it in commits 51b86cff3..4b111535c
Thanks! Laszlo
On 6/6/23 21:06, Richard W.M. Jones wrote:
Michael Henriksen pointed out an issue with this approach.
If the web server is actually generating the content on the fly then it may send it as chunked encoding, and in HTTP/1.1 it's not required that the Content-Length field is present (since it may not be known when the server begins sending the content):
https://www.rfc-editor.org/rfc/rfc2616#section-4.4
The only way to know the true length would be to download all the content, which we definitely don't want to do.
I'm not totally clear if Content-Length, if present, must be valid. Curl source code seems to imply not:
https://github.com/curl/curl/blob/6e4fedeef7ff2aefaa4841a4710d6b8a37c6ae7c/l...
but maybe they are just being over-cautious? The RFC is a bit confusing.
AWS itself _does_ send Content-Length and it appears to be valid ... So one approach might be to assume it is valid, which I believe is what the current patch series does.
I've been aware that some servers don't send Content-Length; you can notice such servers even through simple downloads in Firefox: the download manager can't estimate for you how much time is left until the download completes. You don't get a nice progress bar. I think it's fine if nbdkit's curl plugin just doesn't support such web servers. We can simply require a (valid) Content-Length field. I think that requirement would be similar to another requirement we already place: I think our curl plugin depends on the server supporting byte ranges. Without that, we have no random access to the file on the web server, and it's not possible to expose it as a "block device". Right? Laszlo
On Tue, Jun 06, 2023 at 12:22:12PM +0100, Richard W.M. Jones wrote:
I haven't been able to test it against a real server, but I modified our internal test web server so it can behave like this and added a test.
Fixes: https://github.com/kubevirt/containerized-data-importer/issues/2737
The bug reporter gave me a URL I could use and I have now tested it successfully. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
participants (2)
-
Laszlo Ersek -
Richard W.M. Jones