[PATCH nbdkit 0/3] server: Implement TCP fastopen (TFO)
TCP fastopen is a little-known feature for omitting part of the normal 3-way handshake. It's implemented in Linux (since ~2014), Windows (since 2016), macOS and FreeBSD, but this implementation probably only works on Linux. The heavy lifting is done by the kernel, and the userspace changes needed (on Linux, at least) are very minor. I'm mainly posting this to save the patch series. After going down a rabbit hole testing and benchmarking this, it seems to make no discernable difference that I can see, or any benefit must be under the margin of error from measurement. It's probably safe to include this feature, but on the other hand if it doesn't do anything it seems to just add unnecessary complexity. There is a corresponding libnbd series which also contains the benchmark program, which I'll post in a moment. Rich.
Since NBD makes several TCP connections (especially if using multi-conn, as you should), it may benefit from using TFO on platforms which support it. This commit simply adds new command line options for enabling and disabling the flag, a way to fetch the default from --dump-plugin, and updates the documentation. It does not (yet) implement the feature. I found these articles about TFO interesting: https://blog.apnic.net/2021/07/05/tcp-fast-open-not-so-fast/ https://candrews.integralblue.com/2019/03/the-sad-story-of-tcp-fast-open/ --- docs/nbdkit.pod | 23 ++++++++++++++++++++ docs/synopsis.txt | 1 + tests/Makefile.am | 2 ++ server/internal.h | 1 + server/options.h | 4 ++++ server/main.c | 11 ++++++++++ tests/test-tfo-default.sh | 46 +++++++++++++++++++++++++++++++++++++++ 7 files changed, 88 insertions(+) diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod index b088d4ef19..5b7b87741d 100644 --- a/docs/nbdkit.pod +++ b/docs/nbdkit.pod @@ -476,6 +476,29 @@ may require additional permissions, such as starting the server as root or raising the C<RLIMIT_MEMLOCK> (L<ulimit(1)> I<-l>) limit on the process. +=item B<--tfo> + +=item B<--no-tfo> + +Enable or disable TCP fastopen (TFO). When enabled, this allows +faster connections over TCP by omitting part of the TCP 3-way handshake. + +It is not supported for non-IP sockets (like Unix domain sockets). +The platform must also enable TFO. On Linux you must set +sysctl C<net.ipv4.tcp_fastopen> = 1. + +Currently only libnbd E<ge> 1.22 implements client side support +for TCP fastopen. + +The default is not specified and may depend on the platform and +version of nbdkit. You can find the default for the current version +of nbdkit using: + + $ nbdkit --dump-config | grep ^tfo + tfo_default=1 + +(or C<tfo_default=0>). + =item B<-t> THREADS =item B<--threads=>THREADS diff --git a/docs/synopsis.txt b/docs/synopsis.txt index ecbc6d5c50..ad55dc108b 100644 --- a/docs/synopsis.txt +++ b/docs/synopsis.txt @@ -9,6 +9,7 @@ nbdkit [-4|--ipv4-only] [-6|--ipv6-only] [-P|--pidfile PIDFILE] [-p|--port PORT] [--print-uri] [-r|--readonly] [--run 'COMMAND ARGS ...'] [--selinux-label=LABEL] [-s|--single] [--swap] + [--tfo|--no-tfo] [-t|--threads THREADS] [--timeout=TIMEOUT] [--tls=off|on|require] [--tls-certificates=/path/to/certificates] diff --git a/tests/Makefile.am b/tests/Makefile.am index 29c762b636..4bf9a4568f 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -312,6 +312,7 @@ TESTS += \ test-timeout.sh \ test-timeout-cancel.sh \ test-keepalive.sh \ + test-tfo-default.sh \ $(NULL) if !IS_WINDOWS TESTS += \ @@ -369,6 +370,7 @@ EXTRA_DIST += \ test-start.sh \ test-stdio.sh \ test-swap.sh \ + test-tfo-default.sh \ test-timeout.sh \ test-timeout.py \ test-timeout-cancel.sh \ diff --git a/server/internal.h b/server/internal.h index fb838c60d8..4413acecfb 100644 --- a/server/internal.h +++ b/server/internal.h @@ -125,6 +125,7 @@ extern const char *service_mode_string (enum service_mode); extern int tcpip_sock_af; extern struct debug_flag *debug_flags; extern const char *export_name; +extern bool fastopen; extern bool foreground; extern const char *ipaddr; extern bool keepalive; diff --git a/server/options.h b/server/options.h index 0bc8207caf..48c37be880 100644 --- a/server/options.h +++ b/server/options.h @@ -43,11 +43,13 @@ enum { DUMP_CONFIG_OPTION, DUMP_PLUGIN_OPTION, EXIT_WITH_PARENT_OPTION, + FASTOPEN_OPTION, FILTER_OPTION, KEEPALIVE_OPTION, LOG_OPTION, LONG_OPTIONS_OPTION, MASK_HANDSHAKE_OPTION, + NO_FASTOPEN_OPTION, NO_MC_OPTION, NO_SR_OPTION, PRINT_URI, @@ -92,6 +94,7 @@ static const struct option long_options[] = { { "no-meta-contexts", no_argument, NULL, NO_MC_OPTION }, { "no-sr", no_argument, NULL, NO_SR_OPTION }, { "no-structured-replies", no_argument, NULL, NO_SR_OPTION }, + { "no-tfo", no_argument, NULL, NO_FASTOPEN_OPTION }, { "old-style", no_argument, NULL, 'o' }, { "oldstyle", no_argument, NULL, 'o' }, { "pid-file", required_argument, NULL, 'P' }, @@ -109,6 +112,7 @@ static const struct option long_options[] = { { "single", no_argument, NULL, 's' }, { "stdin", no_argument, NULL, 's' }, { "swap", no_argument, NULL, SWAP_OPTION }, + { "tfo", no_argument, NULL, FASTOPEN_OPTION }, { "threads", required_argument, NULL, 't' }, { "timeout", required_argument, NULL, TIMEOUT_OPTION }, { "time-out", required_argument, NULL, TIMEOUT_OPTION }, diff --git a/server/main.c b/server/main.c index 0b558698cb..d6e93a67ff 100644 --- a/server/main.c +++ b/server/main.c @@ -80,6 +80,8 @@ #define main fuzzer_main #endif +#define FASTOPEN_DEFAULT 0 + static char *make_random_fifo (void); static struct backend *open_plugin_so (size_t i, const char *filename, int short_name); @@ -96,6 +98,7 @@ int tcpip_sock_af = AF_UNSPEC; /* -4, -6 */ struct debug_flag *debug_flags; /* -D */ bool exit_with_parent; /* --exit-with-parent */ const char *export_name; /* -e */ +bool fastopen = FASTOPEN_DEFAULT; /* --tfo, --no-tfo */ bool foreground; /* -f */ const char *ipaddr; /* -i */ bool keepalive; /* --keepalive */ @@ -224,6 +227,7 @@ dump_config (void) #endif printf ("%s=%s\n", "soext", SOEXT); printf ("%s=%s\n", "sysconfdir", sysconfdir); + printf ("%s=%d\n", "tfo_default", FASTOPEN_DEFAULT); #ifdef HAVE_GNUTLS printf ("tls=yes\n"); #else @@ -326,6 +330,13 @@ main (int argc, char *argv[]) } break; + case FASTOPEN_OPTION: + fastopen = true; + break; + case NO_FASTOPEN_OPTION: + fastopen = false; + break; + case FILTER_OPTION: { struct filter_filename *t; diff --git a/tests/test-tfo-default.sh b/tests/test-tfo-default.sh new file mode 100755 index 0000000000..7ca35166fe --- /dev/null +++ b/tests/test-tfo-default.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright Red Hat +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +# Test the --dump-plugin / tfo_default setting. + +source ./functions.sh +set -e +set -x + +out=tfo-default.out +rm -f $out +cleanup_fn rm -f $out + +nbdkit --dump-config > $out +# Check the key is present and the format is correct. We do +# not know what the default will be. +grep "^tfo_default=[01]$" $out -- 2.46.0
Actually implement the socket option. --- tests/Makefile.am | 2 ++ server/sockets.c | 12 +++++++++++ tests/test-tfo-unix.sh | 45 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 59 insertions(+) diff --git a/tests/Makefile.am b/tests/Makefile.am index 4bf9a4568f..dc5b6c9a6c 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -313,6 +313,7 @@ TESTS += \ test-timeout-cancel.sh \ test-keepalive.sh \ test-tfo-default.sh \ + test-tfo-unix.sh \ $(NULL) if !IS_WINDOWS TESTS += \ @@ -371,6 +372,7 @@ EXTRA_DIST += \ test-stdio.sh \ test-swap.sh \ test-tfo-default.sh \ + test-tfo-unix.sh \ test-timeout.sh \ test-timeout.py \ test-timeout-cancel.sh \ diff --git a/server/sockets.c b/server/sockets.c index 348da029a9..07b0cf4787 100644 --- a/server/sockets.c +++ b/server/sockets.c @@ -229,6 +229,18 @@ bind_tcpip_socket (sockets *socks) if (setsockopt (sock, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof opt) == -1) perror ("setsockopt: SO_REUSEADDR"); + if (fastopen) { + /* 5 is what dnsmasq uses. I've also seen 1 being used here. + * The manpages documentation says this is the maximum length of + * the SYN packet, which is wrong. + */ + const int qlen = 5; + + if (setsockopt (sock, IPPROTO_TCP, TCP_FASTOPEN, + &qlen, sizeof qlen) == -1) + perror ("setsockopt: TCP_FASTOPEN"); + } + #ifdef IPV6_V6ONLY if (a->ai_family == PF_INET6) { if (setsockopt (sock, IPPROTO_IPV6, IPV6_V6ONLY, &opt, sizeof opt) == -1) diff --git a/tests/test-tfo-unix.sh b/tests/test-tfo-unix.sh new file mode 100755 index 0000000000..36306d8b6d --- /dev/null +++ b/tests/test-tfo-unix.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright Red Hat +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +# Test the --tfo and --no-tfo options do nothing for non-supported +# Unix sockets. + +source ./functions.sh +set -e +set -x + +requires_run +requires_plugin null +requires_nbdinfo + +nbdkit --tfo null --run 'nbdinfo "$uri"' +nbdkit --no-tfo null --run 'nbdinfo "$uri"' -- 2.46.0
On platforms which support it, default to requesting TCP fastopen. Note that we do not document the default, so if this turns out to be a mistake we can simply revert this change. --- server/main.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/server/main.c b/server/main.c index d6e93a67ff..c5700b48c0 100644 --- a/server/main.c +++ b/server/main.c @@ -59,6 +59,10 @@ #include <sys/vsock.h> #endif +#ifdef HAVE_NETINET_TCP_H +#include <netinet/tcp.h> +#endif + #include <pthread.h> #include <dlfcn.h> @@ -80,7 +84,11 @@ #define main fuzzer_main #endif +#ifdef TCP_FASTOPEN +#define FASTOPEN_DEFAULT 1 +#else #define FASTOPEN_DEFAULT 0 +#endif static char *make_random_fifo (void); static struct backend *open_plugin_so (size_t i, const char *filename, -- 2.46.0
participants (1)
-
Richard W.M. Jones