[PATCH nbdkit v2 0/4] Add -D nbdkit.environ=1, Windows fixes and quoting issues
by Richard W.M. Jones
This grew a bit from last time ...
The basic change to allow dumping the environment is the same, except
I enabled this for Windows (and checked that it does in fact work --
Windows has environment variables, who knew?)
The second patch fixes a problem I noticed while testing the above on
Windows.
The third & fourth patches escape debug strings similarly to C
strings, so for example if a debug string contains '\n' then it will
be printed with ...\n... and not (as currently) break the debug
message across multiple lines. Other non-printable characters are
handled in the obvious way too.
Rich.
1 year, 7 months
[PATCH nbdkit] server: Add -D nbdkit.environ=1 to dump the environment
by Richard W.M. Jones
This is not secure so should not be used routinely. Also we do not
attempt to filter environment variables, so even ones containing
multiple lines or special characters are all sent to nbdkit_debug.
The reason for adding this is to allow for debugging the new
nbd_set_socket_activation_name(3) API added in libnbd 1.16.
---
docs/nbdkit.pod | 10 ++++++++++
server/main.c | 28 ++++++++++++++++++++++++++++
2 files changed, 38 insertions(+)
diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod
index 83cf6a11c..22350cc08 100644
--- a/docs/nbdkit.pod
+++ b/docs/nbdkit.pod
@@ -619,6 +619,16 @@ are numerous and not usually very interesting.
S<I<-D nbdkit.backend.controlpath=0>> suppresses the non-datapath
commands (config, open, close, can_write, etc.)
+=item B<-D nbdkit.environ=1>
+
+(not Windows)
+
+Print nbdkit's environment variables in the debug output at start up.
+This is insecure because environment variables may contain both
+sensitive and user-controlled information, so it should not be used
+routinely. But it is useful for tracking down problems related to
+environment variables.
+
=item B<-D nbdkit.tls.log=>N
Enable TLS logging. C<N> can be in the range 0 (no logging) to 99.
diff --git a/server/main.c b/server/main.c
index 1df5d69ac..7b6adc3cc 100644
--- a/server/main.c
+++ b/server/main.c
@@ -210,6 +210,26 @@ dump_config (void)
#endif
}
+#ifndef WIN32
+
+/* -D nbdkit.environ=1 to dump the environment at start up. */
+NBDKIT_DLL_PUBLIC int nbdkit_debug_environ;
+
+#ifndef HAVE_ENVIRON_DECL
+extern char **environ;
+#endif
+
+static void
+dump_environment (void)
+{
+ size_t i;
+
+ for (i = 0; environ[i]; ++i)
+ nbdkit_debug ("%s", environ[i]);
+}
+
+#endif /* !WIN32 */
+
int
main (int argc, char *argv[])
{
@@ -662,6 +682,14 @@ main (int argc, char *argv[])
/* Check all debug flags were used, and free them. */
free_debug_flags ();
+#ifndef WIN32
+ /* Dump the environment if asked. This is the earliest we can do it
+ * because it uses a debug flag.
+ */
+ if (nbdkit_debug_environ && verbose)
+ dump_environment ();
+#endif
+
if (help) {
struct backend *b;
--
2.39.2
1 year, 7 months
[PATCH libnbd] tests: Add a test of nbd_{set, get}_socket_activation_name
by Richard W.M. Jones
Also test that the expected environment variable is set when
connecting to nbdkit.
This test requires nbdkit >= 1.35.2 which added support for -D
nbdkit.environ=1 to dump the environment, and will be skipped with
other versions.
---
.gitignore | 1 +
tests/Makefile.am | 8 +++
tests/socket-activation-name.c | 99 ++++++++++++++++++++++++++++++++++
3 files changed, 108 insertions(+)
diff --git a/.gitignore b/.gitignore
index df4b6ca249..fe7feffa5b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -254,6 +254,7 @@ Makefile.in
/tests/read-write-flag
/tests/server-death
/tests/shutdown-flags
+/tests/socket-activation-name
/tests/synch-parallel
/tests/synch-parallel-tls
/ublk/nbdublk
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 81a7db38f6..3a93251efe 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -231,6 +231,7 @@ check_PROGRAMS += \
meta-base-allocation \
closure-lifetimes \
pread-initialize \
+ socket-activation-name \
$(NULL)
TESTS += \
@@ -301,6 +302,7 @@ TESTS += \
meta-base-allocation \
closure-lifetimes \
pread-initialize \
+ socket-activation-name \
$(NULL)
# This test is compiled but not run because it requires a fixed port:
@@ -642,6 +644,12 @@ closure_lifetimes_LDADD = $(top_builddir)/lib/libnbd.la
pread_initialize_SOURCES = pread-initialize.c
pread_initialize_LDADD = $(top_builddir)/lib/libnbd.la
+socket_activation_name_SOURCES = \
+ socket-activation-name.c \
+ requires.c \
+ requires.h
+socket_activation_name_LDADD = $(top_builddir)/lib/libnbd.la
+
#----------------------------------------------------------------------
# Testing TLS support.
diff --git a/tests/socket-activation-name.c b/tests/socket-activation-name.c
new file mode 100644
index 0000000000..22902a4db9
--- /dev/null
+++ b/tests/socket-activation-name.c
@@ -0,0 +1,99 @@
+/* NBD client library in userspace
+ * Copyright Red Hat
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/* Test nbd_{set,get}_socket_activation_name API. */
+
+#undef NDEBUG
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <unistd.h>
+#include <assert.h>
+
+#include <libnbd.h>
+
+#include "requires.h"
+
+#define DEBUG_FILE "socket-activation-name.out"
+
+int
+main (int argc, char *argv[])
+{
+ struct nbd_handle *nbd;
+ char *r;
+
+ /* Test that this version of nbdkit supports -D nbdkit.environ=1
+ * added in nbdkit 1.35.2.
+ */
+ requires ("libnbd_sentinel=42 "
+ "nbdkit -fv -D nbdkit.environ=1 null --run true 2>&1 | "
+ "grep -sq 'debug.*libnbd_sentinel=42'");
+
+ nbd = nbd_create ();
+ if (nbd == NULL) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* Check firstly that it defaults to empty string. */
+ r = nbd_get_socket_activation_name (nbd);
+ assert (r != NULL);
+ assert (strcmp (r, "") == 0);
+ free (r);
+
+ /* Check we can set it to something and read that back. */
+ assert (nbd_set_socket_activation_name (nbd, "hello") == 0);
+ r = nbd_get_socket_activation_name (nbd);
+ assert (r != NULL);
+ assert (strcmp (r, "hello") == 0);
+ free (r);
+
+ /* Run external nbdkit and check the LISTEN_FDNAMES environment
+ * variable is set. We need to capture the debug output of nbdkit,
+ * hence the journey through the shell.
+ */
+ unlink (DEBUG_FILE);
+ char *cmd[] = {
+ "sh", "-c",
+ "exec 2> " DEBUG_FILE "\n"
+ "exec nbdkit -v -D nbdkit.environ=1 null 1024\n",
+ NULL
+ };
+ if (nbd_connect_systemd_socket_activation (nbd, cmd) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* Check the size is expected so we know we managed to connect to nbdkit. */
+ assert (nbd_get_size (nbd) == 1024);
+
+ nbd_close (nbd);
+
+ /* nbdkit doesn't know anything about socket activation names, but
+ * the LISTEN_FDNAMES environment variable should appear in the
+ * debug output.
+ */
+ assert (system ("grep 'debug.*LISTEN_FDNAMES=hello' " DEBUG_FILE) == 0);
+ unlink (DEBUG_FILE);
+
+ exit (EXIT_SUCCESS);
+}
--
2.39.2
1 year, 7 months
[libnbd PATCH 00/23] continue wrapping generated C code harder
by Laszlo Ersek
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2172516
Repo: https://gitlab.com/lersek/libnbd.git
Branch: wrap-2172516-wave4
This series is a continuation of the one noted in
<https://bugzilla.redhat.com/show_bug.cgi?id=2172516#c15>. It reduces
the widths of the following generated C files:
file width before width after
------------------ ------------ -----------
include/libnbd.h 81 79
lib/api.c 206 79
ocaml/nbd-c.c 108 79
python/libnbdmod.c 130 79
python/methods.h 104 80
The following generated C files remain for the next wave:
file width
------------------ -----
lib/states-run.c 102
lib/states.c 116
lib/states.h 123
Quite a few patches in the series are easier to review with "git show
-b" or "git show --color-words", due to code reindentation or other
relatively mechanical changes. (And I've just found out about "git show
--color-moved", which works really well on patch #08, 'generator/utils:
factor out "pr_buf"', for example.) Most patches where such display
tweaks are beneficial mention that fact.
These formatting tweaks are also the reason why I've pushed the branch
to my libnbd fork on gitlab (see URL & branch name above). The posted
patches should be possible to apply with git-am, for merging (thus, they
couldn't be formatted with the "-b" option, for example), but for
review, it may be easier to just fetch them (or to fetch them in
addition) and to show them locally.
Thanks for reviewing,
Laszlo
Laszlo Ersek (23):
generator: split LIBNBD_ATTRIBUTE_NONNULL replacement text
generator: wrap "python/methods.h" at 80 characters
generator: wrap "python/libnbdmod.c" at 80 characters
generator: split "int64_from_uint32_array" calls in "ocaml/nbd-c.c"
generator: wrap nbd_internal_ocaml_raise_closed() calls in
"ocaml/nbd-c.c"
generator: factor out "num_args" in "print_ocaml_binding"
generator: wrap byte-code compat function bodies in "ocaml/nbd-c.c"
generator/utils: factor out "pr_buf"
generator/utils: comment on "maxcol" handling in "pr_wrap"
generator/utils: add "pr_wrap_cstr"
generator/C: print_trace_enter: break "enter" to a common new line
generator/C: print_trace_enter: expand "function" shorthand in value
iters
generator/C: print_trace_enter: print Closure values uniformly with
others
generator/C: print_trace_enter: generate commas as suffixes, not
prefixes
generator/C: print_trace_enter: strip spaces around PRI* macros
generator/C: print_trace_enter: wrap "enter" debug() / debug_direct()
call
generator/C: print_wrapper: break apart the "ready" & "processing"
states
generator/C: print_wrapper: fold permitted states error message
generator/C: print_trace_leave: separate arg list from function
designator
generator/C: print_trace_leave: wrap "leave" debug() / debug_direct()
call
generator/C: print_flags_check: break guard condition to a new line
generator/C: print_wrapper: use helper variable for permitted state
check
generator/C: lib/api.c: indent arg list 2 spaces relative to function
name
generator/C.ml | 195 ++++++++++++--------
generator/OCaml.ml | 27 ++-
generator/Python.ml | 13 +-
generator/utils.ml | 85 ++++++++-
generator/utils.mli | 1 +
ocaml/helpers.c | 2 +-
ocaml/nbd-c.h | 3 +-
7 files changed, 225 insertions(+), 101 deletions(-)
base-commit: 0744f748ed900fb0537da9a5b6532538f3c78e83
1 year, 7 months
[PATCH nbdkit 0/2] allocators: sparse: Split the highly contended mutex
by Richard W.M. Jones
nbdkit-memory-plugin is a RAM disk, but it is quite a slow one because
of the lock we must acquire on the sparse array. Try to improve
performance by changing this mutex for a read-write lock.
The results are somewhat mixed.
Note that in the real world, workloads which constantly update the
metadata are assumed to be fairly rare. The test program added in the
first patch exercises metadata updates a lot more aggressively than
would be expected.
Rich.
1 year, 7 months
[PATCH v2] daemon/tar: support more compression methods.
by Zixun LI
Add support for lzma, zstd and auto detection of compression methods.
Removed "auto" setting and updated documentation since v1.
Signed-off-by: Zixun LI <admin(a)hifiphile.com>
---
daemon/tar.c | 8 ++++++++
generator/actions_core.ml | 12 ++++++------
2 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/daemon/tar.c b/daemon/tar.c
index 57b5b37ff..ff29fe500 100644
--- a/daemon/tar.c
+++ b/daemon/tar.c
@@ -159,6 +159,10 @@ do_tar_in (const char *dir, const char *compress, int xattrs, int selinux, int a
filter = " --xz";
else if (STREQ (compress, "lzop"))
filter = " --lzop";
+ else if (STREQ (compress, "lzma"))
+ filter = " --lzma";
+ else if (STREQ (compress, "zstd"))
+ filter = " --zstd";
else {
reply_with_error ("unknown compression type: %s", compress);
return -1;
@@ -312,6 +316,10 @@ do_tar_out (const char *dir, const char *compress, int numericowner,
filter = " --xz";
else if (STREQ (compress, "lzop"))
filter = " --lzop";
+ else if (STREQ (compress, "lzma"))
+ filter = " --lzma";
+ else if (STREQ (compress, "zstd"))
+ filter = " --zstd";
else {
reply_with_error ("unknown compression type: %s", compress);
return -1;
diff --git a/generator/actions_core.ml b/generator/actions_core.ml
index c8d9949ba..273ea1a7f 100644
--- a/generator/actions_core.ml
+++ b/generator/actions_core.ml
@@ -2710,9 +2710,9 @@ This command uploads and unpacks local file C<tarfile> into F<directory>.
The optional C<compress> flag controls compression. If not given,
then the input should be an uncompressed tar file. Otherwise one
of the following strings may be given to select the compression
-type of the input file: C<compress>, C<gzip>, C<bzip2>, C<xz>, C<lzop>.
-(Note that not all builds of libguestfs will support all of these
-compression types).
+type of the input file: C<compress>, C<gzip>, C<bzip2>, C<xz>, C<lzop>,
+C<lzma>, C<zstd>. (Note that not all builds of libguestfs will support
+all of these compression types).
The other optional arguments are:
@@ -2745,9 +2745,9 @@ it to local file C<tarfile>.
The optional C<compress> flag controls compression. If not given,
then the output will be an uncompressed tar file. Otherwise one
of the following strings may be given to select the compression
-type of the output file: C<compress>, C<gzip>, C<bzip2>, C<xz>, C<lzop>.
-(Note that not all builds of libguestfs will support all of these
-compression types).
+type of the output file: C<compress>, C<gzip>, C<bzip2>, C<xz>, C<lzop>,
+C<lzma>, C<zstd>. (Note that not all builds of libguestfs will support
+all of these compression types).
The other optional arguments are:
--
2.40.1
1 year, 7 months
