August 2020 - Libguestfs - libguestfs List Archives

[PATCH nbdkit 1/2] server: Call .get_ready before redirecting stdin/stdout to /dev/null.
by Richard W.M. Jones 05 Aug '20

05 Aug '20

VDDK plugin + --run was broken because of the following sequence of events: (1) After .config_complete, server redirects stdin/stdout to /dev/null. (2) Server then calls vddk_get_ready which reexecs. (3) We restart the server from the top, but with stdin/stdout redirected to /dev/null. So saved_stdin/saved_stdout save /dev/null. (4) run_command is called which "restores" /dev/null as stdin/stdout. (5) The output of the --run option is sent to /dev/null. In addition to fixing this problem with VDDK, it also makes general sense not to redirect stdin/stdout before calling .get_ready since this callback is supposed to be the last chance before the server daemonizes, and redirecting stdin/stdout is part of daemonization. Before this commit: $ ./nbdkit vddk libdir=tests/.libs /dev/null --run 'qemu-img info $nbd' [no output] After this commit: $ ./nbdkit vddk libdir=tests/.libs /dev/null --run 'qemu-img info $nbd' image: nbd://localhost:10809 file format: raw virtual size: 512 KiB (524288 bytes) disk size: unavailable One test (test-stdio.sh) made the assumption that stdin/stdout had already been redirected to /dev/null in .get_ready. I adjusted this test so it checks for redirection in .after_fork instead. --- server/main.c | 10 +++++----- tests/test-stdio-plugin.c | 9 +++++++++ 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/server/main.c b/server/main.c index 2f0aaf07..11c1614b 100644 --- a/server/main.c +++ b/server/main.c @@ -693,6 +693,11 @@ main (int argc, char *argv[]) /* Select the correct thread model based on config. */ lock_init_thread_model (); + /* Tell the plugin that we are about to start serving. This must be + * called before we change user, fork, or open any sockets. + */ + top->get_ready (top); + /* Sanitize stdin/stdout to /dev/null, after saving the originals * when needed. We are still single-threaded at this point, and * already checked that stdin/out were open, so we don't have to @@ -726,12 +731,7 @@ main (int argc, char *argv[]) perror ("open"); exit (EXIT_FAILURE); } - - /* Tell the plugin that we are about to start serving. This must be - * called before we change user, fork, or open any sockets. - */ configured = true; - top->get_ready (top); start_serving (); diff --git a/tests/test-stdio-plugin.c b/tests/test-stdio-plugin.c index 618eae83..86447278 100644 --- a/tests/test-stdio-plugin.c +++ b/tests/test-stdio-plugin.c @@ -122,6 +122,14 @@ stdio_config_complete (void) static int stdio_get_ready (void) +{ + bool check = stdio_check (); + assert (check == false); + return 0; +} + +static int +stdio_after_fork (void) { bool check = stdio_check (); assert (check == true); @@ -163,6 +171,7 @@ static struct nbdkit_plugin plugin = { .config = stdio_config, .config_complete = stdio_config_complete, .get_ready = stdio_get_ready, + .after_fork = stdio_after_fork, .open = stdio_open, .get_size = stdio_get_size, .pread = stdio_pread, -- 2.27.0

2 2

using SPDX tags in libnbd and/or nbdkit?
by Eric Blake 04 Aug '20

04 Aug '20

As the topic has been coming up in other projects recently (for example, see https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/39) should we update our files to either add SPDX tags, or even outright condense our existing copyright blurbs down to an SPDX tag plus documentation in a top-level file? The former (adding a line) is less problematic, the latter definitely requires buy-in from all contributors (although the list of contributors is relatively small, so it might be easy enough to get such buy-in). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org

2 1

nbdkit build failure in Koji
by Richard W.M. Jones 04 Aug '20

04 Aug '20

Hi Eric, I wonder if you have any thoughts about this build failure in tests/test-nozero.sh? https://koji.fedoraproject.org/koji/taskinfo?taskID=48259627 log: https://kojipkgs.fedoraproject.org//work/tasks/9762/48259762/build.log The error is “nozero6.img was trimmed by mistake”. I added “set -x” to the script earlier today so we can see exactly what's wrong, and it is that: ++ stat -c %b nozero2.img ++ stat -c %b nozero6.img + test 4096 '!=' 2048 + echo 'nozero6.img was trimmed by mistake' AFAICT what this means is that nozero2.img is growing during the test (from 2048 to 4096 blocks). When I run the test locally this file stays at 2048 blocks the whole time, and the test does not fail. One other unfortunate problem is that Fedora is having lots of toolchain problems right now (see Fedora devel list passim) so we cannot really be sure that *any* other tool we are using has been built correctly :-( I've already disabled LTO in qemu and libguestfs, but possibly there are other toolchain bugs. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

4 10

[libnbd PATCH 0/4] More nbdinfo fixes
by Eric Blake 03 Aug '20

03 Aug '20

This rounds up the remaining bugs that I originally identified in: https://www.redhat.com/archives/libguestfs/2020-July/msg00153.html Eric Blake (4): api: Permit export list APIs when Connected info: Support --list with serializing servers info: Fix --json output when list size != 1 info: Permit --size --json generator/API.ml | 6 +++--- info/info-list-json.sh | 9 +++++++++ info/info-list.sh | 9 +++++++++ info/nbdinfo.c | 41 ++++++++++++++++++++++++++++------------- 4 files changed, 49 insertions(+), 16 deletions(-) -- 2.28.0

2 5

[RFC nbdkit PATCH 0/4] Progress towards .list_exports
by Eric Blake 03 Aug '20

03 Aug '20

This takes Rich's API proposal and starts fleshing it out with enough meat that I was able to test 'nbdkit eval' advertising multiple exports with descriptions paired with 'qemu-nbd --list'. Eric Blake (3): server: Add exports list functions server: Prepare to use export list from plugin sh, eval: Add .list_exports support Richard W.M. Jones (1): server: Implement list_exports. docs/nbdkit-plugin.pod | 57 +++++++++- docs/nbdkit-protocol.pod | 7 +- plugins/eval/nbdkit-eval-plugin.pod | 2 + plugins/sh/nbdkit-sh-plugin.pod | 31 ++++++ include/nbdkit-common.h | 4 + include/nbdkit-filter.h | 12 +++ include/nbdkit-plugin.h | 3 + server/Makefile.am | 2 + tests/Makefile.am | 2 + server/internal.h | 6 ++ common/utils/cleanup.h | 3 + server/backend.c | 22 ++++ server/exports.c | 150 +++++++++++++++++++++++++++ server/filters.c | 9 ++ server/nbdkit.syms | 5 + server/plugins.c | 15 +++ server/protocol-handshake-newstyle.c | 80 ++++++++------ server/protocol-handshake.c | 24 +++++ common/utils/cleanup-nbdkit.c | 6 ++ plugins/sh/methods.h | 4 +- plugins/eval/eval.c | 2 + plugins/sh/methods.c | 91 ++++++++++++++++ plugins/sh/sh.c | 1 + plugins/sh/example.sh | 8 ++ tests/test-eval-exports.sh | 65 ++++++++++++ 25 files changed, 570 insertions(+), 41 deletions(-) create mode 100644 server/exports.c create mode 100755 tests/test-eval-exports.sh -- 2.28.0

2 7

[nbdkit] Failure in test-retry-size.sh
by Richard W.M. Jones 03 Aug '20

03 Aug '20

This happened on s390 in Koji, once. The key lines from the log are: + start_t=0 nbdkit: sh[1]: debug: retry 1: waiting 1 seconds before retrying nbdkit: sh[1]: debug: retry 1: waiting 1 seconds before retrying + end_t=1 Somehow nbdkit did wait 2 seconds (or at least, nbdkit_nanosleep (1, 0) was called twice by the retry filter). But in the bash script, time (as measured by $SECONDS) advanced from 0 to 1. I'm not sure I can explain that ... It should be possible to make the test more robust by increasing the retry time and subtracting a bit of slop in the test of end_t - start_t, but I'd like to come up with at least a plausible explanation for this first. Rich. + fail=0 + requires qemu-io --version + requires dd iflag=count_bytes + files='retry-size-open-count retry-size-fail' + rm -f retry-size-open-count retry-size-fail + cleanup_fn rm -f retry-size-open-count retry-size-fail + _cleanup_hook[${#_cleanup_hook[@]}]='rm -f retry-size-open-count retry-size-fail' + touch retry-size-open-count + start_t=0 + st=0 + nbdkit -v -U - sh - --filter=retry retry-delay=1 --run 'qemu-io -f raw -r $nbd \ -c "r 0 512" -c "r 512 512"' /builddir/build/BUILD/nbdkit-1.21.20/server/nbdkit -v -U - --filter=/builddir/build/BUILD/nbdkit-1.21.20/filters/retry/.libs/nbdkit-retry-filter.so "--run=qemu-io -f raw -r \$nbd \\ -c \"r 0 512\" -c \"r 512 512\"" -- /builddir/build/BUILD/nbdkit-1.21.20/plugins/sh/.libs/nbdkit-sh-plugin.so - retry-delay=1 nbdkit: debug: TLS disabled: could not load TLS certificates nbdkit: debug: registering plugin /builddir/build/BUILD/nbdkit-1.21.20/plugins/sh/.libs/nbdkit-sh-plugin.so nbdkit: debug: registered plugin /builddir/build/BUILD/nbdkit-1.21.20/plugins/sh/.libs/nbdkit-sh-plugin.so (name sh) nbdkit: debug: sh: load nbdkit: debug: load: tmpdir: /tmp/nbdkitpKY00D nbdkit: debug: registering filter /builddir/build/BUILD/nbdkit-1.21.20/filters/retry/.libs/nbdkit-retry-filter.so nbdkit: debug: registered filter /builddir/build/BUILD/nbdkit-1.21.20/filters/retry/.libs/nbdkit-retry-filter.so (name retry) nbdkit: debug: retry: load nbdkit: debug: retry: config key=script, value=- nbdkit: debug: sh: config key=script, value=- nbdkit: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh load nbdkit: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh load: status 2 nbdkit: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh magic_config_key nbdkit: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh magic_config_key: status 2 nbdkit: debug: retry: config key=retry-delay, value=1 nbdkit: debug: retry: config_complete nbdkit: debug: sh: config_complete nbdkit: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh config_complete nbdkit: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh config_complete: status 2 nbdkit: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh thread_model nbdkit: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh thread_model: status 2 nbdkit: debug: using thread model: serialize_all_requests nbdkit: debug: retry: get_ready nbdkit: debug: sh: get_ready nbdkit: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh get_ready nbdkit: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh get_ready: status 2 nbdkit: debug: bound to unix socket /tmp/nbdkitEtiOcF/socket nbdkit: debug: forked into background (new pid = 4095481) nbdkit: debug: retry: after_fork nbdkit: debug: sh: after_fork nbdkit: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh after_fork nbdkit: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh after_fork: status 2 nbdkit: debug: accepted connection nbdkit: sh[1]: debug: retry: preconnect nbdkit: sh[1]: debug: sh: preconnect nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh preconnect false "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh preconnect: status 2 nbdkit: sh[1]: debug: newstyle negotiation: flags: global 0x3 nbdkit: sh[1]: debug: newstyle negotiation: client flags: 0x3 nbdkit: sh[1]: debug: newstyle negotiation: NBD_OPT_STRUCTURED_REPLY: client requested structured replies nbdkit: sh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: client requested export '' nbdkit: sh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: set count: 1 nbdkit: sh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: set base:allocation nbdkit: sh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: replying with base:allocation id 1 nbdkit: sh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: reply complete nbdkit: sh[1]: debug: newstyle negotiation: NBD_OPT_GO: client requested export '' nbdkit: sh[1]: debug: retry: open readonly=0 exportname="" nbdkit: sh[1]: debug: sh: open readonly=0 exportname="" nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh open false "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh open: status 0 nbdkit: sh[1]: debug: sh: open returned handle 0x2aa2a0ee2a0 nbdkit: sh[1]: debug: retry: open returned handle 0x2aa2a10e5c0 nbdkit: sh[1]: debug: sh: prepare readonly=0 nbdkit: sh[1]: debug: retry: prepare readonly=0 nbdkit: sh[1]: debug: retry: get_size nbdkit: sh[1]: debug: sh: get_size nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh get_size "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh get_size: status 0 nbdkit: sh[1]: debug: retry: can_write nbdkit: sh[1]: debug: sh: can_write nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh can_write "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh can_write: status 2 nbdkit: sh[1]: debug: retry: can_zero nbdkit: sh[1]: debug: retry: can_fast_zero nbdkit: sh[1]: debug: retry: can_trim nbdkit: sh[1]: debug: retry: can_fua nbdkit: sh[1]: debug: retry: can_flush nbdkit: sh[1]: debug: sh: can_flush nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh can_flush "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh can_flush: status 2 nbdkit: sh[1]: debug: retry: is_rotational nbdkit: sh[1]: debug: sh: is_rotational nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh is_rotational "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh is_rotational: status 2 nbdkit: sh[1]: debug: retry: can_multi_conn nbdkit: sh[1]: debug: sh: can_multi_conn nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh can_multi_conn "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh can_multi_conn: status 2 nbdkit: sh[1]: debug: retry: can_cache nbdkit: sh[1]: debug: sh: can_cache nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh can_cache "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh can_cache: status 2 nbdkit: sh[1]: debug: retry: can_extents nbdkit: sh[1]: debug: sh: can_extents nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh can_extents "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh can_extents: status 2 nbdkit: sh[1]: debug: newstyle negotiation: flags: export 0x83 nbdkit: sh[1]: debug: newstyle negotiation: NBD_OPT_GO: ignoring NBD_INFO_* request 3 (NBD_INFO_BLOCK_SIZE) nbdkit: sh[1]: debug: handshake complete, processing requests serially nbdkit: sh[1]: debug: retry: pread count=512 offset=0 nbdkit: sh[1]: debug: sh: pread count=512 offset=0 nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh pread "" 512 0 nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh pread: status 1 nbdkit: sh[1]: error: /tmp/nbdkitpKY00D/inline-script.sh: too soon to read nbdkit: sh[1]: debug: pread failed: original errno = 5 nbdkit: sh[1]: debug: retry 1: waiting 1 seconds before retrying nbdkit: sh[1]: debug: sh: reopen readonly=0 exportname="" nbdkit: sh[1]: debug: sh: finalize nbdkit: sh[1]: debug: sh: close nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh close "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh close: status 2 nbdkit: sh[1]: debug: sh: open readonly=0 exportname="" nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh open false "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh open: status 0 nbdkit: sh[1]: debug: sh: open returned handle 0x2aa2a0ee2a0 nbdkit: sh[1]: debug: sh: prepare readonly=0 nbdkit: sh[1]: debug: sh: get_size nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh get_size "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh get_size: status 0 nbdkit: sh[1]: debug: sh: pread count=512 offset=0 nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh pread "" 512 0 nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh pread: status 0 nbdkit: sh[1]: debug: retry: pread count=512 offset=512 nbdkit: sh[1]: debug: pread failed: original errno = 5 nbdkit: sh[1]: debug: retry 1: waiting 1 seconds before retrying nbdkit: sh[1]: debug: sh: reopen readonly=0 exportname="" nbdkit: sh[1]: debug: sh: finalize nbdkit: sh[1]: debug: sh: close nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh close "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh close: status 2 nbdkit: sh[1]: debug: sh: open readonly=0 exportname="" nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh open false "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh open: status 0 nbdkit: sh[1]: debug: sh: open returned handle 0x2aa2a0ee2a0 nbdkit: sh[1]: debug: sh: prepare readonly=0 nbdkit: sh[1]: debug: sh: get_size nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh get_size "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh get_size: status 0 nbdkit: sh[1]: debug: sh: pread count=512 offset=512 nbdkit: sh[1]: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh pread "" 512 512 nbdkit: sh[1]: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh pread: status 0 read 512/512 bytes at offset 0 512 bytes, 1 ops; 0:00:01.03 (496.196279 bytes/sec and 0.9691 ops/sec) read 512/512 bytes at offset 512 512 bytes, 1 ops; 0:00:01.02 (503.602359 bytes/sec and 0.9836 ops/sec) nbdkit: sh[1]: debug: client sent NBD_CMD_DISC, closing connection nbdkit: sh[1]: debug: retry: finalize nbdkit: sh[1]: debug: sh: finalize nbdkit: debug: sh: unload plugin nbdkit: debug: calling: /tmp/nbdkitpKY00D/inline-script.sh unload nbdkit: debug: completed: /tmp/nbdkitpKY00D/inline-script.sh unload: status 2 nbdkit: debug: retry: unload filter + end_t=1 + '[' 1 -lt 2 ']' + echo './test-retry-size.sh: test ran too quickly' ./test-retry-size.sh: test ran too quickly + fail=1 + read open_count + '[' 3 -ne 3 ']' + '[' -e retry-size-fail ']' + exit 1 + _run_cleanup_hooks + local _status=1 _i + set +e + trap '' INT QUIT TERM EXIT ERR + echo ./test-retry-size.sh: run cleanup hooks: exit code 1 ./test-retry-size.sh: run cleanup hooks: exit code 1 + (( _i = 0 )) + (( _i < 1 )) + rm -f retry-size-open-count retry-size-fail + (( ++_i )) + (( _i < 1 )) + exit 1 FAIL test-retry-size.sh (exit status: 1) -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

2 2

[libnbd PATCH] generator: Trace return even when in wrong state
by Eric Blake 03 Aug '20

03 Aug '20

Place the tracing of return values after any generated 'out:' label, as it is important to see even the errors caused by calling a function in the wrong handle state. Fixes: 9df088bfa9 --- Multiple bugs found today, but only time to submit one patch so far. I found them all while trying to implement .list_exports in nbdkit. Bug 1: 'nbdinfo --list --json $uri' produces invalid JSON if there are 0 or more than 1 exports. We need to move the '"exports":[' and ']' into list_all_exports(), not list_one_export(), and manage correct trailing comma output. (We only tested nbdinfo with qemu-nbd, which only lists one export, explaining how we missed this) Bug 2: nbd_get_nr_list_exports() fails if the handle is still in READY state, even though there were exports listed and available in that case. I see no reason to forbid this during READY; but I've also mentioned that there are probably more changes coming to the way we do list export handling in libnbd to allow the reuse of a single handle rather than our current hacks of one handle per export. (We only tested nbdinfo with qemu-nbd, which generally does NOT allow NBD_OPT_GO on the '' export if it was advertising some other export; nbdkit is more forgiving, explaining why I hit a case where nbdinfo was calling this function in the wrong state) Bug 3: this patch. When a function fails due to use in the wrong state, the debug output was useless in telling me why. generator/C.ml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/generator/C.ml b/generator/C.ml index 4a0b45d..2c62d4e 100644 --- a/generator/C.ml +++ b/generator/C.ml @@ -542,13 +542,13 @@ let generate_lib_api_c () = pr " ret = nbd_unlocked_%s " name; print_arg_list ~wrap:true ~types:false ~handle:true args optargs; pr ";\n"; - if may_set_error then ( - pr "\n"; - print_trace_leave ret; - pr "\n" - ); + pr "\n"; if !need_out_label then pr " out:\n"; + if may_set_error then ( + print_trace_leave ret; + pr "\n" + ); if is_locked then ( pr " if (h->public_state != get_next_state (h))\n"; pr " h->public_state = get_next_state (h);\n"; -- 2.28.0

2 1