[replying here, as I seem to have been dropped from cc on the subthread at https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o... - maybe I should subscribe to devel@ instead of seeing this second-hand... hmm - I can't even post to devel@ without subscribing, so now just sending this to libguestfs] [adding libguestfs - now that devel@ has helped point to a bug in nbdkit itself] On 3/18/20 4:49 AM, Richard W.M. Jones wrote: Dan Horak added: Reconstructing state from those command lines - we have a TLS test that operates 3 processes: client <=> nbdkit nbd <=> nbdkit example1 it looks like this particular test was checking a plain-text client connecting to nbdkit nbd, which in turn was connecting as a TLS client to nbdkit example1. I also know that 'nbdkit nbd' uses libnbd to support TLS, and that we have not fully implemented clean TLS teardown in libnbd - so it could be that the nbd side has told the example1 side that it will be shutting down soon, but due to unclean TLS library usage, is missing a poll() wakeup to realize that there will be no further response coming from the example1 side; while the example1 side is doing blocking I/O waiting for the nbd side to close the socket. The overall test that spawned both nbdkit processes in the background (tests/test-nbd-tls-psk.sh) has completed, though, stranding those two hung child processes without their original parent but letting 'make check' report testsuite success. As to why make is hanging, that is beyond me. Maybe something new in make 4.3 is detecting that we have stranded indirect processes, and is waiting for them to complete? Ideally, we need to fix libnbd TLS support to do cleaner shutdown. Pragmatically, nbdkit's tests/functions.sh start_nbdkit() function right now tries only a single: cleanup_fn kill "$(cat "$pidfile")" without waiting to see if it actually worked. We could probably turn that into a more robust kill_nbdkit() function that first tries the graceful SIGTERM, waits a few seconds to confirm whether the process actually died, and follows up with a harder SIGKILL as needed (preferably failing a test whenever SIGTERM was insufficient). It may not solve the bug in libnbd TLS shutdown, but would at least prevent stuck processes. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org