October 2020 - Libguestfs - libguestfs List Archives

virt-v2v: Virtio-Scsi patch
by Jean-Louis Dupond 01 Dec '20

01 Dec '20

Hi, I was trying to migrate some VM's to Virtio-SCSI block devices, as this gives some advantages. While checking the virt-v2v code, I found out that it supported Virtio-SCSI, but some bits were missing. In attachment a small patch that adds the missing bits :) Best regards Jean-Louis

2 4

[common PATCH 0/3] SELinux_relabel: relabel only if enforcing (RHBZ#1828952)
by Pino Toscano 16 Nov '20

16 Nov '20

Continuation/rework of: https://www.redhat.com/archives/libguestfs/2020-May/msg00020.html This is my approach, as I explained here: https://bugzilla.redhat.com/show_bug.cgi?id=1828952#c4 https://www.redhat.com/archives/libguestfs/2020-May/msg00035.html IOW: do not attempt to relabel if the guest is not enforcing, as it is either useless or may fail; few words more are in the comments of patch #3. Pino Toscano (2): mlcustomize: refactor reading from /etc/selinux/config mlcustomize: do not relabel if not enforcing (RHBZ#1828952) Richard W.M. Jones (1): mlcustomize: Refactor SELinux_relabel code. mlcustomize/SELinux_relabel.ml | 153 ++++++++++++++++++++------------- 1 file changed, 94 insertions(+), 59 deletions(-) -- 2.26.2

3 12

Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
by Nir Soffer 09 Nov '20

09 Nov '20

On Mon, Oct 12, 2020 at 9:05 AM Yedidyah Bar David <didi(a)redhat.com> wrote: > > Hi all, > > On Mon, Oct 12, 2020 at 5:17 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > > Above failed with: > > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > > vdsm.log has: > > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > > 2020-10-11 22:05:14,695-0400 INFO (jsonrpc/1) [api.host] FINISH > getJobs return={'jobs': {'05eaea44-7e4c-4442-9926-2bcb696520f1': > {'id': '05eaea44-7e4c-4442-9926-2bcb696520f1', 'status': 'failed', > 'description': 'sparsify_volume', 'job_type': 'storage', 'error': > {'code': 100, 'message': 'General Exception: (\'Command > [\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\', > \\\'--in-place\\\', > \\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67\\\'] > failed with rc=1 out=b\\\'3/12\\\\n{ "message": "libguestfs error: > guestfs_launch failed.\\\\\\\\nThis usually means the libguestfs > appliance failed to start or crashed.\\\\\\\\nDo:\\\\\\\\n export > LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\\\\\\nand run the command > again. For further information, read:\\\\\\\\n > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\\\\\nYou > can also run \\\\\\\'libguestfs-test-tool\\\\\\\' and post the > *complete* output\\\\\\\\ninto a bug report or message to the > libguestfs mailing list.", "timestamp": > "2020-10-11T22:05:08.397538670-04:00", "type": "error" }\\\\n\\\' > err=b"virt-sparsify: error: libguestfs error: guestfs_launch > failed.\\\\nThis usually means the libguestfs appliance failed to > start or crashed.\\\\nDo:\\\\n export LIBGUESTFS_DEBUG=1 > LIBGUESTFS_TRACE=1\\\\nand run the command again. For further > information, read:\\\\n > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\nYou > can also run \\\'libguestfs-test-tool\\\' and post the *complete* > output\\\\ninto a bug report or message to the libguestfs mailing > list.\\\\n\\\\nIf reporting bugs, run virt-sparsify with debugging > enabled and include the \\\\ncomplete output:\\\\n\\\\n virt-sparsify > -v -x [...]\\\\n"\',)'}}}, 'status': {'code': 0, 'message': 'Done'}} > from=::ffff:192.168.201.4,43318, > flow_id=365642f4-2fe2-45df-937a-f4ca435eea38 (api:54) > 2020-10-11 22:05:14,695-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer] > Return 'Host.getJobs' in bridge with > {'05eaea44-7e4c-4442-9926-2bcb696520f1': {'id': > '05eaea44-7e4c-4442-9926-2bcb696520f1', 'status': 'failed', > 'description': 'sparsify_volume', 'job_type': 'storage', 'error': > {'code': 100, 'message': 'General Exception: (\'Command > [\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\', > \\\'--in-place\\\', > \\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67\\\'] > failed with rc=1 out=b\\\'3/12\\\\n{ "message": "libguestfs error: > guestfs_launch failed.\\\\\\\\nThis usually means the libguestfs > appliance failed to start or crashed.\\\\\\\\nDo:\\\\\\\\n export > LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\\\\\\nand run the command > again. For further information, read:\\\\\\\\n > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\\\\\nYou > can also run \\\\\\\'libguestfs-test-tool\\\\\\\' and post the > *complete* output\\\\\\\\ninto a bug report or message to the > libguestfs mailing list.", "timestamp": > "2020-10-11T22:05:08.397538670-04:00", "type": "error" }\\\\n\\\' > err=b"virt-sparsify: error: libguestfs error: guestfs_launch > failed.\\\\nThis usually means the libguestfs appliance failed to > start or crashed.\\\\nDo:\\\\n export LIBGUESTFS_DEBUG=1 > LIBGUESTFS_TRACE=1\\\\nand run the command again. For further > information, read:\\\\n > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\nYou > can also run \\\'libguestfs-test-tool\\\' and post the *complete* > output\\\\ninto a bug report or message to the libguestfs mailing > list.\\\\n\\\\nIf reporting bugs, run virt-sparsify with debugging > enabled and include the \\\\ncomplete output:\\\\n\\\\n virt-sparsify > -v -x [...]\\\\n"\',)'}}} (__init__:356) > > /var/log/messages has: > > Oct 11 22:04:51 lago-basic-suite-master-host-0 kvm[80601]: 1 guest now active > Oct 11 22:05:06 lago-basic-suite-master-host-0 journal[80557]: Domain > id=1 name='guestfs-hl0ntvn92rtkk2u0' > uuid=05ea5a53-562f-49f8-a8ca-76b45c5325b4 is tainted: custom-argv > Oct 11 22:05:06 lago-basic-suite-master-host-0 journal[80557]: Domain > id=1 name='guestfs-hl0ntvn92rtkk2u0' > uuid=05ea5a53-562f-49f8-a8ca-76b45c5325b4 is tainted: host-cpu > Oct 11 22:05:06 lago-basic-suite-master-host-0 kvm[80801]: 2 guests now active > Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: > internal error: End of file from qemu monitor > Oct 11 22:05:08 lago-basic-suite-master-host-0 kvm[80807]: 1 guest now active > Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot > resolve symlink /tmp/libguestfseTG8xF/console.sock: No such file or > directoryKN<F3>L^? > Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot > resolve symlink /tmp/libguestfseTG8xF/guestfsd.sock: No such file or > directoryKN<F3>L^? > Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot > lookup default selinux label for /tmp/libguestfs1WkcF7/overlay1.qcow2 > Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot > lookup default selinux label for > /var/tmp/.guestfs-36/appliance.d/kernel > Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot > lookup default selinux label for > /var/tmp/.guestfs-36/appliance.d/initrd > Oct 11 22:05:08 lago-basic-suite-master-host-0 vdsm[74096]: ERROR Job > '05eaea44-7e4c-4442-9926-2bcb696520f1' failed#012Traceback (most > recent call last):#012 File > "/usr/lib/python3.6/site-packages/vdsm/jobs.py", line 159, in run#012 > self._run()#012 File > "/usr/lib/python3.6/site-packages/vdsm/storage/sdm/api/sparsify_volume.py", > line 57, in _run#012 > virtsparsify.sparsify_inplace(self._vol_info.path)#012 File > "/usr/lib/python3.6/site-packages/vdsm/virtsparsify.py", line 40, in > sparsify_inplace#012 commands.run(cmd)#012 File > "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 101, > in run#012 raise cmdutils.Error(args, p.returncode, out, > err)#012vdsm.common.cmdutils.Error: Command ['/usr/bin/virt-sparsify', > '--machine-readable', '--in-place', > '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67'] > failed with rc=1 out=b'3/12\n{ "message": "libguestfs error: > guestfs_launch failed.\\nThis usually means the libguestfs appliance > failed to start or crashed.\\nDo:\\n export LIBGUESTFS_DEBUG=1 > LIBGUESTFS_TRACE=1\\nand run the command again. For further > information, read:\\n > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\nYou > can also run \'libguestfs-test-tool\' and post the *complete* > output\\ninto a bug report or message to the libguestfs mailing > list.", "timestamp": "2020-10-11T22:05:08.397538670-04:00", "type": > "error" }\n' err=b"virt-sparsify: error: libguestfs error: > guestfs_launch failed.\nThis usually means the libguestfs appliance > failed to start or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 > LIBGUESTFS_TRACE=1\nand run the command again. For further > information, read:\n > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can > also run 'libguestfs-test-tool' and post the *complete* output\ninto a > bug report or message to the libguestfs mailing list.\n\nIf reporting > bugs, run virt-sparsify with debugging enabled and include the > \ncomplete output:\n\n virt-sparsify -v -x [...]\n" > > The next run of the job (480) did finish successfully. No idea if it > was already fixed by a patch, or is simply a random/env issue. I think this is env issue, we run on overloaded vms with small amount of memory. I have seen such radnom failures before. > Is it possible to pass LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 without > patching vdsm? Not sure. In any case, if this does not cause too much > excess debug logging, perhaps better always pass it, to help > retroactively analyze such failures. Unfortunately using LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 creates huge amount of logs so it is not possible to include these log for every run. Maybe we can show the logs only if the command failed, but vdsm infrastructure make this hard to do since we always log commnad stderr, even if the command was successful. Also I'm not sure this is a good idea to always run virt-sparsify in debug level. I think I posted a patch to run libguestfs-test-tool after virt-something errors for collecting data, but I cannot find it. Maybe this can be useful, so after every virt-something error we will have enough info about the issue in vdsm log. > Or, patch > virt-sparsify/libguestfs/whatever to always log at least enough > information on failure even without passing these. I think this is the right solution - when virt-something tool fails, it should log the reason for the failure - the error that caused the tool to fail. I'm not sure this is easy to do as the failing code run inside a special VM. Maybe the code running in the VM should log the output in a machine readable way, so once an error is detected virt-something can report the error as the reason, without running in debug mode. Nir

4 8

Re: [Libguestfs] Debugging libguestfs
by Richard W.M. Jones 31 Oct '20

31 Oct '20

On Sat, Oct 31, 2020 at 12:50:22PM -0400, Dave Houser wrote: > Hello Richard, > > I am having the same issue this user is having on their Kali box from this > post. Happens when trying to use guestmount which uses libguestfs. > > https://www.mail-archive.com/libguestfs@redhat.com/msg21239.html. > > I have tried all you recommended but nothing worked either. Below is the output > of my last strace command after removing /var/tmp.guest*. > I am not sure how to read the strace output. Can you shed any light on this? ... Please try running 'libguestfs-test-tool' and send the full output to the mailing list (you don't need to subscribe). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

1 0

Re: [Libguestfs] [libguestfs/nbdkit] Publish Rust bindings to crates.io (#4)
by Richard W.M. Jones 31 Oct '20

31 Oct '20

On Fri, Oct 30, 2020 at 03:47:36PM -0700, Alan Somers wrote: > https://crates.io/crates/nbdkit/ > This is great stuff, thanks for doing all this hard work. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

1 0

[PATCH libnbd 0/5] info: --map: Coalesce adjacent extents of the same type.
by Richard W.M. Jones 27 Oct '20

27 Oct '20

This adds coalescing of adjacent extents of the same type, as mentioned by Eric Blake in the commit message here: https://github.com/libguestfs/libnbd/commit/46072f6611f80245846a445766da071… The patch series is rather long because it detours through adding the <vector.h> library from nbdkit into libnbd and replacing ad hoc uses of realloc, char ** etc in various places. Rich.

2 8

[libnbd PATCH] info: Add support for new 'qemu-nbd -A' qemu:allocation-depth
by Eric Blake 27 Oct '20

27 Oct '20

A rather trivial decoding; we may enhance it further if qemu extends things to give an integer depth alongside its tri-state encoding. --- I'll wait to push this to libnbd until the counterpart qemu patches land upstream, although it looks like I've got positive review. info/nbdinfo.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/info/nbdinfo.c b/info/nbdinfo.c index 24ec129..1afdf98 100644 --- a/info/nbdinfo.c +++ b/info/nbdinfo.c @@ -780,6 +780,13 @@ extent_description (const char *metacontext, uint32_t type) case 1: return "dirty"; } } + else if (strcmp (metacontext, "qemu:allocation-depth") == 0) { + switch (type & 3) { + case 0: return "unallocated"; + case 1: return "local"; + case 2: return "backing"; + } + } return NULL; /* Don't know - description field will be omitted. */ } -- 2.29.0.rc1

2 3

CI for nbdkit
by alan somers 27 Oct '20

27 Oct '20

Would libguestfs be willing to enable CI for the nbdkit project? It's very easy to set up, at least for the Rust portion. I'm comfortable with both Cirrus CI and Github's native CI. I can get it started if you agree to do it. -Alan

3 6

[PATCH nbdkit] New filter: exitwhen: exit gracefully when an event occurs.
by Richard W.M. Jones 21 Oct '20

21 Oct '20

--- docs/nbdkit-captive.pod | 6 +- docs/nbdkit-service.pod | 1 + filters/exitlast/nbdkit-exitlast-filter.pod | 3 + filters/exitwhen/nbdkit-exitwhen-filter.pod | 156 ++++++ filters/ip/nbdkit-ip-filter.pod | 1 + filters/limit/nbdkit-limit-filter.pod | 1 + filters/rate/nbdkit-rate-filter.pod | 1 + configure.ac | 2 + filters/exitwhen/Makefile.am | 67 +++ tests/Makefile.am | 24 + filters/exitwhen/exitwhen.c | 473 ++++++++++++++++++ tests/test-exitwhen-file-already-created.sh | 45 ++ .../test-exitwhen-file-created-reject-new.sh | 88 ++++ tests/test-exitwhen-file-created-when-idle.sh | 63 +++ tests/test-exitwhen-file-created.sh | 91 ++++ tests/test-exitwhen-file-deleted.sh | 68 +++ tests/test-exitwhen-process-exits.sh | 69 +++ tests/test-exitwhen-script.sh | 70 +++ tests/test-exitwhen-pipe-closed.c | 81 +++ .gitignore | 1 + 20 files changed, 1309 insertions(+), 2 deletions(-) diff --git a/docs/nbdkit-captive.pod b/docs/nbdkit-captive.pod index 472f1951..0de963bc 100644 --- a/docs/nbdkit-captive.pod +++ b/docs/nbdkit-captive.pod @@ -15,8 +15,9 @@ You can run nbdkit under another process and have nbdkit reliably clean up. There are two techniques depending on whether you want nbdkit to start the other process (L</CAPTIVE NBDKIT>), or if you want the other process to start nbdkit (L</EXIT WITH PARENT>). Another way -is to have nbdkit exit after the last client connection, see -L<nbdkit-exitlast-filter(1)>. +is to have nbdkit exit after the last client connection +(L<nbdkit-exitlast-filter(1)>) or after an event +(L<nbdkit-exitwhen-filter(1)>). =head1 CAPTIVE NBDKIT @@ -154,6 +155,7 @@ reliably on all operating systems). L<nbdkit(1)>, L<nbdkit-exitlast-filter(1)>, +L<nbdkit-exitwhen-filter(1)>, L<prctl(2)> (on Linux), L<procctl(2)> (on FreeBSD). diff --git a/docs/nbdkit-service.pod b/docs/nbdkit-service.pod index 42fbedd8..cd76ef52 100644 --- a/docs/nbdkit-service.pod +++ b/docs/nbdkit-service.pod @@ -145,6 +145,7 @@ L</SOCKET ACTIVATION>. L<nbdkit(1)>, L<nbdkit-client(1)>, L<nbdkit-exitlast-filter(1)>, +L<nbdkit-exitwhen-filter(1)>, L<nbdkit-ip-filter(1)>, L<nbdkit-limit-filter(1)>, L<systemd(1)>, diff --git a/filters/exitlast/nbdkit-exitlast-filter.pod b/filters/exitlast/nbdkit-exitlast-filter.pod index 207ad4c8..1791d2cf 100644 --- a/filters/exitlast/nbdkit-exitlast-filter.pod +++ b/filters/exitlast/nbdkit-exitlast-filter.pod @@ -17,6 +17,8 @@ resources when nbdkit is not in use (see L<nbdkit-service(1)>). Another use is to ensure nbdkit exits after the client has finished (but see also nbdkit-captive(1) for other ways to do this). +To exit when an event occurs, try L<nbdkit-exitwhen-filter(1)>. + =head1 PARAMETERS There are no parameters specific to nbdkit-exitlast-filter. Any @@ -42,6 +44,7 @@ C<nbdkit-exitlast-filter> first appeared in nbdkit 1.20. =head1 SEE ALSO L<nbdkit(1)>, +L<nbdkit-exitwhen-filter(1)>, L<nbdkit-ip-filter(1)>, L<nbdkit-limit-filter(1)>, L<nbdkit-rate-filter(1)>, diff --git a/filters/exitwhen/nbdkit-exitwhen-filter.pod b/filters/exitwhen/nbdkit-exitwhen-filter.pod new file mode 100644 index 00000000..b9c51561 --- /dev/null +++ b/filters/exitwhen/nbdkit-exitwhen-filter.pod @@ -0,0 +1,156 @@ +=head1 NAME + +nbdkit-exitwhen-filter - exit gracefully when an event occurs + +=head1 SYNOPSIS + + nbdkit --filter=exitwhen PLUGIN + [exit-when-file-created=FILENAME] + [exit-when-file-deleted=FILENAME] + [exit-when-pipe-closed=FD] + [exit-when-process-exits=PID] + [exit-when-script=SCRIPT] + [exit-when-poll=SECS] + +=head1 DESCRIPTION + +C<nbdkit-exitwhen-filter> is an nbdkit filter that causes nbdkit to +exit gracefully when some external event occurs. Built-in events are: +a file being created or deleted, a pipe being closed, or a process +exiting. You can also define custom events using an external script +or command. + +After the event occurs nbdkit refuses new connections, waits for all +current clients to disconnect, and then exits. + +A similar filter is L<nbdkit-exitlast-filter(1)>. For other ways to +ensure that nbdkit exits when you want see L<nbdkit-captive(1)> and +L<nbdkit-service(1)>. + +=head1 EXAMPLES + + nbdkit --filter=exitwhen memory 1G exit-when-file-created=/tmp/stop + +nbdkit will run normally until something creates F</tmp/stop>, +whereupon nbdkit will refuse new connections and exit as soon as the +last client has disconnected. If F</tmp/stop> exists before nbdkit +starts, it will exit immediately. + + nbdkit --filter=exitwhen memory 1G exit-when-process-exits=1234 + +nbdkit will exit gracefully when PID 1234 exits and all connections +close. If you want to exit when the parent process of nbdkit exits, +consider using the I<--exit-with-parent> flag instead. + +=head1 PARAMETERS + +You can define multiple C<exit-when-*> events on the command line: +nbdkit will exit if any of the events happens. If there are no +C<exit-when-*> events then the filter does nothing. + +=over 4 + +=item B<exit-when-file-created=>FILENAME + +=item B<exit-when-file-deleted=>FILENAME + +Exit when the named file is created or deleted. + +=item B<exit-when-pipe-closed=>FD + +The read end of a L<pipe(2)> is passed to nbdkit in the given file +descriptor number. Exit when the pipe is closed. The filter does not +read any data from the pipe. + +=item B<exit-when-process-exits=>PID + +Exit when process ID C<PID> exits. + +Note there is a small race between passing the process ID to the +filter and the filter checking it for the first time. During this +window the original PID might exit and an unrelated program might get +the same PID, thus holding nbdkit open for longer than wanted. The +pipe method above is more reliable if you are able to modify the other +process. + +=item B<exit-when-script=">SCRIPTB<"> + +Create a custom event using the script or command C<SCRIPT>. The +C<SCRIPT> can be a program, shell script or a command with optional +parameters. Note if using a separate program or script that you may +need to use the absolute path because nbdkit changes directory when it +daemonizes. + +The script should exit with code 88 if the event is detected. The +filter does different things depending on the exit code of the script: + +=over 4 + +=item C<0> + +I<The event has not been triggered>, so nbdkit continues to process +requests as normal. + +=item C<1-87> + +An error is logged, but the event is I<not> triggered and nbdkit +continues to process requests as normal. + +=item C<88> + +I<The event has been triggered>. nbdkit will refuse new connections +and exit gracefully as soon as all current clients disconnect. + +=item C<89-> + +Exit codes 89 and above are reserved for future use. The behaviour +may change in future if scripts return any of these exit codes. + +=back + +=item B<exit-when-poll=>SECS + +When nbdkit is serving clients this filter does not need to poll +because it can check for events when a client connects or disconnects. +However when nbdkit is idle the filter needs to poll for events every +C<SECS> seconds and if any event happens exit immediately. + +The default is 60 seconds. + +=back + +=head1 FILES + +=over 4 + +=item F<$filterdir/nbdkit-exitwhen-filter.so> + +The filter. + +Use C<nbdkit --dump-config> to find the location of C<$filterdir>. + +=back + +=head1 VERSION + +C<nbdkit-exitwhen-filter> first appeared in nbdkit 1.24. + +=head1 SEE ALSO + +L<nbdkit(1)>, +L<nbdkit-exitlast-filter(1)>, +L<nbdkit-ip-filter(1)>, +L<nbdkit-limit-filter(1)>, +L<nbdkit-rate-filter(1)>, +L<nbdkit-captive(1)>, +L<nbdkit-service(1)>, +L<nbdkit-filter(3)>, +L<nbdkit-plugin(3)>. + +=head1 AUTHORS + +Richard W.M. Jones + +=head1 COPYRIGHT + +Copyright (C) 2020 Red Hat Inc. diff --git a/filters/ip/nbdkit-ip-filter.pod b/filters/ip/nbdkit-ip-filter.pod index 5d9e4616..48731a98 100644 --- a/filters/ip/nbdkit-ip-filter.pod +++ b/filters/ip/nbdkit-ip-filter.pod @@ -225,6 +225,7 @@ C<nbdkit-ip-filter> first appeared in nbdkit 1.18. L<nbdkit(1)>, L<nbdkit-exitlast-filter(1)>, +L<nbdkit-exitwhen-filter(1)>, L<nbdkit-limit-filter(1)>, L<nbdkit-filter(3)>. diff --git a/filters/limit/nbdkit-limit-filter.pod b/filters/limit/nbdkit-limit-filter.pod index 434dde5c..5d211047 100644 --- a/filters/limit/nbdkit-limit-filter.pod +++ b/filters/limit/nbdkit-limit-filter.pod @@ -46,6 +46,7 @@ C<nbdkit-limit-filter> first appeared in nbdkit 1.20. L<nbdkit(1)>, L<nbdkit-exitlast-filter(1)>, +L<nbdkit-exitwhen-filter(1)>, L<nbdkit-ip-filter(1)>, L<nbdkit-noparallel-filter(1)>, L<nbdkit-rate-filter(1)>, diff --git a/filters/rate/nbdkit-rate-filter.pod b/filters/rate/nbdkit-rate-filter.pod index 7aef4ec6..8956e641 100644 --- a/filters/rate/nbdkit-rate-filter.pod +++ b/filters/rate/nbdkit-rate-filter.pod @@ -130,6 +130,7 @@ L<nbdkit(1)>, L<nbdkit-blocksize-filter(1)>, L<nbdkit-delay-filter(1)>, L<nbdkit-exitlast-filter(1)>, +L<nbdkit-exitwhen-filter(1)>, L<nbdkit-limit-filter(1)>, L<nbdkit-pause-filter(1)>, L<nbdkit-filter(3)>, diff --git a/configure.ac b/configure.ac index b4302dfc..ce7209da 100644 --- a/configure.ac +++ b/configure.ac @@ -104,6 +104,7 @@ filters="\ delay \ error \ exitlast \ + exitwhen \ exportname \ ext2 \ extentlist \ @@ -1228,6 +1229,7 @@ AC_CONFIG_FILES([Makefile filters/delay/Makefile filters/error/Makefile filters/exitlast/Makefile + filters/exitwhen/Makefile filters/exportname/Makefile filters/ext2/Makefile filters/extentlist/Makefile diff --git a/filters/exitwhen/Makefile.am b/filters/exitwhen/Makefile.am new file mode 100644 index 00000000..dfef6abe --- /dev/null +++ b/filters/exitwhen/Makefile.am @@ -0,0 +1,67 @@ +# nbdkit +# Copyright (C) 2019-2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +include $(top_srcdir)/common-rules.mk + +EXTRA_DIST = nbdkit-exitwhen-filter.pod + +filter_LTLIBRARIES = nbdkit-exitwhen-filter.la + +nbdkit_exitwhen_filter_la_SOURCES = \ + exitwhen.c \ + $(top_srcdir)/include/nbdkit-filter.h \ + $(NULL) + +nbdkit_exitwhen_filter_la_CPPFLAGS = \ + -I$(top_srcdir)/include \ + -I$(top_srcdir)/common/utils \ + $(NULL) +nbdkit_exitwhen_filter_la_CFLAGS = $(WARNINGS_CFLAGS) +nbdkit_exitwhen_filter_la_LIBADD = \ + $(top_builddir)/common/utils/libutils.la \ + $(IMPORT_LIBRARY_ON_WINDOWS) \ + $(NULL) +nbdkit_exitwhen_filter_la_LDFLAGS = \ + -module -avoid-version -shared $(NO_UNDEFINED_ON_WINDOWS) \ + -Wl,--version-script=$(top_srcdir)/filters/filters.syms \ + $(NULL) + +if HAVE_POD + +man_MANS = nbdkit-exitwhen-filter.1 +CLEANFILES += $(man_MANS) + +nbdkit-exitwhen-filter.1: nbdkit-exitwhen-filter.pod + $(PODWRAPPER) --section=1 --man $@ \ + --html $(top_builddir)/html/$@.html \ + $< + +endif HAVE_POD diff --git a/tests/Makefile.am b/tests/Makefile.am index ee342cc4..071ac3b5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -1382,6 +1382,30 @@ EXTRA_DIST += \ TESTS += test-exitlast.sh EXTRA_DIST += test-exitlast.sh +# exitwhen filter test. +check_PROGRAMS += test-exitwhen-pipe-closed +TESTS += \ + test-exitwhen-file-already-created.sh \ + test-exitwhen-file-created.sh \ + test-exitwhen-file-created-reject-new.sh \ + test-exitwhen-file-created-when-idle.sh \ + test-exitwhen-file-deleted.sh \ + test-exitwhen-pipe-closed \ + test-exitwhen-process-exits.sh \ + test-exitwhen-script.sh \ + $(NULL) +EXTRA_DIST += \ + test-exitwhen-file-already-created.sh \ + test-exitwhen-file-created.sh \ + test-exitwhen-file-created-reject-new.sh \ + test-exitwhen-file-created-when-idle.sh \ + test-exitwhen-file-deleted.sh \ + test-exitwhen-process-exits.sh \ + test-exitwhen-script.sh \ + $(NULL) + +test_exitwhen_pipe_closed_CFLAGS = $(WARNINGS_CFLAGS) + # exportname filter test. TESTS += test-exportname.sh EXTRA_DIST += test-exportname.sh diff --git a/filters/exitwhen/exitwhen.c b/filters/exitwhen/exitwhen.c new file mode 100644 index 00000000..801c355a --- /dev/null +++ b/filters/exitwhen/exitwhen.c @@ -0,0 +1,473 @@ +/* nbdkit + * Copyright (C) 2019-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <unistd.h> +#include <fcntl.h> +#include <poll.h> +#include <errno.h> +#include <sys/types.h> + +#include <pthread.h> + +#include <nbdkit-filter.h> + +#include "cleanup.h" +#include "utils.h" +#include "vector.h" + +static unsigned pollsecs = 60; + +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; +static unsigned connections = 0; +static bool exiting = false; + +/* The list of events generated from command line parameters. */ +struct event { + enum { EVENT_FILE_CREATED = 1, EVENT_FILE_DELETED, EVENT_PROCESS_EXITS, + EVENT_FD_CLOSED, EVENT_SCRIPT } type; + union { + char *filename; /* Filename or script. */ + int fd; /* For PROCESS_EXITS or FD_CLOSED. */ +#ifndef __linux__ + pid_t pid; /* For PROCESS_EXITS on non-Linux. */ +#endif + } u; +}; +DEFINE_VECTOR_TYPE(event_list, struct event); +static event_list events = empty_vector; + +static void +free_event (struct event event) +{ + switch (event.type) { + case EVENT_FILE_CREATED: + case EVENT_FILE_DELETED: + case EVENT_SCRIPT: + free (event.u.filename); + break; + case EVENT_PROCESS_EXITS: +#ifdef __linux__ + case EVENT_FD_CLOSED: +#endif + close (event.u.fd); + break; +#ifndef __linux__ + case EVENT_FD_CLOSED: + break; +#endif + } +} + +static void +exitwhen_unload (void) +{ + event_list_iter (&events, free_event); + free (events.ptr); +} + +/* If exiting is already true, this does nothing and returns true. + * Otherwise it checks if any event in the list has happened. If an + * event has happened, sets exiting to true. It returns the exiting + * flag. + * + * This must only be called with the lock held. + */ +static void check_for_event_file_created (const struct event *); +static void check_for_event_file_deleted (const struct event *); +static void check_for_event_process_exits (const struct event *); +static void check_for_event_fd_closed (const struct event *); +static void check_for_event_script (const struct event *); + +static bool +check_for_event (void) +{ + size_t i; + + if (!exiting) { + for (i = 0; i < events.size; ++i) { + const struct event *event = &events.ptr[i]; + + switch (event->type) { + case EVENT_FILE_CREATED: + check_for_event_file_created (event); + break; + case EVENT_FILE_DELETED: + check_for_event_file_deleted (event); + break; + case EVENT_PROCESS_EXITS: + check_for_event_process_exits (event); + break; + case EVENT_FD_CLOSED: + check_for_event_fd_closed (event); + break; + case EVENT_SCRIPT: + check_for_event_script (event); + break; + } + } + } + + return exiting; +} + +static void +check_for_event_file_created (const struct event *event) +{ + if (access (event->u.filename, R_OK) == 0) { + nbdkit_debug ("exit-when-file-created: detected %s created", + event->u.filename); + exiting = true; + } +} + +static void +check_for_event_file_deleted (const struct event *event) +{ + if (access (event->u.filename, R_OK) == -1) { + if (errno == ENOTDIR || errno == ENOENT) { + nbdkit_debug ("exit-when-file-deleted: detected %s deleted", + event->u.filename); + exiting = true; + } + else { + /* Log the error but continue. */ + nbdkit_error ("exit-when-file-deleted: access: %s: %m", + event->u.filename); + } + } +} + +static void +check_for_event_process_exits (const struct event *event) +{ + char c; + +#ifdef __linux__ + /* https://gitlab.freedesktop.org/polkit/polkit/-/issues/75 + * + * event->u.fd holds /proc/PID/stat of the original process open. + * If we can still read a byte from it then the original process is + * still around. If we get ESRCH then the process has exited. + */ + lseek (event->u.fd, 0, SEEK_SET); + if (read (event->u.fd, &c, 1) == -1) { + if (errno == ESRCH) { + nbdkit_debug ("exit-when-process-exits: detected process exit"); + exiting = true; + } + else { + /* Log the error but continue. */ + nbdkit_error ("exit-when-process-exits: read: %m"); + } + } +#else /* !__linux__ */ + /* XXX Find a safe way to do this on BSD at least. */ + if (kill (event->u.pid, 0) == -1 && errno == ESRCH) { + nbdkit_debug ("exit-when-process-exits: detected process exit"); + exiting = true; + } +#endif /* !__linux__ */ +} + +static void +check_for_event_fd_closed (const struct event *event) +{ + int r; + struct pollfd fds[1]; + + /* event->u.fd is the read side of a pipe or socket. Check it is + * not closed. We don't actually read anything from the pipe. + */ + fds[0].fd = event->u.fd; + fds[0].events = 0; + r = poll (fds, 1, 0); + if (r == 1) { + if ((fds[0].revents & POLLHUP) != 0) { + nbdkit_debug ("exit-when-pipe-closed: detected pipe closed"); + exiting = true; + } + else if ((fds[0].revents & POLLNVAL) != 0) { + /* If we were passed a bad file descriptor that is user error + * and we should exit with an error early. Because + * check_for_event() is called first in get_ready() this should + * cause this to happen. + */ + nbdkit_error ("exit-when-pipe-closed: invalid file descriptor"); + exiting = true; + } + } + else if (r == -1) { + /* Log the error but continue. */ + nbdkit_error ("exit-when-pipe-closed: poll: %m"); + } +} + +static void +check_for_event_script (const struct event *event) +{ + int r; + + /* event->u.filename is a script filename or command. Exit code 88 + * indicates the event has happened. + */ + r = system (event->u.filename); + if (r == -1) { + /* Log the error but continue. */ + nbdkit_error ("exit-when-script: %m"); + } + else if (WIFEXITED (r) && WEXITSTATUS (r) == 0) { + /* Normal case, do nothing. */ + } + else if (WIFEXITED (r) && WEXITSTATUS (r) == 88) { + nbdkit_debug ("exit-when-script: detected scripted event"); + exiting = true; + } + else { + /* Log the error but continue. */ + exit_status_to_nbd_error (r, "exit-when-script"); + } +} + +/* The background polling thread. + * + * This runs continuously in the background, but you can pause it by + * grabbing the "pause_lock" (use the pause/resume_polling_thread() + * wrappers). + */ +static pthread_mutex_t pause_lock = PTHREAD_MUTEX_INITIALIZER; + +static void * +polling_thread (void *vp) +{ + for (;;) { + { + /* Note the order here is chosen to avoid possible deadlock + * because the callers of pause/resume_polling_thread() always + * acquire &lock before &pause_lock. + */ + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&pause_lock); + if (check_for_event ()) { + nbdkit_debug ("exitwhen: shutdown from polling thread"); + nbdkit_shutdown (); + } + } + + sleep (pollsecs); + } +} + +static void +pause_polling_thread (void) +{ + pthread_mutex_lock (&pause_lock); +} + +static void +resume_polling_thread (void) +{ + pthread_mutex_unlock (&pause_lock); +} + +static int +exitwhen_config (nbdkit_next_config *next, void *nxdata, + const char *key, const char *value) +{ + struct event event; + + if (strcmp (key, "exit-when-file-created") == 0 || + strcmp (key, "exit-when-file-deleted") == 0) { + event.type = key[15] == 'c' ? EVENT_FILE_CREATED : EVENT_FILE_DELETED; + event.u.filename = nbdkit_absolute_path (value); + if (event.u.filename == NULL) + return -1; + if (event_list_append (&events, event) == -1) + return -1; + return 0; + } + else if (strcmp (key, "exit-when-pipe-closed") == 0 || + strcmp (key, "exit-when-fd-closed") == 0) { + event.type = EVENT_FD_CLOSED; + if (nbdkit_parse_int ("exit-when-pipe-closed", value, &event.u.fd) == -1) + return -1; + if (event_list_append (&events, event) == -1) + return -1; + return 0; + } + else if (strcmp (key, "exit-when-process-exits") == 0 || + strcmp (key, "exit-when-pid-exits") == 0) { + uint64_t pid; + CLEANUP_FREE char *str = NULL; + + event.type = EVENT_PROCESS_EXITS; + if (nbdkit_parse_uint64_t ("exit-when-process-exits", value, &pid) == -1) + return -1; +#ifdef __linux__ + /* See: https://gitlab.freedesktop.org/polkit/polkit/-/issues/75 */ + if (asprintf (&str, "/proc/%" PRIu64 "/stat", pid) == -1) { + nbdkit_error ("asprintf: %m"); + return -1; + } + event.u.fd = open (str, O_RDONLY); + if (event.u.fd == -1) { + nbdkit_error ("exit-when-process-exits: %s: %m", str); + return -1; + } +#else + event.u.pid = (pid_t) pid; +#endif + if (event_list_append (&events, event) == -1) + return -1; + return 0; + } + else if (strcmp (key, "exit-when-script") == 0) { + event.type = EVENT_SCRIPT; + event.u.filename = strdup (value); + if (event.u.filename == NULL) { + nbdkit_error ("strdup: %m"); + return -1; + } + if (event_list_append (&events, event) == -1) + return -1; + return 0; + } + else if (strcmp (key, "exit-when-poll") == 0) { + if (nbdkit_parse_unsigned ("exit-when-poll", value, &pollsecs) == -1) + return -1; + return 0; + } + else + return next (nxdata, key, value); +} + +/* Before forking, run the check. If the event has already happened + * then we exit immediately. + */ +static int +exitwhen_get_ready (nbdkit_next_get_ready *next, void *nxdata, + int thread_model) +{ + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + + if (check_for_event ()) + exit (EXIT_SUCCESS); + + return next (nxdata); +} + +static int +exitwhen_after_fork (nbdkit_next_after_fork *next, void *nxdata) +{ + int err; + pthread_t thread; + + /* Start background polling thread. Initially it is running. */ + err = pthread_create (&thread, NULL, polling_thread, NULL); + if (err != 0) { + errno = err; + nbdkit_error ("pthread_create: %m"); + return -1; + } + return next (nxdata); +} + +static int +exitwhen_preconnect (nbdkit_next_preconnect *next, void *nxdata, int readonly) +{ + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + + if (check_for_event ()) { + nbdkit_error ("exitwhen: nbdkit is exiting: rejecting new connection"); + return -1; + } + + if (next (nxdata, readonly) == -1) + return -1; + + return 0; +} + +static void * +exitwhen_open (nbdkit_next_open *next, nbdkit_backend *nxdata, + int readonly, const char *exportname, int is_tls) +{ + if (next (nxdata, readonly, exportname) == -1) + return NULL; + + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + connections++; + if (connections == 1) + pause_polling_thread (); + + return NBDKIT_HANDLE_NOT_NEEDED; +} + +static void +exitwhen_close (void *handle) +{ + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + + check_for_event (); + + --connections; + if (connections == 0) { + if (exiting) { + nbdkit_debug ("exitwhen: exiting on last client connection"); + nbdkit_shutdown (); + } + else + resume_polling_thread (); + } +} + +static struct nbdkit_filter filter = { + .name = "exitwhen", + .longname = "nbdkit exitwhen filter", + .unload = exitwhen_unload, + + .config = exitwhen_config, + .get_ready = exitwhen_get_ready, + .after_fork = exitwhen_after_fork, + + .preconnect = exitwhen_preconnect, + .open = exitwhen_open, + .close = exitwhen_close, +}; + +NBDKIT_REGISTER_FILTER(filter) diff --git a/tests/test-exitwhen-file-already-created.sh b/tests/test-exitwhen-file-already-created.sh new file mode 100755 index 00000000..145864b5 --- /dev/null +++ b/tests/test-exitwhen-file-already-created.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +source ./functions.sh +set -x + +requires_filter exitwhen + +eventfile=exitwhen-file-already-created.event +rm -f $eventfile +cleanup_fn rm -f $eventfile + +# nbdkit should exit immediately if the event file already exists. + +touch $eventfile +nbdkit -fv -U - --filter=exitwhen null exit-when-file-created=$eventfile diff --git a/tests/test-exitwhen-file-created-reject-new.sh b/tests/test-exitwhen-file-created-reject-new.sh new file mode 100755 index 00000000..7736e067 --- /dev/null +++ b/tests/test-exitwhen-file-created-reject-new.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +source ./functions.sh +set -x + +requires_filter exitwhen +requires nbdsh --version + +sock=$(mktemp -u /tmp/nbdkit-test-sock.XXXXXX) +pidfile=exitwhen-file-created-reject-new.pid +eventfile=exitwhen-file-created-reject-new.event +files="$pidfile $sock $eventfile" +cleanup_fn rm -f $files + +# Start nbdkit with the exitwhen filter. +start_nbdkit -P $pidfile -U $sock \ + --filter=exitwhen memory size=1M \ + exit-when-file-created=$eventfile + +# Connect and test. +export eventfile sock +nbdsh -c - <<'EOF' +import os +from pathlib import Path + +eventfile = os.environ['eventfile'] +sock = os.environ['sock'] + +# Open a single connection. +h.connect_unix(sock) + +# Check the connection is functional. +buf = h.pread(512, 0) + +# Create the event. +Path(eventfile).touch() + +# A new connection should be rejected. +h2 = nbd.NBD() +try: + h2.connect_unix(sock) + exit(1) +except nbd.Error: + pass +EOF + +# Now nbdkit should exit automatically. Wait for it to do so. +pid=`cat $pidfile` +for i in {1..60}; do + if ! kill -s 0 $pid; then + break + fi + sleep 1 +done +if kill -s 0 $pid; then + echo "$0: nbdkit did not exit after last client connection" + exit 1 +fi diff --git a/tests/test-exitwhen-file-created-when-idle.sh b/tests/test-exitwhen-file-created-when-idle.sh new file mode 100755 index 00000000..ff8a106b --- /dev/null +++ b/tests/test-exitwhen-file-created-when-idle.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +source ./functions.sh +set -x + +requires_filter exitwhen + +pidfile=exitwhen-file-created-when-idle.pid +eventfile=exitwhen-file-created-when-idle.event +rm -f $eventfile $pidfile +cleanup_fn rm -f $eventfile $pidfile + +start_nbdkit -P $pidfile -U - \ + --filter=exitwhen null \ + exit-when-file-created=$eventfile \ + exit-when-poll=1 + +# Creating the file should cause nbdkit to exit after a short time. +sleep 1 +touch $eventfile +sleep 1 + +pid=`cat $pidfile` +for i in {1..60}; do + if ! kill -s 0 $pid; then + break + fi + sleep 1 +done +if kill -s 0 $pid; then + echo "$0: nbdkit did not exit when idle" + exit 1 +fi diff --git a/tests/test-exitwhen-file-created.sh b/tests/test-exitwhen-file-created.sh new file mode 100755 index 00000000..ff62604d --- /dev/null +++ b/tests/test-exitwhen-file-created.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +source ./functions.sh +set -x + +requires_filter exitwhen +requires nbdsh --version + +sock=$(mktemp -u /tmp/nbdkit-test-sock.XXXXXX) +pidfile=exitwhen-file-created.pid +eventfile=exitwhen-file-created.event +files="$pidfile $sock $eventfile" +cleanup_fn rm -f $files + +# Start nbdkit with the exitwhen filter. +start_nbdkit -P $pidfile -U $sock \ + --filter=exitwhen memory size=1M \ + exit-when-file-created=$eventfile + +# Connect and test. +export eventfile sock +nbdsh -c - <<'EOF' +import os +from pathlib import Path + +eventfile = os.environ['eventfile'] +sock = os.environ['sock'] + +# Open a couple of connections. +h.connect_unix(sock) +h2 = nbd.NBD() +h2.connect_unix(sock) + +# The connections should both be functional. +buf = h.pread(512, 0) +buf = h2.pread(512, 0) + +# Create the event. +Path(eventfile).touch() + +# The connections should still be functional. +buf = h.pread(512, 0) +h.shutdown() +del h +buf = h2.pread(512, 0) +h2.shutdown() +del h2 +EOF + +# Now nbdkit should exit automatically. Wait for it to do so. +pid=`cat $pidfile` +for i in {1..60}; do + if ! kill -s 0 $pid; then + break + fi + sleep 1 +done +if kill -s 0 $pid; then + echo "$0: nbdkit did not exit after last client connection" + exit 1 +fi diff --git a/tests/test-exitwhen-file-deleted.sh b/tests/test-exitwhen-file-deleted.sh new file mode 100755 index 00000000..2c0cafe2 --- /dev/null +++ b/tests/test-exitwhen-file-deleted.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +source ./functions.sh +set -x + +requires_filter exitwhen +requires nbdsh --version + +sock=$(mktemp -u /tmp/nbdkit-test-sock.XXXXXX) +pidfile=exitwhen-file-deleted.pid +eventfile=exitwhen-file-deleted.event +files="$pidfile $sock $eventfile" +cleanup_fn rm -f $files + +touch $eventfile + +# Start nbdkit with the exitwhen filter. +start_nbdkit -P $pidfile -U $sock \ + --filter=exitwhen memory size=1M \ + exit-when-file-deleted=$eventfile \ + exit-when-poll=1 + +sleep 1 +rm $eventfile +sleep 1 + +# Now nbdkit should exit automatically. Wait for it to do so. +pid=`cat $pidfile` +for i in {1..60}; do + if ! kill -s 0 $pid; then + break + fi + sleep 1 +done +if kill -s 0 $pid; then + echo "$0: nbdkit did not exit after last client connection" + exit 1 +fi diff --git a/tests/test-exitwhen-process-exits.sh b/tests/test-exitwhen-process-exits.sh new file mode 100755 index 00000000..22a17b58 --- /dev/null +++ b/tests/test-exitwhen-process-exits.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +source ./functions.sh +set -x + +requires_filter exitwhen +requires nbdsh --version + +sock=$(mktemp -u /tmp/nbdkit-test-sock.XXXXXX) +pidfile=exitwhen-process-exits.pid +files="$pidfile $sock" +cleanup_fn rm -f $files + +# Start the unrelated process. +sleep 1h & +sleeppid=$! + +# Start nbdkit with the exitwhen filter. +start_nbdkit -P $pidfile -U $sock \ + --filter=exitwhen memory size=1M \ + exit-when-process-exits=$sleeppid \ + exit-when-poll=1 + +# Killing the sleep process should cause nbdkit to exit after a few +# seconds. +kill $sleeppid + +# Now nbdkit should exit automatically. Wait for it to do so. +pid=`cat $pidfile` +for i in {1..60}; do + if ! kill -s 0 $pid; then + break + fi + sleep 1 +done +if kill -s 0 $pid; then + echo "$0: nbdkit did not exit after last client connection" + exit 1 +fi diff --git a/tests/test-exitwhen-script.sh b/tests/test-exitwhen-script.sh new file mode 100755 index 00000000..43eeb58a --- /dev/null +++ b/tests/test-exitwhen-script.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +source ./functions.sh +set -x + +requires_filter exitwhen +requires nbdsh --version + +sock=$(mktemp -u /tmp/nbdkit-test-sock.XXXXXX) +pidfile=exitwhen-script.pid +files="$pidfile $sock" +cleanup_fn rm -f $files + +# Start the unrelated process. +sleep 1h & +sleeppid=$! + +# Start nbdkit with the exitwhen filter. +# The command tests if the sleep process is still running. +start_nbdkit -P $pidfile -U $sock \ + --filter=exitwhen memory size=1M \ + exit-when-script="kill -0 $sleeppid || exit 88" \ + exit-when-poll=1 + +# Killing the sleep process should cause nbdkit to exit after a few +# seconds. +kill $sleeppid + +# Now nbdkit should exit automatically. Wait for it to do so. +pid=`cat $pidfile` +for i in {1..60}; do + if ! kill -s 0 $pid; then + break + fi + sleep 1 +done +if kill -s 0 $pid; then + echo "$0: nbdkit did not exit after last client connection" + exit 1 +fi diff --git a/tests/test-exitwhen-pipe-closed.c b/tests/test-exitwhen-pipe-closed.c new file mode 100644 index 00000000..b14bacc4 --- /dev/null +++ b/tests/test-exitwhen-pipe-closed.c @@ -0,0 +1,81 @@ +/* nbdkit + * Copyright (C) 2017-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> + +int +main (int argc, char *argv[]) +{ + char *param; + int fd[2]; + pid_t pid; + + if (pipe (fd) == -1) { + perror ("pipe"); + exit (EXIT_FAILURE); + } + if (asprintf (&param, "exit-when-pipe-closed=%d", fd[0]) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + + /* Run nbdkit. */ + pid = fork (); + if (pid == -1) { + perror ("fork"); + exit (EXIT_FAILURE); + } + if (pid == 0) { /* Child - run nbdkit. */ + /* Close the write side of the pipe. */ + close (fd[1]); + + /* Run nbdkit. */ + execlp ("nbdkit", "nbdkit", "-v", "--filter=exitwhen", + "null", "1M", param, "exit-when-poll=1", + NULL); + perror ("execvp"); + _exit (EXIT_FAILURE); + } + + /* Close the read side of the pipe. */ + close (fd[0]); + + /* The test here is simply that nbdkit exits because we exit and our + * side of the pipe is closed. + */ + free (param); + exit (EXIT_SUCCESS); +} diff --git a/.gitignore b/.gitignore index 6434f83d..f98c3e9a 100644 --- a/.gitignore +++ b/.gitignore @@ -127,6 +127,7 @@ plugins/*/*.3 /tests/test-data /tests/test-delay /tests/test-exit-with-parent +/tests/test-exitwhen-pipe-closed /tests/test-ext2 /tests/test-file-block /tests/test-golang -- 2.29.0.rc2

1 0

[PATCH nbdkit INCOMPLETE] New filter: exitwhen: exit gracefully when an event occurs.
by Richard W.M. Jones 20 Oct '20

20 Oct '20

This incomplete patch adds a new filter allowing more control over when nbdkit exits. You can now get nbdkit to exit gracefully on certain events, such as a file being created, a pipe held open by another process going away, or when another PID exits. There is also a script option to allow for completely custom events. It is untested at the moment, I'm posting it to get feedback on the idea. The manual page is a complete description of the proposed feature. To do: - implement the polling thread - add tests Rich.

1 1