November 2018 - Libguestfs - libguestfs List Archives

[PATCH v3 0/3] Install QEMU-GA from oVirt guest tools ISO on Linux
by Tomáš Golembiovský 13 Nov '18

13 Nov '18

changes in v3: - fix call to install_local changes in v2: - moved copy_drivers above copy_files - renamed copy_files to copy_from_virtio_win - renamed install to install_local - use rpm instead of yum This installs packages with QEMU Guest Agent when converting Linux machine. The packages should be available on guest tools ISO. The patches work "as-is" but probably deserve some more attention: - it is "abusing" Winows_virtio code but renaming/refactoring everything to remove "windows" from the name and use "guest tools" seems like a lot of unnecesary work - support for Debian is missing I don't know how to install the package only when all it's dependencies are already installed. dpkg cannot be used to check that (simulate the install). And attempting to install the package will leave it half-installed (dpkg cannot roll-back). Tomáš Golembiovský (3): v2v: refactor copy_drivers() in Windows_virtio v2v: linux: install packages v2v: linux: install QEMU-GA (RHBZ#1619665) v2v/convert_linux.ml | 2 ++ v2v/linux.ml | 14 ++++++++ v2v/linux.mli | 3 ++ v2v/windows_virtio.ml | 78 +++++++++++++++++++++++++++++++++--------- v2v/windows_virtio.mli | 4 +++ 5 files changed, 84 insertions(+), 17 deletions(-) -- 2.19.0

3 14

Further nbdkit patches for Haiku
by Richard W.M. Jones 13 Nov '18

13 Nov '18

François, thanks for the patches so far. This morning I pushed: https://github.com/libguestfs/nbdkit/commit/e26fb6cea3fdba13b5472daf6767e76… build: add missing #include "byte-swapping.h" https://github.com/libguestfs/nbdkit/commit/b962272a561bfed087316d24b7d510e… build: more fallbacks for mkostemp The iconv patch makes iconv a requirement. I don't know how many systems actually lack iconv -- probably not very many -- but even so I don't want nbdkit to require a library just to compile a single plugin, as a feature of nbdkit is that the minimal set of requirements. So I have modified your patch (see attachment) to make the iconv dependency optional. Can you let me know if this works on Haiku or if I broke anything? The final patch was: https://github.com/mmuman/nbdkit/commit/50ee315525258b0defae06f6e29da841ddc… build: avoid hardcoding /var/tmp by default, use P_tmpdir Unfortunately this won't work well on some Linux systems which use the /tmp-on-tmpdir misfeature. On these systems, /tmp has only a small, limited amount of space and is backed by memory, and we must use /var/tmp for large files. I don't know how to fix this. Perhaps a simple: AS_CASE([$host_os], [haiku*], [LARGE_TMPDIR=/tmp] [LARGE_TMPDIR=/var/tmp] ) AC_DEFINE_UNQUOTED([LARGE_TMPDIR],["$LARGE_TMPDIR"], [Temporary directory for large files]) would be enough for now? Finally about the list of failing tests: https://pastebin.com/1PFG85DS Mostly these are because of the old qemu-io binary. The qemu-io -f option was added 4 years ago :-/ so it could be time to upgrade. This error: ./test-shebang-perl.sh: PID file was not created rect ELF header FAIL test-shebang-perl.sh (exit status: 1) (I think the error message is "incorrect ELF header") could be something to do with the way we use shebangs (#!). IIRC this also fails on FreeBSD so probably nbdkit is doing something which is not correct POSIX. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

3 10

[PATCH] tests: Fix shebang tests on FreeBSD.
by Richard W.M. Jones 13 Nov '18

13 Nov '18

On FreeBSD we cannot go through the shell script (./nbdkit). Instead replace the path with a direct link to both the nbdkit binary and locally compiled plugin. Note this does not go through valgrinding when you run ‘make check-valgrind’ so that needs to be fixed at some point in the future. Thanks: François Revol and Eric Blake. See also: https://www.redhat.com/archives/libguestfs/2018-November/msg00126.html --- tests/shebang.pl | 2 +- tests/shebang.py | 2 +- tests/shebang.rb | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/shebang.pl b/tests/shebang.pl index 17c7821..e6f1630 100755 --- a/tests/shebang.pl +++ b/tests/shebang.pl @@ -1,4 +1,4 @@ -#!../nbdkit perl +#!../src/nbdkit ../plugins/perl/.libs/nbdkit-perl-plugin.so use strict; diff --git a/tests/shebang.py b/tests/shebang.py index 6f33623..e836255 100755 --- a/tests/shebang.py +++ b/tests/shebang.py @@ -1,4 +1,4 @@ -#!../nbdkit python +#!../src/nbdkit ../plugins/python/.libs/nbdkit-python-plugin.so disk = bytearray(1024 * 1024) diff --git a/tests/shebang.rb b/tests/shebang.rb index 34588d1..ccf4519 100755 --- a/tests/shebang.rb +++ b/tests/shebang.rb @@ -1,4 +1,4 @@ -#!../nbdkit ruby +#!../src/nbdkit ../plugins/ruby/.libs/nbdkit-ruby-plugin.so include Nbdkit -- 2.19.0.rc0

1 0

collectd leaks SIGCHLD == SIG_IGN into plugins
by Richard W.M. Jones 13 Nov '18

13 Nov '18

Peter Dimitrov and myself were debugging a very peculiar bug when libguestfs is run as a plugin from collectd: https://www.redhat.com/archives/libguestfs/2018-November/thread.html#00023 The long story short is that collectd leaks SIGCHLD == SIG_IGN setting into plugins: https://www.redhat.com/archives/libguestfs/2018-November/msg00095.html This means that any plugin that does the usual pattern of: pid = fork (); ... if (waitpid (pid, NULL, 0) == -1) { perror ("waitpid"); exit (EXIT_FAILURE); } will fail, because the forked subprocess is automatically reaped before waitpid is called, resulting in the wait failing with errno == ECHILD. It is possible to work around this by adding: signal (SIGCHLD, SIG_DFL); to the plugin. However I believe this is a bug in collectd, and it should sanitize signals (and maybe other things) before running plugins. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

5 12

ANNOUNCE: nbdkit 1.8 - an NBD server toolkit with stable plugin API and permissive license
by Richard W.M. Jones 12 Nov '18

12 Nov '18

I’m pleased to announce the next stable release of nbdkit. NBD — Network Block Device — is a protocol for accessing Block Devices (hard disks and disk-like things) over a Network. nbdkit is a toolkit for creating NBD servers. The key features are: * Multithreaded NBD server written in C with good performance. * Minimal dependencies for the basic server. * Liberal license (BSD) allows nbdkit to be linked to proprietary libraries or included in proprietary code. * Well-documented, simple plugin API with a stable ABI guarantee. Lets you export “unconventional” block devices easily. * You can write plugins in C, Lua, Perl, Python, OCaml, Ruby, [new!] shell script or Tcl. * Filters can be stacked in front of plugins to transform the output. Clone source: https://github.com/libguestfs/nbdkit Download tarball: http://download.libguestfs.org/nbdkit/1.8-stable/ (other downloads: http://download.libguestfs.org/nbdkit/ ) New in this release: - New plugin: ‘sh’ for running shell scripts and other external programs. (https://rwmj.wordpress.com/2018/09/09/write-nbdkit-plugins-in-shell-script/) - New plugin: ‘partitioning’ which creates a partitioned disk from a list of partitions in files. - New plugin: ‘iso’ which creates an ISO (CD-ROM) image on the fly from a directory of files. (https://rwmj.wordpress.com/2018/10/18/new-in-nbdkit-create-an-iso-image-on-…) - New plugin: ‘floppy’ which creates a virtual FAT-formatted floppy disk from a directory of files. (https://rwmj.wordpress.com/2018/10/29/new-in-nbdkit-create-a-virtual-floppy…) - Magic config key allows simpler invocation of many plugins, eg: nbdkit file file=foo becomes: nbdkit file foo - data: Add disk2data.pl script to generate data parameters. - data: Use ‘<file’ to inline a file. - vddk: Use ‘-D vddk.diskinfo=1’ to enable extra debug information. - tests: Use a helper function to refactor and simplify most tests. - partition filter: Can handle > 128 GPT partitions. - nbdkit can be compiled on the Haiku operating system (François Revol). - New nbdkit_parse_bool API for parsing boolean parameters (Eric Blake). - log: Add logappend=[true|false] (Eric Blake). - ocaml: New set_error and debug calls. - Analyze code under AddressSanitizer and ThreadSanitizer and fix bugs (Eric Blake). - Analyze code under Coverity and fix bugs. - Tests now run more quickly. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

1 0

Plan for nbdkit 1.8
by Richard W.M. Jones 09 Nov '18

09 Nov '18

I would really like to release a new stable nbdkit (1.8) soon. The reasons are not only because of the many great new features (which I summarise below) but also a few practical reasons: - Dropping mandatory [key=].. (ie. "magic config key") for many plugin parameters makes nbdkit much easier to use and easier to describe to potential users. - Demonstrated utility of log filter and error filter for testing and visualizing block devices. (https://rwmj.wordpress.com/2018/11/06/nbd-graphical-viewer-raid-5-edition/) - Lots of attention to limits / large virtual sizes (eg https://rwmj.wordpress.com/2018/09/05/nbdkit-for-loopback-pt-5-8-exabyte-bt…) I hope to give a FOSDEM talk next year and it would be nice to have a stable set of features available in most distros. If we release 1.8 soon then it could become the baseline next February. ---- The release notes would look something like this [preliminary!]: - Magic config key allows simpler invocation of many plugins, eg: nbdkit file file=foo becomes: nbdkit file foo - New plugin: 'sh' for running shell scripts and other external programs. (https://rwmj.wordpress.com/2018/09/09/write-nbdkit-plugins-in-shell-script/) - New plugin: 'partitioning' which creates a partitioned disk from a list of partitions in files. - New plugin: 'iso' which creates an ISO (CD-ROM) image on the fly from a directory of files. (https://rwmj.wordpress.com/2018/10/18/new-in-nbdkit-create-an-iso-image-on-…) - New plugin: 'floppy' which creates a virtual FAT-formatted floppy disk from a directory of files. (https://rwmj.wordpress.com/2018/10/29/new-in-nbdkit-create-a-virtual-floppy…) - data: Add disk2data.pl script to generate data parameters. - data: Use <file to inline a file. - vddk: Use '-D vddk.diskinfo=1' to enable extra debug information. - tests: Use a helper function to refactor and simplify most tests. - partition filter: Can handle > 128 GPT partitions. - nbdkit can be compiled on the Haiku operating system (François Revol). ---- Anyhow, if there's something you'd like to get into nbdkit 1.8 and/or you strongly object to any of the above, please follow up. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

1 1

guestfs_launch() fails when C application is started as a systemd service
by Peter Dimitrov 09 Nov '18

09 Nov '18

Hello, I have a simple C program that uses libguestfs to extract info about disk usage from a libvirt domain. It works when ran manually as root, but fails when started as a systemd service. I'm attaching the service file, source code and verbose logs from both the successful manual run and from the service journal. SELinix is disabled. Error messages: libguestfs: set_socket_create_context: getcon failed: (none): Invalid argument [you can ignore this message if you are not using SELinux + sVirt] libguestfs: clear_socket_create_context: setsockcreatecon failed: NULL: Invalid argument [you can ignore this message if you are not using SELinux + sVirt] libguestfs: error: chown: /tmp/libguestfsvMMaec/guestfsd.sock: Operation not permitted libguestfs: clear_socket_create_context: setsockcreatecon failed: NULL: Invalid argument [you can ignore this message if you are not using SELinux + sVirt] libguestfs: trace: launch = -1 (error) failed to launch domain: Invalid argument (Note on the service file: CapabilityBoundingSet is for future development - it seems to make no difference) Best Regards, Peter

2 8

[nbdkit PATCH v2 0/5] log appends
by Eric Blake 09 Nov '18

09 Nov '18

v2 turned out to be much more involved, as I ended up fixing several things along the way that I noticed while debugging a feature addition. Eric Blake (5): maint: Improve ./nbdkit option parsing main: Saner newline printing during --help utils: Add nbdkit_parse_bool main: Use new bool parser for --tls log: Allow user option of appending to log docs/nbdkit-plugin.pod | 11 +++++++++++ filters/log/nbdkit-log-filter.pod | 5 +++-- include/nbdkit-common.h | 1 + src/filters.c | 13 +++++++----- src/main.c | 16 ++++++--------- src/plugins.c | 13 +++++++----- src/utils.c | 25 +++++++++++++++++++++++ filters/log/log.c | 12 +++++++++-- tests/test-log.sh | 6 +++++- nbdkit.in | 33 +++++++++++++++++++------------ 10 files changed, 97 insertions(+), 38 deletions(-) -- 2.17.2

2 8

[nbdkit PATCH] log: Allow user option of appending to log
by Eric Blake 08 Nov '18

08 Nov '18

Always truncating a log can wipe out useful history. Add an option to change this. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- Hmm - logappend=false enables appending. Maybe we need a common utility function for parsing bool values (parse 0 and case-insensitive "off" or "false" as off, all others as true, or even report values that can't be parsed reasonably) filters/log/nbdkit-log-filter.pod | 5 +++-- filters/log/log.c | 8 +++++++- tests/test-log.sh | 6 +++++- 3 files changed, 15 insertions(+), 4 deletions(-) diff --git a/filters/log/nbdkit-log-filter.pod b/filters/log/nbdkit-log-filter.pod index 0903329..88b9f9b 100644 --- a/filters/log/nbdkit-log-filter.pod +++ b/filters/log/nbdkit-log-filter.pod @@ -4,7 +4,7 @@ nbdkit-log-filter - nbdkit log filter =head1 SYNOPSIS - nbdkit --filter=log plugin logfile=FILE [plugin-args...] + nbdkit --filter=log plugin logfile=FILE [logappend=VAL] [plugin-args...] =head1 DESCRIPTION @@ -22,7 +22,8 @@ work even when nbdkit is run as a daemon. The nbdkit-log-filter requires a single parameter C<logfile> which specifies the path of the file to use for logging. If the file -already exists, it will be truncated. +already exists, it will be truncated unless the C<logappend> parameter +was specified with any non-empty value other than "0". =head1 EXAMPLES diff --git a/filters/log/log.c b/filters/log/log.c index 2079084..e6ed3b4 100644 --- a/filters/log/log.c +++ b/filters/log/log.c @@ -43,6 +43,7 @@ #include <pthread.h> #include <sys/time.h> #include <assert.h> +#include <stdbool.h> #include <nbdkit-filter.h> @@ -51,6 +52,7 @@ static uint64_t connections; static char *logfilename; static FILE *logfile; +static bool append; static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; static void @@ -75,6 +77,10 @@ log_config (nbdkit_next_config *next, void *nxdata, } return 0; } + if (strcmp (key, "logappend") == 0) { + append = value[0] && (value[0] != '0' || value[1]); + return 0; + } return next (nxdata, key, value); } @@ -86,7 +92,7 @@ log_config_complete (nbdkit_next_config_complete *next, void *nxdata) nbdkit_error ("missing logfile= parameter for the log filter"); return -1; } - logfile = fopen (logfilename, "w"); + logfile = fopen (logfilename, append ? "a" : "w"); if (!logfile) { nbdkit_error ("fopen: %m"); return -1; diff --git a/tests/test-log.sh b/tests/test-log.sh index f0eacb7..b30d20e 100755 --- a/tests/test-log.sh +++ b/tests/test-log.sh @@ -45,7 +45,9 @@ if ! qemu-io -f raw -c 'w 1M 2M' log.img; then fi # Run nbdkit with logging enabled to file. -start_nbdkit -P log.pid -U log.sock --filter=log file log.img logfile=log.log +echo '# My log' > log.log +start_nbdkit -P log.pid -U log.sock --filter=log file log.img \ + logfile=log.log logappend=1 # For easier debugging, dump the final log files before removing them # on exit. @@ -61,6 +63,8 @@ cleanup_fn cleanup qemu-io -f raw -c 'w -P 11 1M 2M' 'nbd+unix://?socket=log.sock' qemu-io -r -f raw -c 'r -P 11 2M 1M' 'nbd+unix://?socket=log.sock' +# The log should have been appended, preserving our marker. +grep '# My log' log.log # The log should show a write on connection 1, and read on connection 2. grep 'connection=1 Write id=1 offset=0x100000 count=0x200000 ' log.log grep 'connection=2 Read id=1 offset=0x200000 count=0x100000 ' log.log -- 2.17.2

2 1

[nbdkit PATCH] test-layers: Avoid thread leak
by Eric Blake 08 Nov '18

08 Nov '18

Running 'make check' under a ThreadSanitizer build complains that test-layers leaks a pthread; while harmless, it is also an easy fix, by detaching the thread to ensure it gets cleaned up. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- tests/test-layers.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/test-layers.c b/tests/test-layers.c index 00e6031..7b58699 100644 --- a/tests/test-layers.c +++ b/tests/test-layers.c @@ -152,6 +152,12 @@ main (int argc, char *argv[]) perror ("pthread_create"); exit (EXIT_FAILURE); } + err = pthread_detach(&thread); + if (err) { + errno = err; + perror ("pthread_detach"); + exit (EXIT_FAILURE); + } /* Note for the purposes of this test we're not very careful about * checking for errors (except for the bare minimum) or handling the -- 2.17.2

2 2