October 2021 - Libguestfs - libguestfs List Archives

[PATCH v2 0/9] Configurable policy for handling unstable interfaces
by Markus Armbruster 29 Oct '21

29 Oct '21

Option -compat lets you configure what to do when deprecated interfaces get used. This series extends this to unstable interfaces. Works the same way. Intended for testing users of the management interfaces. It is experimental. To make it possible, I replace the "x-" naming convention by special feature flag "unstable". See PATCH 1 for rationale. v2: * Rebased * PATCH 1: Commit message revamped [Kevin], R-bys kept * PATCH 6: gen_special_features() rewritten [John] * PATCH 7: disastrous typos fixed [Philippe] Markus Armbruster (9): qapi: New special feature flag "unstable" qapi: Mark unstable QMP parts with feature 'unstable' qapi: Eliminate QCO_NO_OPTIONS for a slight simplification qapi: Tools for sets of special feature flags in generated code qapi: Generalize struct member policy checking qapi: Generalize command policy checking qapi: Generalize enum member policy checking qapi: Factor out compat_policy_input_ok() qapi: Extend -compat to set policy for unstable interfaces docs/devel/qapi-code-gen.rst | 9 +- qapi/block-core.json | 123 +++++++++++++++++------- qapi/compat.json | 6 +- qapi/migration.json | 35 +++++-- qapi/misc.json | 6 +- qapi/qom.json | 11 ++- include/qapi/compat-policy.h | 7 ++ include/qapi/qmp/dispatch.h | 6 +- include/qapi/util.h | 8 +- include/qapi/visitor-impl.h | 6 +- include/qapi/visitor.h | 17 +++- monitor/misc.c | 7 +- qapi/qapi-forward-visitor.c | 16 +-- qapi/qapi-visit-core.c | 41 ++++---- qapi/qmp-dispatch.c | 57 ++++++++--- qapi/qmp-registry.c | 4 +- qapi/qobject-input-visitor.c | 22 ++--- qapi/qobject-output-visitor.c | 13 ++- storage-daemon/qemu-storage-daemon.c | 3 +- qapi/trace-events | 4 +- qemu-options.hx | 20 +++- scripts/qapi/commands.py | 12 +-- scripts/qapi/events.py | 10 +- scripts/qapi/gen.py | 8 ++ scripts/qapi/schema.py | 11 ++- scripts/qapi/types.py | 22 +++-- scripts/qapi/visit.py | 14 +-- tests/qapi-schema/qapi-schema-test.json | 7 +- tests/qapi-schema/qapi-schema-test.out | 5 + 29 files changed, 348 insertions(+), 162 deletions(-) -- 2.31.1

4 27

Re: [Libguestfs] [libguestfs/libguestfs] Tools are not compatible with qemu (Issue #73)
by Richard W.M. Jones 29 Oct '21

29 Oct '21

On Thu, Oct 28, 2021 at 02:44:52PM -0700, Roman Shagun wrote: > Hi > I was trying to list files inside a directory of my qcow2 image using > virt-ls -a hda.qcow2 /boot > > Seems that call to qemu-img is not aligned with current version if qemu since I > have this in debug info: > Backing file specified without backing format > now flag -F for backing format is required > > Here's full debug log: > > libguestfs: trace: set_verbose true > libguestfs: trace: set_verbose = 0 > libguestfs: create: flags = 0, handle = 0x55bc796a24f0, program = virt-ls > libguestfs: trace: add_drive "hda.qcow2" "readonly:true" > libguestfs: creating COW overlay to protect original drive content > libguestfs: trace: get_tmpdir > libguestfs: trace: get_tmpdir = "/tmp" > libguestfs: trace: disk_create "/tmp/libguestfsjjrQ6y/overlay1.qcow2" "qcow2" -1 "backingfile:/home/user/Emulators/qemu/hda.qcow2" > libguestfs: command: run: qemu-img > libguestfs: command: run: \ create > libguestfs: command: run: \ -f qcow2 > libguestfs: command: run: \ -o backing_file=/home/user/Emulators/qemu/hda.qcow2 > libguestfs: command: run: \ /tmp/libguestfsjjrQ6y/overlay1.qcow2 > qemu-img: /tmp/libguestfsjjrQ6y/overlay1.qcow2: Backing file specified without backing format > Detected format of qcow2.libguestfs: error: qemu-img: /tmp/libguestfsjjrQ6y/overlay1.qcow2: qemu-img exited with error status 1, see debug messages above > libguestfs: trace: disk_create = -1 (error) > libguestfs: trace: add_drive = -1 (error) > libguestfs: trace: close > libguestfs: closing guestfs handle 0x55bc796a24f0 (state 0) > libguestfs: command: run: rm > libguestfs: command: run: \ -rf /tmp/libguestfsjjrQ6y > > I've seen similar bug described here: https://www.mail-archive.com/ > debian-bugs-dist(a)lists.debian.org/msg1816903.html You need this commit: https://github.com/libguestfs/libguestfs/commit/45de287447bb18d59749fbfc1ec… Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

1 0

[PATCH] Skip vsock-related checks if AF_VSOCK is not supported.
by Hilko Bengen 28 Oct '21

28 Oct '21

--- tests/functions.sh.in | 10 ++++++++++ tests/test-ip-filter-anyvsock.sh | 1 + tests/test-nbd-vsock.sh | 1 + tests/test-vsock.sh | 1 + 4 files changed, 13 insertions(+) diff --git a/tests/functions.sh.in b/tests/functions.sh.in index 403fa56e..d6c4255c 100644 --- a/tests/functions.sh.in +++ b/tests/functions.sh.in @@ -239,6 +239,16 @@ requires_non_root () fi } +# Tests that use the vsock interface will fail if vsock is not +# supported. +requires_vsock_support () +{ + if ! grep -q ^AF_VSOCK /proc/net/protocols; then + echo "$0: test skipped because AF_VSOCK is not supported." + exit 77 + fi +} + # start_nbdkit -P pidfile args... # # Run nbdkit with args and wait for it to start up. If it fails to diff --git a/tests/test-ip-filter-anyvsock.sh b/tests/test-ip-filter-anyvsock.sh index 3160e982..016b1da9 100755 --- a/tests/test-ip-filter-anyvsock.sh +++ b/tests/test-ip-filter-anyvsock.sh @@ -47,6 +47,7 @@ requires nbdsh --version requires nbdsh -c 'print(h.connect_vsock)' requires_nbdsh_uri requires_linux_kernel_version 5.6 +requires_vsock_support # Not supported on Windows. if is_windows; then diff --git a/tests/test-nbd-vsock.sh b/tests/test-nbd-vsock.sh index e3736898..f1f839a0 100755 --- a/tests/test-nbd-vsock.sh +++ b/tests/test-nbd-vsock.sh @@ -47,6 +47,7 @@ set -x requires_nbdsh_uri requires nbdsh -c 'print(h.connect_vsock)' requires_linux_kernel_version 5.6 +requires_vsock_support # Because vsock ports are 32 bits, we can basically pick one at random # and be sure that it's not used. However we must pick one >= 1024 diff --git a/tests/test-vsock.sh b/tests/test-vsock.sh index fc39cb7b..0bcfc4ba 100755 --- a/tests/test-vsock.sh +++ b/tests/test-vsock.sh @@ -47,6 +47,7 @@ requires nbdsh --version requires nbdsh -c 'print(h.connect_vsock)' requires_nbdsh_uri requires_linux_kernel_version 5.6 +requires_vsock_support # Because vsock ports are 32 bits, we can basically pick one at random # and be sure that it's not used. However we must pick one >= 1024 -- 2.33.0

2 3

libnbd | Failed pipeline for master | b3440853
by GitLab 28 Oct '21

28 Oct '21

Pipeline #397743538 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: b3440853 ( https://gitlab.com/nbdkit/libnbd/-/commit/b3440853cdeca0e44ad9c526e71faaa6c… ) Commit Message: lib/poll.c: Retry poll after EINTR I see a rar... Commit Author: Nir Soffer Pipeline #397743538 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/397743538 ) triggered by Nir Soffer ( https://gitlab.com/nirs ) had 4 failed jobs. Job #1726761139 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1726761139/raw ) Stage: builds Name: x86_64-freebsd-13 Job #1726761135 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1726761135/raw ) Stage: builds Name: x86_64-freebsd-12 Job #1726761146 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1726761146/raw ) Stage: builds Name: x86_64-macos-11 Job #1726761144 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1726761144/raw ) Stage: builds Name: x86_64-freebsd-current -- You're receiving this email because of your account on gitlab.com.

1 0

[PATCH libnbd] lib/poll.c: Retry poll after EINTR
by Nir Soffer 28 Oct '21

28 Oct '21

I see a rare random failure when calling BlockStatus via Go binding: block_status: nbd_block_status: poll: Interrupted system call I could not reproduce this with "nbdinfo --map", even after modifying it to call nbd_block_status() for every 128 MiB. Fixing this in nbd_unlock_poll() avoids this issue in the entire library, when we wait for command completion. This seems more useful that fixing it in all libnbd clients. Tested using a go client listing all extents in an image, calling BlockStatus for every 128m with fedora 34 qcow2 image. Without this fix, this was always failing. $ hyperfine -r1000 --show-output "./client nbd+unix://?socket=/tmp/nbd.sock > /dev/null" Benchmark 1: ./client nbd+unix://?socket=/tmp/nbd.sock > /dev/null Time (mean ± σ): 31.6 ms ± 3.1 ms [User: 8.8 ms, System: 7.2 ms] Range (min … max): 26.1 ms … 52.3 ms 1000 runs Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- lib/poll.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/poll.c b/lib/poll.c index edfcc59..df01d94 100644 --- a/lib/poll.c +++ b/lib/poll.c @@ -57,8 +57,11 @@ nbd_unlocked_poll (struct nbd_handle *h, int timeout) * would allow other threads to close file descriptors which we have * passed to poll. */ - r = poll (fds, 1, timeout); - debug (h, "poll end: r=%d revents=%x", r, fds[0].revents); + do { + r = poll (fds, 1, timeout); + debug (h, "poll end: r=%d revents=%x", r, fds[0].revents); + } while (r == -1 && errno == EINTR); + if (r == -1) { set_error (errno, "poll"); return -1; -- 2.31.1

4 5

[PATCH nbdkit 0/5 ] vddk: Implement parallel thread model
by Richard W.M. Jones 28 Oct '21

28 Oct '21

NOTE: This is not complete, I'm just posting it for early review. In particular I need to test and benchmark this a lot more to make sure it really is a performance improvement. At the moment nbdkit-vddk-plugin suffers because it works synchronously. We found in testing that copying multiple VMs in parallel is much faster than copying a single VM, the obvious reason being that each single nbdkit spends a lot of time waiting on round trips to the VMware server. Since VDDK 6.0 (but only implemented in VDDK 6.7+) the library has offered a form of asynchronous read and write operation. We can use that to implement the parallel thread model, hopefully allowing multiple outstanding read/write commands to overlap in flight. One thing of note is that multi-conn is probably not possible without a lot more work. The reason is that VDDK locks the backing disk when you open a handle, so multiple connections (ie. handles) cannot be opened. I am also not clear if VDDK/VMware offers the correct data integrity guarantees to allow us to implement multi-conn, and since the whole lot of closed source there's no easy way to find out. Rich.

4 19

[PATCH libnbd 0/2] Golang tests fixes
by Nir Soffer 28 Oct '21

28 Oct '21

- Retry interrupted syscall.Select() - Verify LibnbdError.Errno value Nir Soffer (2): golang/tests: Retry syscall.Select on EINTR golang/tests: Test expected Errno .../src/libguestfs.org/libnbd/libnbd_590_aio_copy_test.go | 7 ++++++- golang/src/libguestfs.org/libnbd/libnbd_610_error_test.go | 7 +++---- 2 files changed, 9 insertions(+), 5 deletions(-) -- 2.31.1

3 4

libnbd | Failed pipeline for master | 51195424
by GitLab 26 Oct '21

26 Oct '21

Pipeline #395672746 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 51195424 ( https://gitlab.com/nbdkit/libnbd/-/commit/51195424f8ba73d0ab1fb4145ddfa81cb… ) Commit Message: python: Fix crash when passing None as a buffer... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #395672746 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/395672746 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 4 failed jobs. Job #1716360689 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1716360689/raw ) Stage: builds Name: x86_64-freebsd-12 Job #1716360693 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1716360693/raw ) Stage: builds Name: x86_64-freebsd-13 Job #1716360702 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1716360702/raw ) Stage: builds Name: x86_64-macos-11 Job #1716360695 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1716360695/raw ) Stage: builds Name: x86_64-freebsd-current -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | 51633d0d
by GitLab 25 Oct '21

25 Oct '21

Pipeline #395195517 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 51633d0d ( https://gitlab.com/nbdkit/libnbd/-/commit/51633d0dd486e5db631160d3abcfb6bc3… ) Commit Message: Version 1.11.1. Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #395195517 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/395195517 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 4 failed jobs. Job #1713912637 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1713912637/raw ) Stage: builds Name: x86_64-freebsd-12 Job #1713912639 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1713912639/raw ) Stage: builds Name: x86_64-freebsd-current Job #1713912640 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1713912640/raw ) Stage: builds Name: x86_64-macos-11 Job #1713912638 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1713912638/raw ) Stage: builds Name: x86_64-freebsd-13 -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | 5d7b9f16
by GitLab 25 Oct '21

25 Oct '21

Pipeline #395096064 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 5d7b9f16 ( https://gitlab.com/nbdkit/libnbd/-/commit/5d7b9f16f7f037df5e5274d488780402a… ) Commit Message: golang: Fix incorrect license statement in two ... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #395096064 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/395096064 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 4 failed jobs. Job #1713433270 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1713433270/raw ) Stage: builds Name: x86_64-freebsd-12 Job #1713433273 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1713433273/raw ) Stage: builds Name: x86_64-freebsd-13 Job #1713433276 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1713433276/raw ) Stage: builds Name: x86_64-macos-11 Job #1713433275 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1713433275/raw ) Stage: builds Name: x86_64-freebsd-current -- You're receiving this email because of your account on gitlab.com.

1 0