June 2021 - Libguestfs - libguestfs List Archives

Re: [Libguestfs] oVirt import from VMware
by Richard W.M. Jones 16 Jun '21

16 Jun '21

[Adding upstream mailing list] On Mon, May 17, 2021 at 12:32:12PM -0400, Alan Daniels wrote: > > Thanks Martin. The logs (/var/log/vdsm/import on the host) > show: > > > > qemu-img: /var/tmp/v2vovle5b34c.qcow2: CURL: Error opening > file: > > Server does not support 'range' (byte ranges). > > This is unexpected, but I'd need to see the full log to be sure. > > Given the little information available (but it's vCenter 7 so it would > fit), it could be: > > https://bugzilla.redhat.com/show_bug.cgi?id=1846238 > https://bugzilla.redhat.com/show_bug.cgi?id=1841038 > > (They're the same bug, we didn't fix it in RHEL 7) > Hey Rich, > > Thanks for the help! Some more info about the environment: > > Oracle Linux 7.9 > # cat /etc/redhat-release > Red Hat Enterprise Linux Server release 7.9 (Maipo) > > OLVM / oVirt 4.3 > > virt-v2v.x86_64 1:1.40.2-10.0.1.el7 > libguestfs.x86_64 1:1.40.2-10.0.1.el7 > qemu-kvm.x86_64 15:4.2.1-6.el7 > nbdkit.x86_64 1.8.0-4.el7 > qemu-block-curl.x86_64 15:4.2.1-6.el7 > > Running on Oracle Cloud. > > VMware is on another cloud provider and they're connected via VPN. > > The logs looks essentially the same as the one from https://bugzilla.redhat.com > /show_bug.cgi?id=1846238 > > I can't really upgrade from 7.9 to 8.3, so is this expected behavior? Are there > some potential workarounds? It's a bug in qemu which you will need to patch in order to do the import using this method. Other methods don't suffer from the bug because they don't use qemu in the same way. I would recommend exploring the “-i ova” method. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

2 8

virt-builder fails in fedora 34 container:
by Nir Soffer 16 Jun '21

16 Jun '21

I'm running this inside a fedora 34 container in ovirt-imageio CI: 'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', '--root-password', 'password:', '-o', '/var/tmp/imageio-images/cirros-0.3.5' The command fails in: supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory") libguestfs: trace: launch = -1 (error) virt-builder: error: libguestfs error: /usr/bin/supermin exited with error status 1, see debug messages above For full error log see: https://travis-ci.com/github/nirs/ovirt-imageio/jobs/513305268 The container file is: https://github.com/oVirt/ovirt-imageio/blob/master/docker/Dockerfile.fedora… The same code works with centos stream 8 container and fedora 32, 33 containers. Based on this error: supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory") I think I'm missing some package removed from the fedora 34 container, but I don't have any idea which one. I hope someone has more insight on this. Nir

3 15

[RFC PATCH 0/2] New NBD metacontext
by Eric Blake 14 Jun '21

14 Jun '21

This is my counter-proposal to Nir's request [1] to revert a 6.0 behavior change. It does not expose any new information over NBD, but does make it easier to collect necessary information from a single context rather than requiring the client to have to request two contexts in parallel, then cross-correlate what may be different extent lengths between those contexts. Furthermore, this is easy to backport to downstream based on qemu 6.0, at which point clients could use the existence or absence of qemu:joint-allocation as a witness of whether it can get away with trusting base:allocation when trying to recreate a qcow2 backing chain. [1] https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg01796.html Things I still want to do: - a followup patch to libnbd to teach 'nbdinfo --map=qemu:joint-allocation' to decode the bits - teach 'nbdinfo --map' to read all available contexts, instead of having to manually type each map besides base:allocation - potential followup patches to qemu to automatically feed this information through qemu-img map: - add a new BDRV_BLOCK_BACKING bit for bdrv_block_status(), with opposite semantics from BDRV_BLOCK_ALLOCATED, but where the only thing known is that the data is not local (not how deep it is) - teach qemu to favor qemu:joint-allocation over base:allocation when available, and use it to drive BDRV_BLOCK_BACKING - teach qemu-img map to recognize BDRV_BLOCK_BACKING Eric Blake (2): iotests: Improve and rename test 309 to nbd-qemu-allocation nbd: Add new qemu:joint-allocation metadata context docs/interop/nbd.txt | 31 ++++++- docs/tools/qemu-nbd.rst | 4 +- qapi/block-export.json | 4 +- include/block/nbd.h | 10 ++- nbd/server.c | 87 +++++++++++++++++-- .../{309 => tests/nbd-qemu-allocation} | 5 +- .../nbd-qemu-allocation.out} | 13 ++- 7 files changed, 139 insertions(+), 15 deletions(-) rename tests/qemu-iotests/{309 => tests/nbd-qemu-allocation} (95%) rename tests/qemu-iotests/{309.out => tests/nbd-qemu-allocation.out} (79%) -- 2.31.1

3 18

[libnbd PATCH v2 00/23] Many CI fixes
by Martin Kletzander 11 Jun '21

11 Jun '21

Lot of clean-ups for the CI. I also added support for easily configurable way of skipping particular tests on selected distros. The only job in the pipeline that is not green with these patches is the MacOS build and that will require more work. v2: - struct sockaddr_vm fix for MacOS - bash completion fixed - no need to skip dist on opensuse - fixed check for qemu-storage-daemon version - also run "brew upgrade" on MacOS - incorporated reviews Martin Kletzander (23): Include sys/vsock.h if linux/vm_sockets.h is not available bash: Generate completion files during build fuse: Only support defined fallocate modes interop: Do not test against broken qemu-storage-daemon tests: Fix hexdump probing tests: Clean up qemu-nbd detection tests: Enhance fuse probing tests: Fix port randomisation tests: Require cap_sys_admin where root is required tests: Do not use magic parameter for some nbdkit plugins tests: Rename version to get-version ci: Change docker driver and archive logs ci: Remove cross-compilation targets ci: Remove debian-10-clang build ci: Add support for FreeBSD-13.0 ci: Fix run commands in README ci: Some build script fixes ci: Add support for test skipping ci: Rename build_script.sh to just build.sh ci: Skip some broken tests/distro combinations ci: Only run `make check` on Linux ci: Also perform `brew upgrade` on MacOS ci: Update build files configure.ac | 2 +- lib/connect.c | 2 + tests/Makefile.am | 18 +- tests/functions.sh.in | 58 ++++ tests/{version.c => get-version.c} | 0 interop/interop-qemu-storage-daemon.sh | 6 +- .gitignore | 2 +- .gitlab-ci.yml | 247 ++---------------- bash/Makefile.am | 16 +- ci/README.rst | 16 +- ci/build.sh | 117 +++++++++ ci/build_script.sh | 56 ---- ci/cirrus/build.yml | 3 +- ci/cirrus/freebsd-12.vars | 5 +- ci/cirrus/freebsd-13.vars | 13 + ci/cirrus/freebsd-current.vars | 5 +- ci/cirrus/macos-11.vars | 6 +- ci/containers/centos-8.Dockerfile | 12 +- ci/containers/centos-stream-8.Dockerfile | 12 +- .../debian-10-cross-aarch64.Dockerfile | 86 ------ .../debian-10-cross-armv6l.Dockerfile | 86 ------ .../debian-10-cross-armv7l.Dockerfile | 86 ------ ci/containers/debian-10-cross-i686.Dockerfile | 86 ------ ci/containers/debian-10-cross-mips.Dockerfile | 86 ------ .../debian-10-cross-mips64el.Dockerfile | 86 ------ .../debian-10-cross-mipsel.Dockerfile | 86 ------ .../debian-10-cross-ppc64le.Dockerfile | 86 ------ .../debian-10-cross-s390x.Dockerfile | 86 ------ ci/containers/debian-10.Dockerfile | 9 +- .../debian-sid-cross-aarch64.Dockerfile | 86 ------ .../debian-sid-cross-armv6l.Dockerfile | 86 ------ .../debian-sid-cross-armv7l.Dockerfile | 86 ------ .../debian-sid-cross-i686.Dockerfile | 86 ------ .../debian-sid-cross-mips64el.Dockerfile | 86 ------ .../debian-sid-cross-mipsel.Dockerfile | 86 ------ .../debian-sid-cross-ppc64le.Dockerfile | 86 ------ .../debian-sid-cross-s390x.Dockerfile | 86 ------ ci/containers/debian-sid.Dockerfile | 9 +- ci/containers/fedora-33.Dockerfile | 14 +- ci/containers/fedora-34.Dockerfile | 14 +- .../fedora-rawhide-cross-mingw32.Dockerfile | 71 ----- .../fedora-rawhide-cross-mingw64.Dockerfile | 71 ----- ci/containers/fedora-rawhide.Dockerfile | 14 +- ci/containers/opensuse-leap-152.Dockerfile | 12 +- ci/containers/opensuse-tumbleweed.Dockerfile | 12 +- ci/containers/refresh | 20 -- ci/containers/ubuntu-1804.Dockerfile | 8 +- ci/containers/ubuntu-2004.Dockerfile | 9 +- ci/skipped_tests | 12 + copy/copy-block-to-nbd.sh | 3 +- copy/copy-file-to-qcow2.sh | 17 +- copy/copy-nbd-to-block.sh | 3 +- copy/copy-nbd-to-file.sh | 2 +- copy/copy-nbd-to-nbd.sh | 2 +- copy/copy-nbd-to-small-block-error.sh | 5 +- copy/copy-tls.sh | 2 +- fuse/operations.c | 8 + fuse/test-errors.sh | 2 +- fuse/test-file-mode.sh | 3 +- fuse/test-nbdkit-command.sh | 2 +- fuse/test-nbdkit-file-null.sh | 2 +- fuse/test-nbdkit.sh | 3 +- fuse/test-parallel.sh | 2 +- fuse/test-pattern.sh | 4 +- fuse/test-qcow2.sh | 3 +- fuse/test-trim.sh | 2 +- fuse/test-unix.sh | 2 +- fuse/test-zero.sh | 2 +- info/Makefile.am | 1 + info/info-description-qemu.sh | 17 +- info/info-list-json-qemu.sh | 17 +- info/info-list-qemu.sh | 17 +- info/info-map-qemu-allocation-depth.sh | 16 +- info/info-map-qemu-dirty-bitmap.sh | 23 +- 74 files changed, 400 insertions(+), 2093 deletions(-) rename tests/{version.c => get-version.c} (100%) create mode 100755 ci/build.sh delete mode 100755 ci/build_script.sh create mode 100644 ci/cirrus/freebsd-13.vars delete mode 100644 ci/containers/debian-10-cross-aarch64.Dockerfile delete mode 100644 ci/containers/debian-10-cross-armv6l.Dockerfile delete mode 100644 ci/containers/debian-10-cross-armv7l.Dockerfile delete mode 100644 ci/containers/debian-10-cross-i686.Dockerfile delete mode 100644 ci/containers/debian-10-cross-mips.Dockerfile delete mode 100644 ci/containers/debian-10-cross-mips64el.Dockerfile delete mode 100644 ci/containers/debian-10-cross-mipsel.Dockerfile delete mode 100644 ci/containers/debian-10-cross-ppc64le.Dockerfile delete mode 100644 ci/containers/debian-10-cross-s390x.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-aarch64.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-armv6l.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-armv7l.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-i686.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-mips64el.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-mipsel.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-ppc64le.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-s390x.Dockerfile delete mode 100644 ci/containers/fedora-rawhide-cross-mingw32.Dockerfile delete mode 100644 ci/containers/fedora-rawhide-cross-mingw64.Dockerfile create mode 100644 ci/skipped_tests -- 2.32.0

2 28

[libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping
by Eric Blake 10 Jun '21

10 Jun '21

Qemu has unfortunately used two distinct meanings for 'allocated' over time, and that ambiguity has overloaded the word to make it difficult to know which sense is meant in a given context. In one usage, 'allocated' implies that data is present and occupies space on a block device (the opposite of sparse). In another usage, 'allocated' implies that guest contents come from the current layer of a qcow2 backing chain (even if via a sparse zero cluster), rather than deferring to the backing file. It doesn't help that in qemu source code, BDRV_BLOCK_ALLOCATED refers solely to this latter usage, while a preallocated qcow2 image refers to the former usage. But we can help by using better terminology in nbdinfo output. With base:allocation, either a cluster is known to be sparse (a hole), or it (likely) occupies space (remember, NBD block status bits are advisory - if they are set to 1, we know a given property is true; but if they are left 0, we cannot assume the opposite of the property). Instead of calling the latter allocated, we can call it merely "data". And this maps a bit better to 'qemu-img map --output=json' output; a sparse section really is "data":false, and everything that is not known to be sparse is "data":true. Similarly, in qemu:allocation-depth, either a cluster is known to come from the backing chain (either locally, or from a given backing depth), or it is completely absent from the chain (not allocated in any of the layers). Instead of calling it "unallocated", we can call it "absent". Adjust the testsuite to match our improved terminology. Thanks: Nir Soffer --- info/info-map-base-allocation-json.sh | 4 ++-- info/info-map-base-allocation-large.sh | 8 ++++---- info/info-map-base-allocation-weird.sh | 4 ++-- info/info-map-base-allocation.sh | 6 +++--- info/info-map-qemu-allocation-depth.sh | 4 ++-- info/map.c | 4 ++-- 6 files changed, 15 insertions(+), 15 deletions(-) diff --git a/info/info-map-base-allocation-json.sh b/info/info-map-base-allocation-json.sh index 961eac6..b960295 100755 --- a/info/info-map-base-allocation-json.sh +++ b/info/info-map-base-allocation-json.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -40,7 +40,7 @@ jq . < $out test $( jq -r '.[0].offset' < $out ) -eq 0 test $( jq -r '.[0].length' < $out ) -eq 32768 test $( jq -r '.[0].type' < $out ) -eq 0 -test $( jq -r '.[0].description' < $out ) = "allocated" +test $( jq -r '.[0].description' < $out ) = "data" test $( jq -r '.[3].offset' < $out ) -eq 163840 test $( jq -r '.[3].length' < $out ) -eq 884736 diff --git a/info/info-map-base-allocation-large.sh b/info/info-map-base-allocation-large.sh index e7ff904..ab10dc7 100755 --- a/info/info-map-base-allocation-large.sh +++ b/info/info-map-base-allocation-large.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -36,11 +36,11 @@ nbdkit -U - data data='1 @131072 2 @6442450944 3' size=8G \ cat $out -if [ "$(tr -s ' ' < $out)" != " 0 32768 0 allocated +if [ "$(tr -s ' ' < $out)" != " 0 32768 0 data 32768 98304 3 hole,zero - 131072 32768 0 allocated + 131072 32768 0 data 163840 6442287104 3 hole,zero -6442450944 32768 0 allocated +6442450944 32768 0 data 6442483712 2147450880 3 hole,zero" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-base-allocation-weird.sh b/info/info-map-base-allocation-weird.sh index 20ddec6..0b0343a 100755 --- a/info/info-map-base-allocation-weird.sh +++ b/info/info-map-base-allocation-weird.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -45,7 +45,7 @@ EOF cat $out if [ "$(tr -s ' ' < $out)" != " 0 16 1 hole - 16 4 0 allocated + 16 4 0 data 20 12 1 hole" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-base-allocation.sh b/info/info-map-base-allocation.sh index 9b85c97..76c2024 100755 --- a/info/info-map-base-allocation.sh +++ b/info/info-map-base-allocation.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -36,9 +36,9 @@ nbdkit -U - data data='1 @131072 2' size=1M \ cat $out -if [ "$(tr -s ' ' < $out)" != " 0 32768 0 allocated +if [ "$(tr -s ' ' < $out)" != " 0 32768 0 data 32768 98304 3 hole,zero - 131072 32768 0 allocated + 131072 32768 0 data 163840 884736 3 hole,zero" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-qemu-allocation-depth.sh b/info/info-map-qemu-allocation-depth.sh index a5dab0b..9aeffd1 100755 --- a/info/info-map-qemu-allocation-depth.sh +++ b/info/info-map-qemu-allocation-depth.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -84,7 +84,7 @@ if [ "$(tr -s ' ' < $out)" != "\ 3145728 2097152 3 backing depth 3 5242880 1048576 1 local 6291456 1048576 2 backing depth 2 - 7340032 1048576 0 unallocated" ]; then + 7340032 1048576 0 absent" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 fi diff --git a/info/map.c b/info/map.c index ae6d4fe..82c9507 100644 --- a/info/map.c +++ b/info/map.c @@ -202,7 +202,7 @@ extent_description (const char *metacontext, uint32_t type) if (strcmp (metacontext, "base:allocation") == 0) { switch (type) { - case 0: return strdup ("allocated"); + case 0: return strdup ("data"); case 1: return strdup ("hole"); case 2: return strdup ("zero"); case 3: return strdup ("hole,zero"); @@ -216,7 +216,7 @@ extent_description (const char *metacontext, uint32_t type) } else if (strcmp (metacontext, "qemu:allocation-depth") == 0) { switch (type) { - case 0: return strdup ("unallocated"); + case 0: return strdup ("absent"); case 1: return strdup ("local"); default: if (asprintf (&ret, "backing depth %u", type) == -1) { -- 2.31.1

3 3

[libnbd PATCH 00/20] Many CI fixes
by Martin Kletzander 10 Jun '21

10 Jun '21

Lot of clean-ups for the CI. I also added support for easily configurable way of skipping particular tests on selected distros. However, I only skipped those that do not have and easy way of detecting a broken dependency and such like. What is still missing is: - All the CentOS builds have issues with three tests that seem like a protocol violation, but can very well be a bug in some test dependency. Since this is the closest to RHEL we can get, I think it's worth investigating. - FreeBSDs do not even build and it looks like some libtool wrapper script is fed into a C preprocessor, which fails terribly due to the script comments (but should not be fed there at all, of course. - MacOS has issues, although that one is marked to allow failures and the issue does not seem to be on our side anyway. I won't look into it for a while and see if it sorts itself out. Martin Kletzander (20): fuse: Only support defined fallocate modes interop: Do not test against broken qemu-storage-daemon nbdsh: Check for python-exec2c installations tests: Fix hexdump probing tests: Clean up qemu-nbd detection in tests: Enhance fuse probing tests: Fix port randomisation tests: Require cap_sys_admin where root is required tests: Do not use magic parameter for some nbdkit plugins ci: Change docker driver and archive logs ci: Remove cross-compilation targets ci: Remove debian-10-clang build ci: Update build files ci: Add support for FreeBSD-13.0 ci: Fix run commands in README ci: Some build script fixes ci: Add support for test skipping ci: Rename build_script.sh to just build.sh ci: Skip some broken tests/distro combinations ci: Disable `make dist` on openSUSE tests/Makefile.am | 6 +- tests/functions.sh.in | 58 ++++ interop/interop-qemu-storage-daemon.sh | 5 +- .gitlab-ci.yml | 247 ++---------------- ci/README.rst | 16 +- ci/build.sh | 112 ++++++++ ci/build_script.sh | 56 ---- ci/cirrus/build.yml | 2 +- ci/cirrus/freebsd-12.vars | 5 +- ci/cirrus/freebsd-13.vars | 13 + ci/cirrus/freebsd-current.vars | 5 +- ci/cirrus/macos-11.vars | 6 +- ci/containers/centos-8.Dockerfile | 12 +- ci/containers/centos-stream-8.Dockerfile | 12 +- .../debian-10-cross-aarch64.Dockerfile | 86 ------ .../debian-10-cross-armv6l.Dockerfile | 86 ------ .../debian-10-cross-armv7l.Dockerfile | 86 ------ ci/containers/debian-10-cross-i686.Dockerfile | 86 ------ ci/containers/debian-10-cross-mips.Dockerfile | 86 ------ .../debian-10-cross-mips64el.Dockerfile | 86 ------ .../debian-10-cross-mipsel.Dockerfile | 86 ------ .../debian-10-cross-ppc64le.Dockerfile | 86 ------ .../debian-10-cross-s390x.Dockerfile | 86 ------ ci/containers/debian-10.Dockerfile | 9 +- .../debian-sid-cross-aarch64.Dockerfile | 86 ------ .../debian-sid-cross-armv6l.Dockerfile | 86 ------ .../debian-sid-cross-armv7l.Dockerfile | 86 ------ .../debian-sid-cross-i686.Dockerfile | 86 ------ .../debian-sid-cross-mips64el.Dockerfile | 86 ------ .../debian-sid-cross-mipsel.Dockerfile | 86 ------ .../debian-sid-cross-ppc64le.Dockerfile | 86 ------ .../debian-sid-cross-s390x.Dockerfile | 86 ------ ci/containers/debian-sid.Dockerfile | 9 +- ci/containers/fedora-33.Dockerfile | 14 +- ci/containers/fedora-34.Dockerfile | 14 +- .../fedora-rawhide-cross-mingw32.Dockerfile | 71 ----- .../fedora-rawhide-cross-mingw64.Dockerfile | 71 ----- ci/containers/fedora-rawhide.Dockerfile | 14 +- ci/containers/opensuse-leap-152.Dockerfile | 12 +- ci/containers/opensuse-tumbleweed.Dockerfile | 12 +- ci/containers/refresh | 20 -- ci/containers/ubuntu-1804.Dockerfile | 8 +- ci/containers/ubuntu-2004.Dockerfile | 9 +- ci/skipped_tests | 9 + copy/copy-block-to-nbd.sh | 3 +- copy/copy-file-to-qcow2.sh | 16 +- copy/copy-nbd-to-block.sh | 3 +- copy/copy-nbd-to-file.sh | 2 +- copy/copy-nbd-to-nbd.sh | 2 +- copy/copy-nbd-to-small-block-error.sh | 5 +- copy/copy-tls.sh | 2 +- fuse/operations.c | 8 + fuse/test-errors.sh | 2 +- fuse/test-file-mode.sh | 3 +- fuse/test-nbdkit-command.sh | 2 +- fuse/test-nbdkit-file-null.sh | 2 +- fuse/test-nbdkit.sh | 3 +- fuse/test-parallel.sh | 2 +- fuse/test-pattern.sh | 4 +- fuse/test-qcow2.sh | 3 +- fuse/test-trim.sh | 2 +- fuse/test-unix.sh | 2 +- fuse/test-zero.sh | 2 +- info/Makefile.am | 1 + info/info-description-qemu.sh | 17 +- info/info-list-json-qemu.sh | 17 +- info/info-list-qemu.sh | 17 +- info/info-map-qemu-allocation-depth.sh | 16 +- info/info-map-qemu-dirty-bitmap.sh | 23 +- sh/nbdsh.in | 3 +- 70 files changed, 377 insertions(+), 2074 deletions(-) create mode 100755 ci/build.sh delete mode 100755 ci/build_script.sh create mode 100644 ci/cirrus/freebsd-13.vars delete mode 100644 ci/containers/debian-10-cross-aarch64.Dockerfile delete mode 100644 ci/containers/debian-10-cross-armv6l.Dockerfile delete mode 100644 ci/containers/debian-10-cross-armv7l.Dockerfile delete mode 100644 ci/containers/debian-10-cross-i686.Dockerfile delete mode 100644 ci/containers/debian-10-cross-mips.Dockerfile delete mode 100644 ci/containers/debian-10-cross-mips64el.Dockerfile delete mode 100644 ci/containers/debian-10-cross-mipsel.Dockerfile delete mode 100644 ci/containers/debian-10-cross-ppc64le.Dockerfile delete mode 100644 ci/containers/debian-10-cross-s390x.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-aarch64.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-armv6l.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-armv7l.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-i686.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-mips64el.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-mipsel.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-ppc64le.Dockerfile delete mode 100644 ci/containers/debian-sid-cross-s390x.Dockerfile delete mode 100644 ci/containers/fedora-rawhide-cross-mingw32.Dockerfile delete mode 100644 ci/containers/fedora-rawhide-cross-mingw64.Dockerfile create mode 100644 ci/skipped_tests -- 2.31.1

3 40

GSoC: Next step nbdcopy (file-ops)
by Abhay Raj Singh 09 Jun '21

09 Jun '21

I have been analyzing the code base and I think I should start with nbdcopy file-ops as it's easier to work on. I will try different approaches based on the features provided by liburing. I might need to modify the interfaces `asynch_{read, write}` which is easier in the case of file operations. I will update this thread regarding progress and question. I plan to add iouring_prep_read based functionality till Sunday. Regards, Abhay

1 0

ANNOUNCE: libnbd 1.8 and nbdkit 1.26 - high performance network block device client and server
by Richard W.M. Jones 07 Jun '21

07 Jun '21

I'm pleased to announce the release of nbdkit 1.26, a high performance plugin-based Network Block Device (NBD) server, and libnbd 1.8, a high performance client library. https://en.wikipedia.org/wiki/Network_block_device You can get these from: tarballs: https://download.libguestfs.org/nbdkit/ https://download.libguestfs.org/libnbd/ git: https://gitlab.com/nbdkit Key features of nbdkit: * Multithreaded NBD server written in C with good performance. * Minimal dependencies for the basic server. * Liberal license (BSD) allows nbdkit to be linked to proprietary libraries or included in proprietary code. * Well-documented, simple plugin API with a stable ABI guarantee. Lets you export “unconventional” block devices easily. * You can write plugins in C, Go, Lua, Perl, Python, OCaml, Ruby, Rust, shell script or Tcl. * Filters can be stacked in front of plugins to transform the output. Key features of libnbd: * Synchronous API for ease of use. * Asynchronous API for writing non-blocking, multithreaded clients. You can mix both APIs freely. * High performance. * Minimal dependencies for the basic library. * Well-documented, stable API. * Bindings in several programming languages. * Shell (nbdsh) for command line and scripting. * Copying tool (nbdcopy) for high performance copying and streaming. * Query tool (nbdinfo) to query NBD servers. * FUSE support (nbdfuse) to mount NBD in the local filesystem. Release notes for nbdkit 1.26 nbdkit is now hosted on gitlab: https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html Security There were no security issues found. All past security issues and information about how to report new ones can be found in nbdkit-security(1). Plugins nbdkit-gzip-plugin and nbdkit-tar-plugin have been removed. They were deprecated in nbdkit 1.22. The functionality has been replaced by nbdkit-gzip-filter(1) and nbdkit-tar-filter(1) respectively. nbdkit-streaming-plugin(1) has been deprecated and will be removed in nbdkit 1.30. You can use nbdcopy(1) instead. nbdkit-curl-plugin(1) adds these features: "followlocation=false" to disable following redirects. "cookiefile" and "cookiejar" to control saving cookies. nbdkit-ssh-plugin(1) now advertises multi-conn if the server is OpenSSH, improving performance (Eric Blake). nbdkit-ondemand-plugin(1) has a new "wait=true" flag allowing you to choose between clients waiting for an export to become free or being immediately rejected. nbdkit-sparse-random-plugin(1) now does not generate fully random data within each block. Using perf we observed that the plugin was spending 34% of the total time during benchmarks just doing this. You can enable the old behaviour using "random-content=true". nbdkit-file-plugin(1) now checks if the file is read-only and exports a read-only NBD connection in this case (Eric Blake). nbdkit-vddk-plugin(1) has been tested with VDDK 7.0.2. Filters There is a new API for use by filters to manage "contexts". Instead of the previous 1-1 relationship between filter connections and plugin connections, filters can now open more than one context into the backend from a single connection, or share contexts between connections (Eric Blake). New nbdkit-multi-conn-filter(1) which allows you to add multi-conn support to plugins which don't support it (although the emulation is expensive), or change whether multi-conn is advertised to clients (Eric Blake). nbdkit-cow-filter(1) now ignores flush/FUA requests (which are not useful for a temporary overlay). The filter now implements extents and efficient trimming. Fine-grained locking is used around shared structures, greatly improving parallel performance. Added support for multi-conn. nbdkit-cache-filter(1) advertises multi-conn when possible (Eric Blake). nbdkit-cache-filter and nbdkit-cow-filter can now handle arbitrary sized disks, instead of rounding the disk size down. nbdkit-blocksize-filter(1) supports parallel requests (Eric Blake). nbdkit-checkwrite-filter(1) supports multi-conn (Eric Blake). nbdkit-ext2-filter(1) no longer supports multi-conn. The filter is single-threaded so it does not matter, but it was not safe. Fix trim, zero and flush support (Eric Blake). nbdkit-readahead-filter(1) fixes multi-conn support (Eric Blake). nbdkit-gzip-filter(1) and nbdkit-xz-filter(1) now both support multi- conn (Eric Blake). Language bindings nbdkit-python-plugin(3) supports the following new features: "after_fork" and "preconnect" methods. "nbdkit.shutdown()" binding. nbdkit-rust-plugin(3) updated for newer Rust compiler versions (Alan Somers). nbdkit-golang-plugin(3) now uses and requires golang modules. Golang ≥ 1.13 must be used. API New ".cleanup" callback for plugins and filters. This is called shortly before ".unload", but unlike ".unload" it is guaranteed that no plugin or filter code has been dlclosed (Eric Blake). New "nbdkit_extents_full" is a helper function for filters which can be used to read all extents from a plugin covering a region. Tests Old plugins that we use for testing were rebuilt so they no longer depend on libselinux (thanks Thomas Weißschuh). nbdkit-ssh-plugin(1) can still be built even if ssh-keygen is unavailable for testing it (thanks Thomas Weißschuh). Improve tests/test-captive.sh to avoid crashes during testing. Test coverage on FreeBSD has been improved, fewer tests should skip unnecessarily. Build nbdkit and plugins can now be built with "-fvisibility=hidden", which allows the compiler to make some optimizations. Add a dependency so nbdkit is relinked if the linker script (nbdkit.syms) changes (Eric Blake). xorriso is now preferred over genisoimage or mkisofs (see https://wiki.debian.org/genisoimage) It is now possible to compile nbdkit with clang with warnings enabled. Several warnings have been suppressed or fixed. "MALLOC_CHECK_" and "MALLOC_PERTURB_" are only enabled when running the tests, not when running nbdkit from the build directory. This makes it easier to do benchmarks. "nbdkit_peer_*" implemented on OpenBSD. Internals When fuzzing nbdkit we now recommend using the "-t 1" flag (which disables threads). This improves the fuzz-stability of nbdkit. AUTHORS Authors of nbdkit 1.26: • Alan Somers • Eric Blake • Michael Ablassmeier • Richard W.M. Jones • Timm Bäder Release notes for libnbd 1.8 libnbd is now hosted on gitlab: https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html Security If you find a security issue, please read SECURITY in the source (online here: https://gitlab.com/nbdkit/libnbd/blob/master/SECURITY) To find out about previous security issues in libnbd, see libnbd-security(3). There was one security bug found in libnbd during this release cycle. CVE-2021-20286 denial of service when using nbd_set_opt_mode(3) See the full announcement here: https://listman.redhat.com/archives/libguestfs/2021-March/msg00092.html (Found and fixed by Eric Blake). New APIs nbd_get_private_data(3) nbd_set_private_data(3) These calls allow you to store either an unsigned integer or a pointer in the handle for the application to use for its own purposes. nbd_get_uri(3) This call constructs an NBD URI which can be used to connect back to the same server (using nbd_connect_uri(3) or from other tools that support the NBD URI standard). Enhancements to existing APIs Numeric IPv6 URIs (like "nbd://[::1]/") are now supported. libnbd gives a better error message if the server backlog overflows (thanks Xin Long, Lukas Doktor, Eric Blake, Martin Kletzander). New features Continuous integration (CI) tests now run on every commit and merge request to the upstream repository (Martin Kletzander). Tools A great deal of work has been done to improve the performance of nbdcopy(1) (Nir Soffer, Eric Blake). On Linux, nbdcopy now uses readahead when reading from files, and takes steps to preserve the host page cache. nbdcopy new --request-size option to allow controlling the size of requests made to NBD servers. Adjusting this can affect performance (Nir Soffer). nbdcopy now supports "null:" pseudo-target, where it discards the output. This is useful for benchmarking. nbdcopy new -v flag to enable libnbd and other debugging. nbdinfo(1) --list (which lists all exports of a server) no longer exits early if one of the exports is inaccessible or there is a recoverable error (Eric Blake). nbdinfo now prints the URI of each export in the output making it easier to connect back to each export. nbdfuse(1) is now multithreaded and supports issuing parallel NBD commands on a single connection as well as multi-conn, and other enhancements have been made. This resulted in around 50% performance improvement when tested with fio. nbdfuse now supports nbdcopy-style "[ CMD ]", eg: nbdfuse dir/ramdisk [ nbdkit memory 1G ] nbdfuse now supports trimming. Also it supports fast zeroing, but this requires a forthcoming kernel patch before it will work. nbdfuse sets the virtual file permissions appropriately if the NBD export is read-only. nbdfuse new -v flag to enable libnbd and other debugging. Tests Added a test of interoperability with qemu-storage-daemon(1). Note that qemu-storage-daemon was broken in qemu 6.0.0 so this test can fail unless you update to a fixed version (thanks Stefan Hajnoczi, Daniel Berrangé). Added a test of nbdcopy with TLS which was previously untested. Python code style tests are now opt-in using ./configure --enable-python-code-style More tests should now run on FreeBSD and not be skipped. nbdfuse tests now ensure that errors from NBD are transmitted through FUSE back to the local client. "MALLOC_CHECK_" and "MALLOC_PERTURB_" are only enabled when running the tests, not when running libnbd programs from the build directory. This makes it easier to do benchmarks. Other improvements and bug fixes The minimum C compiler is now ISO C99 (previously ANSI C90). FUSE 3 (instead of FUSE 2) is now required to build nbdfuse. The library tries harder not to leak thread-local storage memory on dlclose or exit. (Because of the design of Linux it is not always possible to avoid this, especially in multithreaded programs.) Fix the progress bar in nbdcopy(1). UTF-8 export names are now processed and displayed properly by nbdinfo(1). The --help output of nbdcopy, nbdfuse and nbdinfo now displays a brief summary of all options and has some examples. The generator now only requires ocamlc, not ocamlfind (Abhay Raj Singh). Golang bindings now use and require golang modules. Documentation Document using libnbd with the Meson build system. Added Python examples. Add example of how to integrate libnbd with libev (Nir Soffer). AUTHORS Eric Blake Richard W.M. Jones Martin Kletzander Nir Soffer -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

1 0

GSoC: Adding liburing dependency to libnbd
by Abhay Raj Singh 07 Jun '21

07 Jun '21

I think I can reuse the code below (libev part in configure.ac) and modify it for liburing, there is the libxml2 part that provides an option to disable libxml2 parts which I can reuse to provide similar behavior which is simpler the without headers and lib checks, I feel libev metho PKG_CHECK_MODULES([LIBEV], [libev], [ AC_SUBST([LIBEV_CFLAGS]) AC_SUBST([LIBEV_LIBS]) ],[ dnl no pkg-config for libev, searching manually: AC_CHECK_HEADERS([ev.h], [ AC_CHECK_LIB([ev], [ev_time], [ AC_SUBST([LIBEV_LIBS], ["-lev"]) ], [ AC_MSG_WARN([libev not found, some examples will not be compiled]) ]) ],[ AC_MSG_WARN([ev.h not found, some examples will not be compiled]) ]) ]) AS_IF([test "x$LIBEV_LIBS" != "x"], [ old_CFLAGS="$CFLAGS" CFLAGS="$CFLAGS $LIBEV_CFLAGS -Werror=strict-aliasing -O2" AC_MSG_CHECKING([if the compiler is new enough for good aliasing rules]) AC_COMPILE_IFELSE([ AC_LANG_PROGRAM([ #include <ev.h> static void cb (struct ev_loop *l, ev_timer *t, int e) { } static ev_timer timer; ], [ ev_timer_init (&timer, cb, 0, .1); ]) ], [ AC_MSG_RESULT([yes]) ], [ AC_MSG_RESULT([no]) LIBEV_CFLAGS="$LIBEV_CFLAGS -Wno-strict-aliasing" ]) CFLAGS="$old_CFLAGS" ]) AM_CONDITIONAL([HAVE_LIBEV], [test "x$LIBEV_LIBS" != "x"])

2 2

Re: [Libguestfs] [libguestfs/virt-v2v] RFE: virt-v2v: Support conversion of FreeBSD guests (#7)
by Richard W.M. Jones 04 Jun '21

04 Jun '21

On Fri, Jun 04, 2021 at 03:38:13AM -0700, tcler.yin wrote: > get fail while convert EMC OneFS simulator(based on FreeBSD) with virt-v2v: > ''' > [jianhong@fstest 9.1.0.0]$ virt-v2v -i ova 9.1.0.0.ova -o local -os tmp > [ 0.0] Opening the source -i ova 9.1.0.0.ova > [ 1.9] Creating an overlay to protect the source from being modified > [ 6.3] Opening the overlay > [ 14.6] Inspecting the overlay > virt-v2v: error: mount: mount_stub: /dev/mirror/root0: No such file or > directory > > If reporting bugs, run virt-v2v with debugging enabled and include the > complete output: > > virt-v2v -v -x [...] > ''' > > Application scenario: > deploy EMC OneFS simulator automatically for automate tests > see also: https://bugzilla.redhat.com/show_bug.cgi?id=1947291#c2 I don't know what "EMC OneFS simulator" is, but virt-v2v only supports certain guests: https://libguestfs.org/virt-v2v-support.1.html Adding support for new guest types is complicated. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

1 0