Re: [Libguestfs] oVirt import from VMware
by Richard W.M. Jones
[Adding upstream mailing list]
On Mon, May 17, 2021 at 12:32:12PM -0400, Alan Daniels wrote:
> > Thanks Martin. The logs (/var/log/vdsm/import on the host)
> show:
> >
> > qemu-img: /var/tmp/v2vovle5b34c.qcow2: CURL: Error opening
> file:
> > Server does not support 'range' (byte ranges).
>
> This is unexpected, but I'd need to see the full log to be sure.
>
> Given the little information available (but it's vCenter 7 so it would
> fit), it could be:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1846238
> https://bugzilla.redhat.com/show_bug.cgi?id=1841038
>
> (They're the same bug, we didn't fix it in RHEL 7)
> Hey Rich,
>
> Thanks for the help! Some more info about the environment:
>
> Oracle Linux 7.9
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.9 (Maipo)
>
> OLVM / oVirt 4.3
>
> virt-v2v.x86_64 1:1.40.2-10.0.1.el7
> libguestfs.x86_64 1:1.40.2-10.0.1.el7
> qemu-kvm.x86_64 15:4.2.1-6.el7
> nbdkit.x86_64 1.8.0-4.el7
> qemu-block-curl.x86_64 15:4.2.1-6.el7
>
> Running on Oracle Cloud.
>
> VMware is on another cloud provider and they're connected via VPN.
>
> The logs looks essentially the same as the one from https://bugzilla.redhat.com
> /show_bug.cgi?id=1846238
>
> I can't really upgrade from 7.9 to 8.3, so is this expected behavior? Are there
> some potential workarounds?
It's a bug in qemu which you will need to patch in order to do the
import using this method.
Other methods don't suffer from the bug because they don't use qemu in
the same way. I would recommend exploring the “-i ova” method.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
2 years, 11 months
virt-builder fails in fedora 34 container:
by Nir Soffer
I'm running this inside a fedora 34 container in ovirt-imageio CI:
'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write',
'/etc/cirros-init/config:DATASOURCE_LIST="nocloud"',
'--root-password', 'password:', '-o',
'/var/tmp/imageio-images/cirros-0.3.5'
The command fails in:
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd
'/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd'
supermin: exception:
Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such
file or directory")
libguestfs: trace: launch = -1 (error)
virt-builder: error: libguestfs error: /usr/bin/supermin exited with error
status 1, see debug messages above
For full error log see:
https://travis-ci.com/github/nirs/ovirt-imageio/jobs/513305268
The container file is:
https://github.com/oVirt/ovirt-imageio/blob/master/docker/Dockerfile.fedo...
The same code works with centos stream 8 container and fedora 32, 33 containers.
Based on this error:
supermin: exception:
Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such
file or directory")
I think I'm missing some package removed from the fedora 34 container,
but I don't have
any idea which one.
I hope someone has more insight on this.
Nir
2 years, 11 months
[RFC PATCH 0/2] New NBD metacontext
by Eric Blake
This is my counter-proposal to Nir's request [1] to revert a 6.0
behavior change. It does not expose any new information over NBD, but
does make it easier to collect necessary information from a single
context rather than requiring the client to have to request two
contexts in parallel, then cross-correlate what may be different
extent lengths between those contexts. Furthermore, this is easy to
backport to downstream based on qemu 6.0, at which point clients could
use the existence or absence of qemu:joint-allocation as a witness of
whether it can get away with trusting base:allocation when trying to
recreate a qcow2 backing chain.
[1] https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg01796.html
Things I still want to do:
- a followup patch to libnbd to teach 'nbdinfo
--map=qemu:joint-allocation' to decode the bits
- teach 'nbdinfo --map' to read all available contexts, instead of
having to manually type each map besides base:allocation
- potential followup patches to qemu to automatically feed this
information through qemu-img map:
- add a new BDRV_BLOCK_BACKING bit for bdrv_block_status(), with
opposite semantics from BDRV_BLOCK_ALLOCATED, but where the only
thing known is that the data is not local (not how deep it is)
- teach qemu to favor qemu:joint-allocation over base:allocation
when available, and use it to drive BDRV_BLOCK_BACKING
- teach qemu-img map to recognize BDRV_BLOCK_BACKING
Eric Blake (2):
iotests: Improve and rename test 309 to nbd-qemu-allocation
nbd: Add new qemu:joint-allocation metadata context
docs/interop/nbd.txt | 31 ++++++-
docs/tools/qemu-nbd.rst | 4 +-
qapi/block-export.json | 4 +-
include/block/nbd.h | 10 ++-
nbd/server.c | 87 +++++++++++++++++--
.../{309 => tests/nbd-qemu-allocation} | 5 +-
.../nbd-qemu-allocation.out} | 13 ++-
7 files changed, 139 insertions(+), 15 deletions(-)
rename tests/qemu-iotests/{309 => tests/nbd-qemu-allocation} (95%)
rename tests/qemu-iotests/{309.out => tests/nbd-qemu-allocation.out} (79%)
--
2.31.1
2 years, 11 months
[libnbd PATCH v2 00/23] Many CI fixes
by Martin Kletzander
Lot of clean-ups for the CI. I also added support for easily configurable way
of skipping particular tests on selected distros. The only job in the pipeline
that is not green with these patches is the MacOS build and that will require
more work.
v2:
- struct sockaddr_vm fix for MacOS
- bash completion fixed - no need to skip dist on opensuse
- fixed check for qemu-storage-daemon version
- also run "brew upgrade" on MacOS
- incorporated reviews
Martin Kletzander (23):
Include sys/vsock.h if linux/vm_sockets.h is not available
bash: Generate completion files during build
fuse: Only support defined fallocate modes
interop: Do not test against broken qemu-storage-daemon
tests: Fix hexdump probing
tests: Clean up qemu-nbd detection
tests: Enhance fuse probing
tests: Fix port randomisation
tests: Require cap_sys_admin where root is required
tests: Do not use magic parameter for some nbdkit plugins
tests: Rename version to get-version
ci: Change docker driver and archive logs
ci: Remove cross-compilation targets
ci: Remove debian-10-clang build
ci: Add support for FreeBSD-13.0
ci: Fix run commands in README
ci: Some build script fixes
ci: Add support for test skipping
ci: Rename build_script.sh to just build.sh
ci: Skip some broken tests/distro combinations
ci: Only run `make check` on Linux
ci: Also perform `brew upgrade` on MacOS
ci: Update build files
configure.ac | 2 +-
lib/connect.c | 2 +
tests/Makefile.am | 18 +-
tests/functions.sh.in | 58 ++++
tests/{version.c => get-version.c} | 0
interop/interop-qemu-storage-daemon.sh | 6 +-
.gitignore | 2 +-
.gitlab-ci.yml | 247 ++----------------
bash/Makefile.am | 16 +-
ci/README.rst | 16 +-
ci/build.sh | 117 +++++++++
ci/build_script.sh | 56 ----
ci/cirrus/build.yml | 3 +-
ci/cirrus/freebsd-12.vars | 5 +-
ci/cirrus/freebsd-13.vars | 13 +
ci/cirrus/freebsd-current.vars | 5 +-
ci/cirrus/macos-11.vars | 6 +-
ci/containers/centos-8.Dockerfile | 12 +-
ci/containers/centos-stream-8.Dockerfile | 12 +-
.../debian-10-cross-aarch64.Dockerfile | 86 ------
.../debian-10-cross-armv6l.Dockerfile | 86 ------
.../debian-10-cross-armv7l.Dockerfile | 86 ------
ci/containers/debian-10-cross-i686.Dockerfile | 86 ------
ci/containers/debian-10-cross-mips.Dockerfile | 86 ------
.../debian-10-cross-mips64el.Dockerfile | 86 ------
.../debian-10-cross-mipsel.Dockerfile | 86 ------
.../debian-10-cross-ppc64le.Dockerfile | 86 ------
.../debian-10-cross-s390x.Dockerfile | 86 ------
ci/containers/debian-10.Dockerfile | 9 +-
.../debian-sid-cross-aarch64.Dockerfile | 86 ------
.../debian-sid-cross-armv6l.Dockerfile | 86 ------
.../debian-sid-cross-armv7l.Dockerfile | 86 ------
.../debian-sid-cross-i686.Dockerfile | 86 ------
.../debian-sid-cross-mips64el.Dockerfile | 86 ------
.../debian-sid-cross-mipsel.Dockerfile | 86 ------
.../debian-sid-cross-ppc64le.Dockerfile | 86 ------
.../debian-sid-cross-s390x.Dockerfile | 86 ------
ci/containers/debian-sid.Dockerfile | 9 +-
ci/containers/fedora-33.Dockerfile | 14 +-
ci/containers/fedora-34.Dockerfile | 14 +-
.../fedora-rawhide-cross-mingw32.Dockerfile | 71 -----
.../fedora-rawhide-cross-mingw64.Dockerfile | 71 -----
ci/containers/fedora-rawhide.Dockerfile | 14 +-
ci/containers/opensuse-leap-152.Dockerfile | 12 +-
ci/containers/opensuse-tumbleweed.Dockerfile | 12 +-
ci/containers/refresh | 20 --
ci/containers/ubuntu-1804.Dockerfile | 8 +-
ci/containers/ubuntu-2004.Dockerfile | 9 +-
ci/skipped_tests | 12 +
copy/copy-block-to-nbd.sh | 3 +-
copy/copy-file-to-qcow2.sh | 17 +-
copy/copy-nbd-to-block.sh | 3 +-
copy/copy-nbd-to-file.sh | 2 +-
copy/copy-nbd-to-nbd.sh | 2 +-
copy/copy-nbd-to-small-block-error.sh | 5 +-
copy/copy-tls.sh | 2 +-
fuse/operations.c | 8 +
fuse/test-errors.sh | 2 +-
fuse/test-file-mode.sh | 3 +-
fuse/test-nbdkit-command.sh | 2 +-
fuse/test-nbdkit-file-null.sh | 2 +-
fuse/test-nbdkit.sh | 3 +-
fuse/test-parallel.sh | 2 +-
fuse/test-pattern.sh | 4 +-
fuse/test-qcow2.sh | 3 +-
fuse/test-trim.sh | 2 +-
fuse/test-unix.sh | 2 +-
fuse/test-zero.sh | 2 +-
info/Makefile.am | 1 +
info/info-description-qemu.sh | 17 +-
info/info-list-json-qemu.sh | 17 +-
info/info-list-qemu.sh | 17 +-
info/info-map-qemu-allocation-depth.sh | 16 +-
info/info-map-qemu-dirty-bitmap.sh | 23 +-
74 files changed, 400 insertions(+), 2093 deletions(-)
rename tests/{version.c => get-version.c} (100%)
create mode 100755 ci/build.sh
delete mode 100755 ci/build_script.sh
create mode 100644 ci/cirrus/freebsd-13.vars
delete mode 100644 ci/containers/debian-10-cross-aarch64.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-armv6l.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-armv7l.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-i686.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-mips.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-mips64el.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-mipsel.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-ppc64le.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-s390x.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-aarch64.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-armv6l.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-armv7l.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-i686.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-mips64el.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-mipsel.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-ppc64le.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-s390x.Dockerfile
delete mode 100644 ci/containers/fedora-rawhide-cross-mingw32.Dockerfile
delete mode 100644 ci/containers/fedora-rawhide-cross-mingw64.Dockerfile
create mode 100644 ci/skipped_tests
--
2.32.0
2 years, 11 months
[libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping
by Eric Blake
Qemu has unfortunately used two distinct meanings for 'allocated' over
time, and that ambiguity has overloaded the word to make it difficult
to know which sense is meant in a given context. In one usage,
'allocated' implies that data is present and occupies space on a block
device (the opposite of sparse). In another usage, 'allocated'
implies that guest contents come from the current layer of a qcow2
backing chain (even if via a sparse zero cluster), rather than
deferring to the backing file. It doesn't help that in qemu source
code, BDRV_BLOCK_ALLOCATED refers solely to this latter usage, while
a preallocated qcow2 image refers to the former usage.
But we can help by using better terminology in nbdinfo output. With
base:allocation, either a cluster is known to be sparse (a hole), or
it (likely) occupies space (remember, NBD block status bits are
advisory - if they are set to 1, we know a given property is true; but
if they are left 0, we cannot assume the opposite of the property).
Instead of calling the latter allocated, we can call it merely "data".
And this maps a bit better to 'qemu-img map --output=json' output; a
sparse section really is "data":false, and everything that is not
known to be sparse is "data":true.
Similarly, in qemu:allocation-depth, either a cluster is known to come
from the backing chain (either locally, or from a given backing
depth), or it is completely absent from the chain (not allocated in
any of the layers). Instead of calling it "unallocated", we can call
it "absent".
Adjust the testsuite to match our improved terminology.
Thanks: Nir Soffer
---
info/info-map-base-allocation-json.sh | 4 ++--
info/info-map-base-allocation-large.sh | 8 ++++----
info/info-map-base-allocation-weird.sh | 4 ++--
info/info-map-base-allocation.sh | 6 +++---
info/info-map-qemu-allocation-depth.sh | 4 ++--
info/map.c | 4 ++--
6 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/info/info-map-base-allocation-json.sh b/info/info-map-base-allocation-json.sh
index 961eac6..b960295 100755
--- a/info/info-map-base-allocation-json.sh
+++ b/info/info-map-base-allocation-json.sh
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# nbd client library in userspace
-# Copyright (C) 2020 Red Hat Inc.
+# Copyright (C) 2020-2021 Red Hat Inc.
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -40,7 +40,7 @@ jq . < $out
test $( jq -r '.[0].offset' < $out ) -eq 0
test $( jq -r '.[0].length' < $out ) -eq 32768
test $( jq -r '.[0].type' < $out ) -eq 0
-test $( jq -r '.[0].description' < $out ) = "allocated"
+test $( jq -r '.[0].description' < $out ) = "data"
test $( jq -r '.[3].offset' < $out ) -eq 163840
test $( jq -r '.[3].length' < $out ) -eq 884736
diff --git a/info/info-map-base-allocation-large.sh b/info/info-map-base-allocation-large.sh
index e7ff904..ab10dc7 100755
--- a/info/info-map-base-allocation-large.sh
+++ b/info/info-map-base-allocation-large.sh
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# nbd client library in userspace
-# Copyright (C) 2020 Red Hat Inc.
+# Copyright (C) 2020-2021 Red Hat Inc.
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -36,11 +36,11 @@ nbdkit -U - data data='1 @131072 2 @6442450944 3' size=8G \
cat $out
-if [ "$(tr -s ' ' < $out)" != " 0 32768 0 allocated
+if [ "$(tr -s ' ' < $out)" != " 0 32768 0 data
32768 98304 3 hole,zero
- 131072 32768 0 allocated
+ 131072 32768 0 data
163840 6442287104 3 hole,zero
-6442450944 32768 0 allocated
+6442450944 32768 0 data
6442483712 2147450880 3 hole,zero" ]; then
echo "$0: unexpected output from nbdinfo --map"
exit 1
diff --git a/info/info-map-base-allocation-weird.sh b/info/info-map-base-allocation-weird.sh
index 20ddec6..0b0343a 100755
--- a/info/info-map-base-allocation-weird.sh
+++ b/info/info-map-base-allocation-weird.sh
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# nbd client library in userspace
-# Copyright (C) 2020 Red Hat Inc.
+# Copyright (C) 2020-2021 Red Hat Inc.
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -45,7 +45,7 @@ EOF
cat $out
if [ "$(tr -s ' ' < $out)" != " 0 16 1 hole
- 16 4 0 allocated
+ 16 4 0 data
20 12 1 hole" ]; then
echo "$0: unexpected output from nbdinfo --map"
exit 1
diff --git a/info/info-map-base-allocation.sh b/info/info-map-base-allocation.sh
index 9b85c97..76c2024 100755
--- a/info/info-map-base-allocation.sh
+++ b/info/info-map-base-allocation.sh
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# nbd client library in userspace
-# Copyright (C) 2020 Red Hat Inc.
+# Copyright (C) 2020-2021 Red Hat Inc.
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -36,9 +36,9 @@ nbdkit -U - data data='1 @131072 2' size=1M \
cat $out
-if [ "$(tr -s ' ' < $out)" != " 0 32768 0 allocated
+if [ "$(tr -s ' ' < $out)" != " 0 32768 0 data
32768 98304 3 hole,zero
- 131072 32768 0 allocated
+ 131072 32768 0 data
163840 884736 3 hole,zero" ]; then
echo "$0: unexpected output from nbdinfo --map"
exit 1
diff --git a/info/info-map-qemu-allocation-depth.sh b/info/info-map-qemu-allocation-depth.sh
index a5dab0b..9aeffd1 100755
--- a/info/info-map-qemu-allocation-depth.sh
+++ b/info/info-map-qemu-allocation-depth.sh
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# nbd client library in userspace
-# Copyright (C) 2020 Red Hat Inc.
+# Copyright (C) 2020-2021 Red Hat Inc.
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -84,7 +84,7 @@ if [ "$(tr -s ' ' < $out)" != "\
3145728 2097152 3 backing depth 3
5242880 1048576 1 local
6291456 1048576 2 backing depth 2
- 7340032 1048576 0 unallocated" ]; then
+ 7340032 1048576 0 absent" ]; then
echo "$0: unexpected output from nbdinfo --map"
exit 1
fi
diff --git a/info/map.c b/info/map.c
index ae6d4fe..82c9507 100644
--- a/info/map.c
+++ b/info/map.c
@@ -202,7 +202,7 @@ extent_description (const char *metacontext, uint32_t type)
if (strcmp (metacontext, "base:allocation") == 0) {
switch (type) {
- case 0: return strdup ("allocated");
+ case 0: return strdup ("data");
case 1: return strdup ("hole");
case 2: return strdup ("zero");
case 3: return strdup ("hole,zero");
@@ -216,7 +216,7 @@ extent_description (const char *metacontext, uint32_t type)
}
else if (strcmp (metacontext, "qemu:allocation-depth") == 0) {
switch (type) {
- case 0: return strdup ("unallocated");
+ case 0: return strdup ("absent");
case 1: return strdup ("local");
default:
if (asprintf (&ret, "backing depth %u", type) == -1) {
--
2.31.1
2 years, 11 months
[libnbd PATCH 00/20] Many CI fixes
by Martin Kletzander
Lot of clean-ups for the CI. I also added support for easily configurable way
of skipping particular tests on selected distros. However, I only skipped those
that do not have and easy way of detecting a broken dependency and such like.
What is still missing is:
- All the CentOS builds have issues with three tests that seem like a protocol
violation, but can very well be a bug in some test dependency. Since this is
the closest to RHEL we can get, I think it's worth investigating.
- FreeBSDs do not even build and it looks like some libtool wrapper script is
fed into a C preprocessor, which fails terribly due to the script comments
(but should not be fed there at all, of course.
- MacOS has issues, although that one is marked to allow failures and the issue
does not seem to be on our side anyway. I won't look into it for a while and
see if it sorts itself out.
Martin Kletzander (20):
fuse: Only support defined fallocate modes
interop: Do not test against broken qemu-storage-daemon
nbdsh: Check for python-exec2c installations
tests: Fix hexdump probing
tests: Clean up qemu-nbd detection in
tests: Enhance fuse probing
tests: Fix port randomisation
tests: Require cap_sys_admin where root is required
tests: Do not use magic parameter for some nbdkit plugins
ci: Change docker driver and archive logs
ci: Remove cross-compilation targets
ci: Remove debian-10-clang build
ci: Update build files
ci: Add support for FreeBSD-13.0
ci: Fix run commands in README
ci: Some build script fixes
ci: Add support for test skipping
ci: Rename build_script.sh to just build.sh
ci: Skip some broken tests/distro combinations
ci: Disable `make dist` on openSUSE
tests/Makefile.am | 6 +-
tests/functions.sh.in | 58 ++++
interop/interop-qemu-storage-daemon.sh | 5 +-
.gitlab-ci.yml | 247 ++----------------
ci/README.rst | 16 +-
ci/build.sh | 112 ++++++++
ci/build_script.sh | 56 ----
ci/cirrus/build.yml | 2 +-
ci/cirrus/freebsd-12.vars | 5 +-
ci/cirrus/freebsd-13.vars | 13 +
ci/cirrus/freebsd-current.vars | 5 +-
ci/cirrus/macos-11.vars | 6 +-
ci/containers/centos-8.Dockerfile | 12 +-
ci/containers/centos-stream-8.Dockerfile | 12 +-
.../debian-10-cross-aarch64.Dockerfile | 86 ------
.../debian-10-cross-armv6l.Dockerfile | 86 ------
.../debian-10-cross-armv7l.Dockerfile | 86 ------
ci/containers/debian-10-cross-i686.Dockerfile | 86 ------
ci/containers/debian-10-cross-mips.Dockerfile | 86 ------
.../debian-10-cross-mips64el.Dockerfile | 86 ------
.../debian-10-cross-mipsel.Dockerfile | 86 ------
.../debian-10-cross-ppc64le.Dockerfile | 86 ------
.../debian-10-cross-s390x.Dockerfile | 86 ------
ci/containers/debian-10.Dockerfile | 9 +-
.../debian-sid-cross-aarch64.Dockerfile | 86 ------
.../debian-sid-cross-armv6l.Dockerfile | 86 ------
.../debian-sid-cross-armv7l.Dockerfile | 86 ------
.../debian-sid-cross-i686.Dockerfile | 86 ------
.../debian-sid-cross-mips64el.Dockerfile | 86 ------
.../debian-sid-cross-mipsel.Dockerfile | 86 ------
.../debian-sid-cross-ppc64le.Dockerfile | 86 ------
.../debian-sid-cross-s390x.Dockerfile | 86 ------
ci/containers/debian-sid.Dockerfile | 9 +-
ci/containers/fedora-33.Dockerfile | 14 +-
ci/containers/fedora-34.Dockerfile | 14 +-
.../fedora-rawhide-cross-mingw32.Dockerfile | 71 -----
.../fedora-rawhide-cross-mingw64.Dockerfile | 71 -----
ci/containers/fedora-rawhide.Dockerfile | 14 +-
ci/containers/opensuse-leap-152.Dockerfile | 12 +-
ci/containers/opensuse-tumbleweed.Dockerfile | 12 +-
ci/containers/refresh | 20 --
ci/containers/ubuntu-1804.Dockerfile | 8 +-
ci/containers/ubuntu-2004.Dockerfile | 9 +-
ci/skipped_tests | 9 +
copy/copy-block-to-nbd.sh | 3 +-
copy/copy-file-to-qcow2.sh | 16 +-
copy/copy-nbd-to-block.sh | 3 +-
copy/copy-nbd-to-file.sh | 2 +-
copy/copy-nbd-to-nbd.sh | 2 +-
copy/copy-nbd-to-small-block-error.sh | 5 +-
copy/copy-tls.sh | 2 +-
fuse/operations.c | 8 +
fuse/test-errors.sh | 2 +-
fuse/test-file-mode.sh | 3 +-
fuse/test-nbdkit-command.sh | 2 +-
fuse/test-nbdkit-file-null.sh | 2 +-
fuse/test-nbdkit.sh | 3 +-
fuse/test-parallel.sh | 2 +-
fuse/test-pattern.sh | 4 +-
fuse/test-qcow2.sh | 3 +-
fuse/test-trim.sh | 2 +-
fuse/test-unix.sh | 2 +-
fuse/test-zero.sh | 2 +-
info/Makefile.am | 1 +
info/info-description-qemu.sh | 17 +-
info/info-list-json-qemu.sh | 17 +-
info/info-list-qemu.sh | 17 +-
info/info-map-qemu-allocation-depth.sh | 16 +-
info/info-map-qemu-dirty-bitmap.sh | 23 +-
sh/nbdsh.in | 3 +-
70 files changed, 377 insertions(+), 2074 deletions(-)
create mode 100755 ci/build.sh
delete mode 100755 ci/build_script.sh
create mode 100644 ci/cirrus/freebsd-13.vars
delete mode 100644 ci/containers/debian-10-cross-aarch64.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-armv6l.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-armv7l.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-i686.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-mips.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-mips64el.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-mipsel.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-ppc64le.Dockerfile
delete mode 100644 ci/containers/debian-10-cross-s390x.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-aarch64.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-armv6l.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-armv7l.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-i686.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-mips64el.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-mipsel.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-ppc64le.Dockerfile
delete mode 100644 ci/containers/debian-sid-cross-s390x.Dockerfile
delete mode 100644 ci/containers/fedora-rawhide-cross-mingw32.Dockerfile
delete mode 100644 ci/containers/fedora-rawhide-cross-mingw64.Dockerfile
create mode 100644 ci/skipped_tests
--
2.31.1
2 years, 11 months
GSoC: Next step nbdcopy (file-ops)
by Abhay Raj Singh
I have been analyzing the code base and I think I should start with nbdcopy
file-ops as it's easier to work on. I will try different approaches based
on the features provided by liburing.
I might need to modify the interfaces `asynch_{read, write}` which is
easier in the case of file operations.
I will update this thread regarding progress and question. I plan to add
iouring_prep_read based functionality till Sunday.
Regards,
Abhay
2 years, 11 months
ANNOUNCE: libnbd 1.8 and nbdkit 1.26 - high performance network block device client and server
by Richard W.M. Jones
I'm pleased to announce the release of nbdkit 1.26, a high performance
plugin-based Network Block Device (NBD) server, and libnbd 1.8, a high
performance client library.
https://en.wikipedia.org/wiki/Network_block_device
You can get these from:
tarballs: https://download.libguestfs.org/nbdkit/
https://download.libguestfs.org/libnbd/
git: https://gitlab.com/nbdkit
Key features of nbdkit:
* Multithreaded NBD server written in C with good performance.
* Minimal dependencies for the basic server.
* Liberal license (BSD) allows nbdkit to be linked to proprietary
libraries or included in proprietary code.
* Well-documented, simple plugin API with a stable ABI guarantee.
Lets you export “unconventional” block devices easily.
* You can write plugins in C, Go, Lua, Perl, Python, OCaml, Ruby,
Rust, shell script or Tcl.
* Filters can be stacked in front of plugins to transform the output.
Key features of libnbd:
* Synchronous API for ease of use.
* Asynchronous API for writing non-blocking, multithreaded clients.
You can mix both APIs freely.
* High performance.
* Minimal dependencies for the basic library.
* Well-documented, stable API.
* Bindings in several programming languages.
* Shell (nbdsh) for command line and scripting.
* Copying tool (nbdcopy) for high performance copying and streaming.
* Query tool (nbdinfo) to query NBD servers.
* FUSE support (nbdfuse) to mount NBD in the local filesystem.
Release notes for nbdkit 1.26
nbdkit is now hosted on gitlab:
https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html
Security
There were no security issues found. All past security issues and
information about how to report new ones can be found in
nbdkit-security(1).
Plugins
nbdkit-gzip-plugin and nbdkit-tar-plugin have been removed. They were
deprecated in nbdkit 1.22. The functionality has been replaced by
nbdkit-gzip-filter(1) and nbdkit-tar-filter(1) respectively.
nbdkit-streaming-plugin(1) has been deprecated and will be removed in
nbdkit 1.30. You can use nbdcopy(1) instead.
nbdkit-curl-plugin(1) adds these features: "followlocation=false" to
disable following redirects. "cookiefile" and "cookiejar" to control
saving cookies.
nbdkit-ssh-plugin(1) now advertises multi-conn if the server is
OpenSSH, improving performance (Eric Blake).
nbdkit-ondemand-plugin(1) has a new "wait=true" flag allowing you to
choose between clients waiting for an export to become free or being
immediately rejected.
nbdkit-sparse-random-plugin(1) now does not generate fully random data
within each block. Using perf we observed that the plugin was spending
34% of the total time during benchmarks just doing this. You can
enable the old behaviour using "random-content=true".
nbdkit-file-plugin(1) now checks if the file is read-only and exports a
read-only NBD connection in this case (Eric Blake).
nbdkit-vddk-plugin(1) has been tested with VDDK 7.0.2.
Filters
There is a new API for use by filters to manage "contexts". Instead of
the previous 1-1 relationship between filter connections and plugin
connections, filters can now open more than one context into the
backend from a single connection, or share contexts between connections
(Eric Blake).
New nbdkit-multi-conn-filter(1) which allows you to add multi-conn
support to plugins which don't support it (although the emulation is
expensive), or change whether multi-conn is advertised to clients (Eric
Blake).
nbdkit-cow-filter(1) now ignores flush/FUA requests (which are not
useful for a temporary overlay). The filter now implements extents and
efficient trimming. Fine-grained locking is used around shared
structures, greatly improving parallel performance. Added support for
multi-conn.
nbdkit-cache-filter(1) advertises multi-conn when possible (Eric
Blake).
nbdkit-cache-filter and nbdkit-cow-filter can now handle arbitrary
sized disks, instead of rounding the disk size down.
nbdkit-blocksize-filter(1) supports parallel requests (Eric Blake).
nbdkit-checkwrite-filter(1) supports multi-conn (Eric Blake).
nbdkit-ext2-filter(1) no longer supports multi-conn. The filter is
single-threaded so it does not matter, but it was not safe. Fix trim,
zero and flush support (Eric Blake).
nbdkit-readahead-filter(1) fixes multi-conn support (Eric Blake).
nbdkit-gzip-filter(1) and nbdkit-xz-filter(1) now both support multi-
conn (Eric Blake).
Language bindings
nbdkit-python-plugin(3) supports the following new features:
"after_fork" and "preconnect" methods. "nbdkit.shutdown()" binding.
nbdkit-rust-plugin(3) updated for newer Rust compiler versions (Alan
Somers).
nbdkit-golang-plugin(3) now uses and requires golang modules. Golang ≥
1.13 must be used.
API
New ".cleanup" callback for plugins and filters. This is called
shortly before ".unload", but unlike ".unload" it is guaranteed that no
plugin or filter code has been dlclosed (Eric Blake).
New "nbdkit_extents_full" is a helper function for filters which can be
used to read all extents from a plugin covering a region.
Tests
Old plugins that we use for testing were rebuilt so they no longer
depend on libselinux (thanks Thomas Weißschuh).
nbdkit-ssh-plugin(1) can still be built even if ssh-keygen is
unavailable for testing it (thanks Thomas Weißschuh).
Improve tests/test-captive.sh to avoid crashes during testing.
Test coverage on FreeBSD has been improved, fewer tests should skip
unnecessarily.
Build
nbdkit and plugins can now be built with "-fvisibility=hidden", which
allows the compiler to make some optimizations.
Add a dependency so nbdkit is relinked if the linker script
(nbdkit.syms) changes (Eric Blake).
xorriso is now preferred over genisoimage or mkisofs (see
https://wiki.debian.org/genisoimage).
It is now possible to compile nbdkit with clang with warnings enabled.
Several warnings have been suppressed or fixed.
"MALLOC_CHECK_" and "MALLOC_PERTURB_" are only enabled when running the
tests, not when running nbdkit from the build directory. This makes it
easier to do benchmarks.
"nbdkit_peer_*" implemented on OpenBSD.
Internals
When fuzzing nbdkit we now recommend using the "-t 1" flag (which
disables threads). This improves the fuzz-stability of nbdkit.
AUTHORS
Authors of nbdkit 1.26:
• Alan Somers
• Eric Blake
• Michael Ablassmeier
• Richard W.M. Jones
• Timm Bäder
Release notes for libnbd 1.8
libnbd is now hosted on gitlab:
https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html
Security
If you find a security issue, please read SECURITY in the source
(online here: https://gitlab.com/nbdkit/libnbd/blob/master/SECURITY).
To find out about previous security issues in libnbd, see
libnbd-security(3).
There was one security bug found in libnbd during this release cycle.
CVE-2021-20286 denial of service when using nbd_set_opt_mode(3)
See the full announcement here:
https://listman.redhat.com/archives/libguestfs/2021-March/msg00092.html
(Found and fixed by Eric Blake).
New APIs
nbd_get_private_data(3)
nbd_set_private_data(3)
These calls allow you to store either an unsigned integer or a
pointer in the handle for the application to use for its own
purposes.
nbd_get_uri(3)
This call constructs an NBD URI which can be used to connect back
to the same server (using nbd_connect_uri(3) or from other tools
that support the NBD URI standard).
Enhancements to existing APIs
Numeric IPv6 URIs (like "nbd://[::1]/") are now supported.
libnbd gives a better error message if the server backlog overflows
(thanks Xin Long, Lukas Doktor, Eric Blake, Martin Kletzander).
New features
Continuous integration (CI) tests now run on every commit and merge
request to the upstream repository (Martin Kletzander).
Tools
A great deal of work has been done to improve the performance of
nbdcopy(1) (Nir Soffer, Eric Blake).
On Linux, nbdcopy now uses readahead when reading from files, and takes
steps to preserve the host page cache.
nbdcopy new --request-size option to allow controlling the size of
requests made to NBD servers. Adjusting this can affect performance
(Nir Soffer).
nbdcopy now supports "null:" pseudo-target, where it discards the
output. This is useful for benchmarking.
nbdcopy new -v flag to enable libnbd and other debugging.
nbdinfo(1) --list (which lists all exports of a server) no longer exits
early if one of the exports is inaccessible or there is a recoverable
error (Eric Blake).
nbdinfo now prints the URI of each export in the output making it
easier to connect back to each export.
nbdfuse(1) is now multithreaded and supports issuing parallel NBD
commands on a single connection as well as multi-conn, and other
enhancements have been made. This resulted in around 50% performance
improvement when tested with fio.
nbdfuse now supports nbdcopy-style "[ CMD ]", eg:
nbdfuse dir/ramdisk [ nbdkit memory 1G ]
nbdfuse now supports trimming. Also it supports fast zeroing, but this
requires a forthcoming kernel patch before it will work.
nbdfuse sets the virtual file permissions appropriately if the NBD
export is read-only.
nbdfuse new -v flag to enable libnbd and other debugging.
Tests
Added a test of interoperability with qemu-storage-daemon(1). Note
that qemu-storage-daemon was broken in qemu 6.0.0 so this test can fail
unless you update to a fixed version (thanks Stefan Hajnoczi, Daniel
Berrangé).
Added a test of nbdcopy with TLS which was previously untested.
Python code style tests are now opt-in using
./configure --enable-python-code-style
More tests should now run on FreeBSD and not be skipped.
nbdfuse tests now ensure that errors from NBD are transmitted through
FUSE back to the local client.
"MALLOC_CHECK_" and "MALLOC_PERTURB_" are only enabled when running the
tests, not when running libnbd programs from the build directory. This
makes it easier to do benchmarks.
Other improvements and bug fixes
The minimum C compiler is now ISO C99 (previously ANSI C90).
FUSE 3 (instead of FUSE 2) is now required to build nbdfuse.
The library tries harder not to leak thread-local storage memory on
dlclose or exit. (Because of the design of Linux it is not always
possible to avoid this, especially in multithreaded programs.)
Fix the progress bar in nbdcopy(1).
UTF-8 export names are now processed and displayed properly by
nbdinfo(1).
The --help output of nbdcopy, nbdfuse and nbdinfo now displays a brief
summary of all options and has some examples.
The generator now only requires ocamlc, not ocamlfind (Abhay Raj
Singh).
Golang bindings now use and require golang modules.
Documentation
Document using libnbd with the Meson build system.
Added Python examples.
Add example of how to integrate libnbd with libev (Nir Soffer).
AUTHORS
Eric Blake
Richard W.M. Jones
Martin Kletzander
Nir Soffer
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
2 years, 11 months
GSoC: Adding liburing dependency to libnbd
by Abhay Raj Singh
I think I can reuse the code below (libev part in configure.ac) and modify
it for liburing, there is the libxml2 part that provides an option to
disable libxml2 parts which I can reuse to provide similar behavior which
is simpler the without headers and lib checks, I feel libev metho
PKG_CHECK_MODULES([LIBEV], [libev], [
AC_SUBST([LIBEV_CFLAGS])
AC_SUBST([LIBEV_LIBS])
],[
dnl no pkg-config for libev, searching manually:
AC_CHECK_HEADERS([ev.h], [
AC_CHECK_LIB([ev], [ev_time], [
AC_SUBST([LIBEV_LIBS], ["-lev"])
],
[
AC_MSG_WARN([libev not found, some examples will not be
compiled])
])
],[
AC_MSG_WARN([ev.h not found, some examples will not be compiled])
])
])
AS_IF([test "x$LIBEV_LIBS" != "x"], [
old_CFLAGS="$CFLAGS"
CFLAGS="$CFLAGS $LIBEV_CFLAGS -Werror=strict-aliasing -O2"
AC_MSG_CHECKING([if the compiler is new enough for good aliasing rules])
AC_COMPILE_IFELSE([
AC_LANG_PROGRAM([
#include <ev.h>
static void cb (struct ev_loop *l, ev_timer *t, int e) { }
static ev_timer timer;
], [
ev_timer_init (&timer, cb, 0, .1);
])
], [
AC_MSG_RESULT([yes])
], [
AC_MSG_RESULT([no])
LIBEV_CFLAGS="$LIBEV_CFLAGS -Wno-strict-aliasing"
])
CFLAGS="$old_CFLAGS"
])
AM_CONDITIONAL([HAVE_LIBEV], [test "x$LIBEV_LIBS" != "x"])
2 years, 11 months