CI for libnbd is now active
by Martin Kletzander
I have (finally) pushed the CI enablement patches upstream and can see
the pipeline running already here:
https://gitlab.com/nbdkit/libnbd/-/pipelines
Clicking on the status opens it up, amongst other clickable objects are
the stage icons.
I managed to fix the easier issues, but there are still more which I do
not know how to solve (and for some of them I tried more than I
should've).
There's a small write-up, partially copied from libvirt, in
ci/README.rst, feel free to change the format. The patch on top of
libvirt-ci that was used for generating the initial CI assets in ci/ was
submitted as a merge request here:
https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/120
and that is were dependency changes need to be tracked.
I'm glad to answer any questions.
Have a nice day,
Martin
3 years, 2 months
question about support for almalinux or centOS stream images?
by Barrette, Paul
Hi All,
I have been using libguestfs to generate centOS and fedora based images. Are there plans to support almalinux or CentOS stream? Or if there is a how to or doc that shows how to build a custom OS please let me know.
thanks
Pb
----------------------------------------------------------------------
This e-mail and any files transmitted with it are proprietary and intended solely for the use of the individual or entity to whom they are addressed. If you have reason to believe that you have received this e-mail in error, please notify the sender and destroy this e-mail and any attached files. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of the Curtiss-Wright Corporation or any of its subsidiaries. Documents attached hereto may contain technology subject to government export regulations. Recipient is solely responsible for ensuring that any re-export, transfer or disclosure of this information is in accordance with applicable government export regulations. The recipient should check this e-mail and any attachments for the presence of viruses. Curtiss-Wright Corporation and its subsidiaries accept no liability for any damage caused by any virus transmitted by this e-mail.
3 years, 2 months
[libnbd PATCH 0/5] Introduce automated testing using GitLab CI
by Martin Kletzander
This is reworked version of
https://listman.redhat.com/archives/libguestfs/2021-March/msg00134.html
with more packages installed so that more code paths are tested, also some fixes
in the code (mostly requirements).
There are still some things that fail.
FreeBSD
=======
The build fails because there is no fallocate() there.
Many platforms (those that have nbd-server in repositories)
===========================================================
interop/interop-nbd-server fails with:
Error: inetd mode requires syslog
Exiting.
which makes sense as it is run with port 0, but how to check for whether it is
compiled with or without syslog (so that we know to skip the test) or whether it
is something else that is missing (capabilities in a container) is beyond me.
interop/list-exports-nbd-server gets stuck and in the CI times out the whole
test suite. Locally I need to manually exec into the container:
podman exec -itu bash
and I see the nbd-server is just not doing anything. After killing it the test
ends and fails. I did not debug nbd-server for this, maybe it is again some
requisite that is missing and we are not checking for it (capability, nbd module
access, etc.), I do not know.
There are also some skips that I did not go into any more after trying to figure
out every "... no" in config.log:
Both interop-qemu-nbd-tls-certs and interop-qemu-nbd-tls-psk get skipped because
they are missing the keys, but I presume those are supposed to be generated.
I think that all these can be figured out after the final patch is merged so
that the CI runs truly automatically. I am posting it for review though as I am
not sure the solutions in first four patches are the preferred ones.
Martin Kletzander (5):
Fix test requirements
Change golang/run_tests.sh shebang to bash
libev: Check for old compiler
golang: Check for module support
Introduction of automated testing
configure.ac | 26 +-
Makefile.am | 1 +
tests/functions.sh.in | 21 +
interop/dirty-bitmap.sh | 4 +-
.gitlab-ci.yml | 479 ++++++++++++++++++
ci/build_script.sh | 43 ++
ci/cirrus/build.yml | 22 +
ci/cirrus/freebsd-12.vars | 14 +
ci/cirrus/freebsd-current.vars | 14 +
ci/cirrus/macos-11.vars | 14 +
ci/cirrus/refresh | 22 +
ci/containers/README.rst | 14 +
ci/containers/centos-8.Dockerfile | 61 +++
ci/containers/centos-stream-8.Dockerfile | 61 +++
.../debian-10-cross-aarch64.Dockerfile | 86 ++++
.../debian-10-cross-armv6l.Dockerfile | 86 ++++
.../debian-10-cross-armv7l.Dockerfile | 86 ++++
ci/containers/debian-10-cross-i686.Dockerfile | 86 ++++
ci/containers/debian-10-cross-mips.Dockerfile | 86 ++++
.../debian-10-cross-mips64el.Dockerfile | 86 ++++
.../debian-10-cross-mipsel.Dockerfile | 86 ++++
.../debian-10-cross-ppc64le.Dockerfile | 86 ++++
.../debian-10-cross-s390x.Dockerfile | 86 ++++
ci/containers/debian-10.Dockerfile | 64 +++
.../debian-sid-cross-aarch64.Dockerfile | 86 ++++
.../debian-sid-cross-armv6l.Dockerfile | 86 ++++
.../debian-sid-cross-armv7l.Dockerfile | 86 ++++
.../debian-sid-cross-i686.Dockerfile | 86 ++++
.../debian-sid-cross-mips64el.Dockerfile | 86 ++++
.../debian-sid-cross-mipsel.Dockerfile | 86 ++++
.../debian-sid-cross-ppc64le.Dockerfile | 86 ++++
.../debian-sid-cross-s390x.Dockerfile | 86 ++++
ci/containers/debian-sid.Dockerfile | 64 +++
ci/containers/fedora-33.Dockerfile | 68 +++
ci/containers/fedora-34.Dockerfile | 68 +++
.../fedora-rawhide-cross-mingw32.Dockerfile | 71 +++
.../fedora-rawhide-cross-mingw64.Dockerfile | 71 +++
ci/containers/fedora-rawhide.Dockerfile | 69 +++
ci/containers/opensuse-leap-152.Dockerfile | 56 ++
ci/containers/opensuse-tumbleweed.Dockerfile | 57 +++
ci/containers/refresh | 42 ++
ci/containers/ubuntu-1804.Dockerfile | 62 +++
ci/containers/ubuntu-2004.Dockerfile | 64 +++
copy/copy-block-to-nbd.sh | 1 +
copy/copy-nbd-to-block.sh | 1 +
copy/copy-nbd-to-nbd.sh | 1 +
copy/copy-nbd-to-small-block-error.sh | 1 +
copy/copy-tls.sh | 1 +
fuse/test-file-mode.sh | 1 +
fuse/test-nbdkit.sh | 1 +
fuse/test-qcow2.sh | 1 +
golang/run-tests.sh | 2 +-
info/info-map-qemu-dirty-bitmap.sh | 4 +-
53 files changed, 3022 insertions(+), 6 deletions(-)
create mode 100644 .gitlab-ci.yml
create mode 100755 ci/build_script.sh
create mode 100644 ci/cirrus/build.yml
create mode 100644 ci/cirrus/freebsd-12.vars
create mode 100644 ci/cirrus/freebsd-current.vars
create mode 100644 ci/cirrus/macos-11.vars
create mode 100755 ci/cirrus/refresh
create mode 100644 ci/containers/README.rst
create mode 100644 ci/containers/centos-8.Dockerfile
create mode 100644 ci/containers/centos-stream-8.Dockerfile
create mode 100644 ci/containers/debian-10-cross-aarch64.Dockerfile
create mode 100644 ci/containers/debian-10-cross-armv6l.Dockerfile
create mode 100644 ci/containers/debian-10-cross-armv7l.Dockerfile
create mode 100644 ci/containers/debian-10-cross-i686.Dockerfile
create mode 100644 ci/containers/debian-10-cross-mips.Dockerfile
create mode 100644 ci/containers/debian-10-cross-mips64el.Dockerfile
create mode 100644 ci/containers/debian-10-cross-mipsel.Dockerfile
create mode 100644 ci/containers/debian-10-cross-ppc64le.Dockerfile
create mode 100644 ci/containers/debian-10-cross-s390x.Dockerfile
create mode 100644 ci/containers/debian-10.Dockerfile
create mode 100644 ci/containers/debian-sid-cross-aarch64.Dockerfile
create mode 100644 ci/containers/debian-sid-cross-armv6l.Dockerfile
create mode 100644 ci/containers/debian-sid-cross-armv7l.Dockerfile
create mode 100644 ci/containers/debian-sid-cross-i686.Dockerfile
create mode 100644 ci/containers/debian-sid-cross-mips64el.Dockerfile
create mode 100644 ci/containers/debian-sid-cross-mipsel.Dockerfile
create mode 100644 ci/containers/debian-sid-cross-ppc64le.Dockerfile
create mode 100644 ci/containers/debian-sid-cross-s390x.Dockerfile
create mode 100644 ci/containers/debian-sid.Dockerfile
create mode 100644 ci/containers/fedora-33.Dockerfile
create mode 100644 ci/containers/fedora-34.Dockerfile
create mode 100644 ci/containers/fedora-rawhide-cross-mingw32.Dockerfile
create mode 100644 ci/containers/fedora-rawhide-cross-mingw64.Dockerfile
create mode 100644 ci/containers/fedora-rawhide.Dockerfile
create mode 100644 ci/containers/opensuse-leap-152.Dockerfile
create mode 100644 ci/containers/opensuse-tumbleweed.Dockerfile
create mode 100755 ci/containers/refresh
create mode 100644 ci/containers/ubuntu-1804.Dockerfile
create mode 100644 ci/containers/ubuntu-2004.Dockerfile
--
2.31.1
3 years, 2 months
[PATCH v3] fuse: Allow fallocate(FALLOC_FL_ZERO_RANGE)
by Richard W.M. Jones
v2 -> v3:
- Call truncate_pagecache_range, same as for hole punching.
- Tidy up the tests of flags in mode.
- Changed the test / example in the commit message to something
a bit simpler to understand.
- Retested it.
Rich.
3 years, 2 months
[PATCH] fuse: Allow fallocate(FALLOC_FL_ZERO_RANGE)
by Richard W.M. Jones
libnbd's nbdfuse utility would like to translate fallocate zero
requests into NBD_CMD_WRITE_ZEROES. Currently the fuse module filters
these out, returning -EOPNOTSUPP. This commit treats these almost the
same way as FALLOC_FL_PUNCH_HOLE except not calling
truncate_pagecache_range.
A way to test this is with the following script:
--------------------
set -e
set -x
export output=$PWD/output
rm -f test.img $output
nbdkit sh - <<'EOF'
case "$1" in
get_size) echo 1M ;;
can_write|can_trim|can_zero|can_fast_zero) ;;
pread) echo "$@" >>$output; dd if=/dev/zero count=$3 iflag=count_bytes ;;
pwrite) echo "$@" >>$output; cat >/dev/null ;;
trim|zero) echo "$@" >>$output ;;
*) exit 2 ;;
esac
EOF
touch test.img
nbdfuse --version
nbdfuse test.img nbd://localhost & sleep 2
ls -lh test.img
dd if=test.img of=/dev/null bs=512 skip=1024 count=1
dd if=/dev/zero of=test.img bs=512 skip=2048 count=1
fallocate -p -l 512 -o 4096 test.img
fallocate -z -l 512 -o 8192 test.img
cat $output
fusermount3 -u test.img
killall nbdkit
rm test.img $output
--------------------
which will print:
pread 4096 524288 # number depends on readahea
pwrite 512 0
trim 512 4096
zero 512 8192 may_trim
with the last line indicating that the FALLOC_FL_ZERO_RANGE request
was successfully passed through by the kernel module to nbdfuse,
translated to NBD_CMD_WRITE_ZEROES and sent through to the server.
Signed-off-by: Richard W.M. Jones <rjones(a)redhat.com>
---
fs/fuse/file.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 09ef2a4d25ed..22e8e88c78d4 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -2907,11 +2907,13 @@ static long fuse_file_fallocate(struct file *file, int mode, loff_t offset,
};
int err;
bool lock_inode = !(mode & FALLOC_FL_KEEP_SIZE) ||
- (mode & FALLOC_FL_PUNCH_HOLE);
+ (mode & FALLOC_FL_PUNCH_HOLE) ||
+ (mode & FALLOC_FL_ZERO_RANGE);
bool block_faults = FUSE_IS_DAX(inode) && lock_inode;
- if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
+ if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE |
+ FALLOC_FL_ZERO_RANGE))
return -EOPNOTSUPP;
if (fm->fc->no_fallocate)
@@ -2926,7 +2928,8 @@ static long fuse_file_fallocate(struct file *file, int mode, loff_t offset,
goto out;
}
- if (mode & FALLOC_FL_PUNCH_HOLE) {
+ if ((mode & FALLOC_FL_PUNCH_HOLE) ||
+ (mode & FALLOC_FL_ZERO_RANGE)) {
loff_t endbyte = offset + length - 1;
err = fuse_writeback_range(inode, offset, endbyte);
--
2.31.1
3 years, 2 months
GNU-isms in sed expressions in libnbd tests
by Richard W.M. Jones
In info/info-text.sh and info/info-list.sh, FreeBSD sed complains:
+ sed -n '/contexts:/ { N; p; q }; $ q1' info-text.out
sed: 1: "/contexts:/ { N; p; q } ...": extra characters at the end of q command
If I understand correctly, 'q1' means exit with error code 1, and the
man page notes that this is a GNU extension.
I couldn't find any obvious/easy fix for this, so just FYI.
Would it help to set POSIXLY_CORRECT when running the tests to catch
these earlier?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
3 years, 2 months
[nbdkit PATCH 0/5] Use shared context to make ext2 filter parallel
by Eric Blake
I've been working on this one for a while, and finally got it working
well enough to post. I'm probably still going to work on adding
another unit test proving that you can open multiple NBD clients in
parallel, and that they can interleave I/O, but my immediate test was
setting up:
nbdkit -f --filter=ext2 --filter=exportname file tests/ext2.img \
ext2file=exportname exportname-list=explicit exportname=hidden
then running two simultaneous nbdsh, one visiting
nbd://localhost:10809/manifest and the other
nbd://localhost:10809/disks/disk.img. Pre-series, the second nbdsh
stalls until the first exits, post-series, both can read their
respective file out of the overall image at the same time.
Eric Blake (5):
filters: Tighter rules on .get_ready/.after_fork
threadlocal: Add way to store current context
filters: Allow nbdkit_next_context_open outside client connection
api: Add .cleanup callback
ext2: Support parallel requests and connections
docs/nbdkit-filter.pod | 65 ++++++--
docs/nbdkit-plugin.pod | 33 +++-
include/nbdkit-filter.h | 10 +-
include/nbdkit-plugin.h | 4 +-
server/internal.h | 14 +-
server/backend.c | 48 +++++-
server/filters.c | 49 +++---
server/main.c | 1 +
server/plugins.c | 32 ++--
server/protocol-handshake.c | 2 +-
server/public.c | 24 ++-
server/test-public.c | 12 +-
server/threadlocal.c | 36 ++++-
filters/exitwhen/exitwhen.c | 9 +-
filters/ext2/ext2.c | 225 ++++++++++++++++------------
filters/extentlist/extentlist.c | 5 +-
filters/log/log.c | 9 +-
filters/multi-conn/multi-conn.c | 5 +-
filters/pause/pause.c | 4 +-
filters/rate/rate.c | 5 +-
filters/retry/retry.c | 2 +-
filters/stats/stats.c | 5 +-
filters/tls-fallback/tls-fallback.c | 5 +-
tests/test-layers-filter.c | 22 ++-
tests/test-layers-plugin.c | 9 +-
tests/test-layers.c | 42 ++++--
26 files changed, 453 insertions(+), 224 deletions(-)
--
2.31.1
3 years, 2 months
FYI: nbdfuse support for trim and writing zeroes
by Richard W.M. Jones
I added support for trimming to nbdfuse over the weekend:
https://gitlab.com/nbdkit/libnbd/-/commit/800c021f39ddfa1dd40d8dcc8645463...
This intercepts Linux fallocate(2) on the file and if the mode
parameter contains FALLOC_FL_PUNCH_HOLE then it will punch a hole
using nbd_trim. This all appears to work fine.
As noted in the above commit, in theory we could also check for
mode & FALLOC_FL_ZERO_RANGE. When I use "fallocate -z ..." on a file,
I see this is the mode parameter being passed to the system call:
fallocate(3, FALLOC_FL_ZERO_RANGE, 512, 512) = 0
However it appears as if the fuse kernel module doesn't pass the flag
through to nbdfuse. (I checked the kernel code too, and that *seems*
to agree with my assessment). So for now I left support for zeroing
in there but disabled the test.
The fuse developer mailing list seems to be mostly spam. There is
this closed github issue which says to contact the kernel developers
which I have not done:
https://github.com/libfuse/libfuse/issues/395
Next step for nbdfuse is to add multithreading support which would
greatly improve performance but is rather complicated to implement.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3 years, 2 months