May 2021 - Libguestfs - libguestfs List Archives

CI for libnbd is now active
by Martin Kletzander 18 May '21

18 May '21

I have (finally) pushed the CI enablement patches upstream and can see the pipeline running already here: https://gitlab.com/nbdkit/libnbd/-/pipelines Clicking on the status opens it up, amongst other clickable objects are the stage icons. I managed to fix the easier issues, but there are still more which I do not know how to solve (and for some of them I tried more than I should've). There's a small write-up, partially copied from libvirt, in ci/README.rst, feel free to change the format. The patch on top of libvirt-ci that was used for generating the initial CI assets in ci/ was submitted as a merge request here: https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/120 and that is were dependency changes need to be tracked. I'm glad to answer any questions. Have a nice day, Martin

2 2

[PATCH libnbd 0/3] fuse: Enable multithread support
by Richard W.M. Jones 13 May '21

13 May '21

First patch pretty uncontroversial, but unexciting. Second patch, exciting, hairy and controversial!

1 4

question about support for almalinux or centOS stream images?
by Barrette, Paul 13 May '21

13 May '21

Hi All, I have been using libguestfs to generate centOS and fedora based images. Are there plans to support almalinux or CentOS stream? Or if there is a how to or doc that shows how to build a custom OS please let me know. thanks Pb ---------------------------------------------------------------------- This e-mail and any files transmitted with it are proprietary and intended solely for the use of the individual or entity to whom they are addressed. If you have reason to believe that you have received this e-mail in error, please notify the sender and destroy this e-mail and any attached files. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of the Curtiss-Wright Corporation or any of its subsidiaries. Documents attached hereto may contain technology subject to government export regulations. Recipient is solely responsible for ensuring that any re-export, transfer or disclosure of this information is in accordance with applicable government export regulations. The recipient should check this e-mail and any attachments for the presence of viruses. Curtiss-Wright Corporation and its subsidiaries accept no liability for any damage caused by any virus transmitted by this e-mail.

2 4

[libnbd PATCH 0/5] Introduce automated testing using GitLab CI
by Martin Kletzander 12 May '21

12 May '21

This is reworked version of https://listman.redhat.com/archives/libguestfs/2021-March/msg00134.html with more packages installed so that more code paths are tested, also some fixes in the code (mostly requirements). There are still some things that fail. FreeBSD ======= The build fails because there is no fallocate() there. Many platforms (those that have nbd-server in repositories) =========================================================== interop/interop-nbd-server fails with: Error: inetd mode requires syslog Exiting. which makes sense as it is run with port 0, but how to check for whether it is compiled with or without syslog (so that we know to skip the test) or whether it is something else that is missing (capabilities in a container) is beyond me. interop/list-exports-nbd-server gets stuck and in the CI times out the whole test suite. Locally I need to manually exec into the container: podman exec -itu bash and I see the nbd-server is just not doing anything. After killing it the test ends and fails. I did not debug nbd-server for this, maybe it is again some requisite that is missing and we are not checking for it (capability, nbd module access, etc.), I do not know. There are also some skips that I did not go into any more after trying to figure out every "... no" in config.log: Both interop-qemu-nbd-tls-certs and interop-qemu-nbd-tls-psk get skipped because they are missing the keys, but I presume those are supposed to be generated. I think that all these can be figured out after the final patch is merged so that the CI runs truly automatically. I am posting it for review though as I am not sure the solutions in first four patches are the preferred ones. Martin Kletzander (5): Fix test requirements Change golang/run_tests.sh shebang to bash libev: Check for old compiler golang: Check for module support Introduction of automated testing configure.ac | 26 +- Makefile.am | 1 + tests/functions.sh.in | 21 + interop/dirty-bitmap.sh | 4 +- .gitlab-ci.yml | 479 ++++++++++++++++++ ci/build_script.sh | 43 ++ ci/cirrus/build.yml | 22 + ci/cirrus/freebsd-12.vars | 14 + ci/cirrus/freebsd-current.vars | 14 + ci/cirrus/macos-11.vars | 14 + ci/cirrus/refresh | 22 + ci/containers/README.rst | 14 + ci/containers/centos-8.Dockerfile | 61 +++ ci/containers/centos-stream-8.Dockerfile | 61 +++ .../debian-10-cross-aarch64.Dockerfile | 86 ++++ .../debian-10-cross-armv6l.Dockerfile | 86 ++++ .../debian-10-cross-armv7l.Dockerfile | 86 ++++ ci/containers/debian-10-cross-i686.Dockerfile | 86 ++++ ci/containers/debian-10-cross-mips.Dockerfile | 86 ++++ .../debian-10-cross-mips64el.Dockerfile | 86 ++++ .../debian-10-cross-mipsel.Dockerfile | 86 ++++ .../debian-10-cross-ppc64le.Dockerfile | 86 ++++ .../debian-10-cross-s390x.Dockerfile | 86 ++++ ci/containers/debian-10.Dockerfile | 64 +++ .../debian-sid-cross-aarch64.Dockerfile | 86 ++++ .../debian-sid-cross-armv6l.Dockerfile | 86 ++++ .../debian-sid-cross-armv7l.Dockerfile | 86 ++++ .../debian-sid-cross-i686.Dockerfile | 86 ++++ .../debian-sid-cross-mips64el.Dockerfile | 86 ++++ .../debian-sid-cross-mipsel.Dockerfile | 86 ++++ .../debian-sid-cross-ppc64le.Dockerfile | 86 ++++ .../debian-sid-cross-s390x.Dockerfile | 86 ++++ ci/containers/debian-sid.Dockerfile | 64 +++ ci/containers/fedora-33.Dockerfile | 68 +++ ci/containers/fedora-34.Dockerfile | 68 +++ .../fedora-rawhide-cross-mingw32.Dockerfile | 71 +++ .../fedora-rawhide-cross-mingw64.Dockerfile | 71 +++ ci/containers/fedora-rawhide.Dockerfile | 69 +++ ci/containers/opensuse-leap-152.Dockerfile | 56 ++ ci/containers/opensuse-tumbleweed.Dockerfile | 57 +++ ci/containers/refresh | 42 ++ ci/containers/ubuntu-1804.Dockerfile | 62 +++ ci/containers/ubuntu-2004.Dockerfile | 64 +++ copy/copy-block-to-nbd.sh | 1 + copy/copy-nbd-to-block.sh | 1 + copy/copy-nbd-to-nbd.sh | 1 + copy/copy-nbd-to-small-block-error.sh | 1 + copy/copy-tls.sh | 1 + fuse/test-file-mode.sh | 1 + fuse/test-nbdkit.sh | 1 + fuse/test-qcow2.sh | 1 + golang/run-tests.sh | 2 +- info/info-map-qemu-dirty-bitmap.sh | 4 +- 53 files changed, 3022 insertions(+), 6 deletions(-) create mode 100644 .gitlab-ci.yml create mode 100755 ci/build_script.sh create mode 100644 ci/cirrus/build.yml create mode 100644 ci/cirrus/freebsd-12.vars create mode 100644 ci/cirrus/freebsd-current.vars create mode 100644 ci/cirrus/macos-11.vars create mode 100755 ci/cirrus/refresh create mode 100644 ci/containers/README.rst create mode 100644 ci/containers/centos-8.Dockerfile create mode 100644 ci/containers/centos-stream-8.Dockerfile create mode 100644 ci/containers/debian-10-cross-aarch64.Dockerfile create mode 100644 ci/containers/debian-10-cross-armv6l.Dockerfile create mode 100644 ci/containers/debian-10-cross-armv7l.Dockerfile create mode 100644 ci/containers/debian-10-cross-i686.Dockerfile create mode 100644 ci/containers/debian-10-cross-mips.Dockerfile create mode 100644 ci/containers/debian-10-cross-mips64el.Dockerfile create mode 100644 ci/containers/debian-10-cross-mipsel.Dockerfile create mode 100644 ci/containers/debian-10-cross-ppc64le.Dockerfile create mode 100644 ci/containers/debian-10-cross-s390x.Dockerfile create mode 100644 ci/containers/debian-10.Dockerfile create mode 100644 ci/containers/debian-sid-cross-aarch64.Dockerfile create mode 100644 ci/containers/debian-sid-cross-armv6l.Dockerfile create mode 100644 ci/containers/debian-sid-cross-armv7l.Dockerfile create mode 100644 ci/containers/debian-sid-cross-i686.Dockerfile create mode 100644 ci/containers/debian-sid-cross-mips64el.Dockerfile create mode 100644 ci/containers/debian-sid-cross-mipsel.Dockerfile create mode 100644 ci/containers/debian-sid-cross-ppc64le.Dockerfile create mode 100644 ci/containers/debian-sid-cross-s390x.Dockerfile create mode 100644 ci/containers/debian-sid.Dockerfile create mode 100644 ci/containers/fedora-33.Dockerfile create mode 100644 ci/containers/fedora-34.Dockerfile create mode 100644 ci/containers/fedora-rawhide-cross-mingw32.Dockerfile create mode 100644 ci/containers/fedora-rawhide-cross-mingw64.Dockerfile create mode 100644 ci/containers/fedora-rawhide.Dockerfile create mode 100644 ci/containers/opensuse-leap-152.Dockerfile create mode 100644 ci/containers/opensuse-tumbleweed.Dockerfile create mode 100755 ci/containers/refresh create mode 100644 ci/containers/ubuntu-1804.Dockerfile create mode 100644 ci/containers/ubuntu-2004.Dockerfile -- 2.31.1

5 32

[PATCH v3] fuse: Allow fallocate(FALLOC_FL_ZERO_RANGE)
by Richard W.M. Jones 12 May '21

12 May '21

v2 -> v3: - Call truncate_pagecache_range, same as for hole punching. - Tidy up the tests of flags in mode. - Changed the test / example in the commit message to something a bit simpler to understand. - Retested it. Rich.

1 1

[PATCH v2] fuse: Allow fallocate(FALLOC_FL_ZERO_RANGE)
by Richard W.M. Jones 12 May '21

12 May '21

Version 2 restores the #comments in the script in the git commit message. The patch itself is identical. Rich.

2 3

[PATCH] fuse: Allow fallocate(FALLOC_FL_ZERO_RANGE)
by Richard W.M. Jones 12 May '21

12 May '21

libnbd's nbdfuse utility would like to translate fallocate zero requests into NBD_CMD_WRITE_ZEROES. Currently the fuse module filters these out, returning -EOPNOTSUPP. This commit treats these almost the same way as FALLOC_FL_PUNCH_HOLE except not calling truncate_pagecache_range. A way to test this is with the following script: -------------------- set -e set -x export output=$PWD/output rm -f test.img $output nbdkit sh - <<'EOF' case "$1" in get_size) echo 1M ;; can_write|can_trim|can_zero|can_fast_zero) ;; pread) echo "$@" >>$output; dd if=/dev/zero count=$3 iflag=count_bytes ;; pwrite) echo "$@" >>$output; cat >/dev/null ;; trim|zero) echo "$@" >>$output ;; *) exit 2 ;; esac EOF touch test.img nbdfuse --version nbdfuse test.img nbd://localhost & sleep 2 ls -lh test.img dd if=test.img of=/dev/null bs=512 skip=1024 count=1 dd if=/dev/zero of=test.img bs=512 skip=2048 count=1 fallocate -p -l 512 -o 4096 test.img fallocate -z -l 512 -o 8192 test.img cat $output fusermount3 -u test.img killall nbdkit rm test.img $output -------------------- which will print: pread 4096 524288 # number depends on readahea pwrite 512 0 trim 512 4096 zero 512 8192 may_trim with the last line indicating that the FALLOC_FL_ZERO_RANGE request was successfully passed through by the kernel module to nbdfuse, translated to NBD_CMD_WRITE_ZEROES and sent through to the server. Signed-off-by: Richard W.M. Jones <rjones(a)redhat.com> --- fs/fuse/file.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 09ef2a4d25ed..22e8e88c78d4 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -2907,11 +2907,13 @@ static long fuse_file_fallocate(struct file *file, int mode, loff_t offset, }; int err; bool lock_inode = !(mode & FALLOC_FL_KEEP_SIZE) || - (mode & FALLOC_FL_PUNCH_HOLE); + (mode & FALLOC_FL_PUNCH_HOLE) || + (mode & FALLOC_FL_ZERO_RANGE); bool block_faults = FUSE_IS_DAX(inode) && lock_inode; - if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE)) + if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | + FALLOC_FL_ZERO_RANGE)) return -EOPNOTSUPP; if (fm->fc->no_fallocate) @@ -2926,7 +2928,8 @@ static long fuse_file_fallocate(struct file *file, int mode, loff_t offset, goto out; } - if (mode & FALLOC_FL_PUNCH_HOLE) { + if ((mode & FALLOC_FL_PUNCH_HOLE) || + (mode & FALLOC_FL_ZERO_RANGE)) { loff_t endbyte = offset + length - 1; err = fuse_writeback_range(inode, offset, endbyte); -- 2.31.1

1 0

GNU-isms in sed expressions in libnbd tests
by Richard W.M. Jones 11 May '21

11 May '21

In info/info-text.sh and info/info-list.sh, FreeBSD sed complains: + sed -n '/contexts:/ { N; p; q }; $ q1' info-text.out sed: 1: "/contexts:/ { N; p; q } ...": extra characters at the end of q command If I understand correctly, 'q1' means exit with error code 1, and the man page notes that this is a GNU extension. I couldn't find any obvious/easy fix for this, so just FYI. Would it help to set POSIXLY_CORRECT when running the tests to catch these earlier? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

2 2

[nbdkit PATCH 0/5] Use shared context to make ext2 filter parallel
by Eric Blake 11 May '21

11 May '21

I've been working on this one for a while, and finally got it working well enough to post. I'm probably still going to work on adding another unit test proving that you can open multiple NBD clients in parallel, and that they can interleave I/O, but my immediate test was setting up: nbdkit -f --filter=ext2 --filter=exportname file tests/ext2.img \ ext2file=exportname exportname-list=explicit exportname=hidden then running two simultaneous nbdsh, one visiting nbd://localhost:10809/manifest and the other nbd://localhost:10809/disks/disk.img. Pre-series, the second nbdsh stalls until the first exits, post-series, both can read their respective file out of the overall image at the same time. Eric Blake (5): filters: Tighter rules on .get_ready/.after_fork threadlocal: Add way to store current context filters: Allow nbdkit_next_context_open outside client connection api: Add .cleanup callback ext2: Support parallel requests and connections docs/nbdkit-filter.pod | 65 ++++++-- docs/nbdkit-plugin.pod | 33 +++- include/nbdkit-filter.h | 10 +- include/nbdkit-plugin.h | 4 +- server/internal.h | 14 +- server/backend.c | 48 +++++- server/filters.c | 49 +++--- server/main.c | 1 + server/plugins.c | 32 ++-- server/protocol-handshake.c | 2 +- server/public.c | 24 ++- server/test-public.c | 12 +- server/threadlocal.c | 36 ++++- filters/exitwhen/exitwhen.c | 9 +- filters/ext2/ext2.c | 225 ++++++++++++++++------------ filters/extentlist/extentlist.c | 5 +- filters/log/log.c | 9 +- filters/multi-conn/multi-conn.c | 5 +- filters/pause/pause.c | 4 +- filters/rate/rate.c | 5 +- filters/retry/retry.c | 2 +- filters/stats/stats.c | 5 +- filters/tls-fallback/tls-fallback.c | 5 +- tests/test-layers-filter.c | 22 ++- tests/test-layers-plugin.c | 9 +- tests/test-layers.c | 42 ++++-- 26 files changed, 453 insertions(+), 224 deletions(-) -- 2.31.1

2 10

FYI: nbdfuse support for trim and writing zeroes
by Richard W.M. Jones 11 May '21

11 May '21

I added support for trimming to nbdfuse over the weekend: https://gitlab.com/nbdkit/libnbd/-/commit/800c021f39ddfa1dd40d8dcc8645463e6… This intercepts Linux fallocate(2) on the file and if the mode parameter contains FALLOC_FL_PUNCH_HOLE then it will punch a hole using nbd_trim. This all appears to work fine. As noted in the above commit, in theory we could also check for mode & FALLOC_FL_ZERO_RANGE. When I use "fallocate -z ..." on a file, I see this is the mode parameter being passed to the system call: fallocate(3, FALLOC_FL_ZERO_RANGE, 512, 512) = 0 However it appears as if the fuse kernel module doesn't pass the flag through to nbdfuse. (I checked the kernel code too, and that *seems* to agree with my assessment). So for now I left support for zeroing in there but disabled the test. The fuse developer mailing list seems to be mostly spam. There is this closed github issue which says to contact the kernel developers which I have not done: https://github.com/libfuse/libfuse/issues/395 Next step for nbdfuse is to add multithreading support which would greatly improve performance but is rather complicated to implement. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

1 0