copy tests failing in libnbd with ZoL
[Adding the public mailing list] On Thu, Oct 23, 2025 at 02:13:58PM +0200, Arsen Arsenović wrote:
BTW, partially related; the following tests seem to fail on OpenZFS (ZoL, specifically):
FAIL: copy-allocated-destination-zero-asynch.sh FAIL: copy-allocated-destination-zero-synch.sh
I wonder if you can grab 'copy/copy-allocated-destination-zero-asynch.sh.log' and/or 'copy/copy-allocated-destination-zero-synch.sh.log' from the failing tests? The tests themselves check that these functions work respectively: https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53668... https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53668... In particular, the tests check for the path where target_is_zero == true and allocated == true, which should write allocated zeroes to the output file. Looking at the functions now, it seems broken to me to return immediately from the functions when target_is_zero. However if that code was wrong then the functions wouldn't work on any filesystem, it would have nothing to do with ZFS. So I'm not sure ... Having the logs would help.
I suspect the filesystem is at fault (tmpfs works fine).
Have a lovely day! -- Arsen Arsenović
Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
"Richard W.M. Jones" <rjones@redhat.com> writes:
I wonder if you can grab 'copy/copy-allocated-destination-zero-asynch.sh.log' and/or 'copy/copy-allocated-destination-zero-synch.sh.log' from the failing tests?
Sure thing. The contents of the logs of the synchronous version are: + requires cmp --version + requires truncate --version + requires stat --version + requires test -r /dev/zero + inp=copy-allocated-destination-zero-synch.in + out=copy-allocated-destination-zero-synch.out + cleanup_fn rm -f copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out + local _hook=_cleanup_hook0 + declare -ag _cleanup_hook0 + eval '_cleanup_hook0=("$@")' ++ _cleanup_hook0=("$@") + rm -f copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out + truncate -s 33554432 copy-allocated-destination-zero-synch.out + truncate -r copy-allocated-destination-zero-synch.out copy-allocated-destination-zero-synch.in + nbdcopy --allocated --destination-is-zero --synchronous --request-size=32K copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out libnbd: debug: nbd1: nbd_create: allocated the handle (9134 bytes) libnbd: debug: nbd1: nbd_is_uri: enter: uri="copy-allocated-destination-zero-synch.in" libnbd: debug: nbd1: nbd_is_uri: leave: ret=0 libnbd: debug: nbd1: nbd_close: closing handle libnbd: debug: nbd2: nbd_create: allocated the handle (9134 bytes) libnbd: debug: nbd2: nbd_is_uri: enter: uri="copy-allocated-destination-zero-synch.out" libnbd: debug: nbd2: nbd_is_uri: leave: ret=0 libnbd: debug: nbd2: nbd_close: closing handle + echo Output: Output: + ls -lsh copy-allocated-destination-zero-synch.out 512 -rw-r--r-- 1 portage portage 32M Oct 23 13:49 copy-allocated-destination-zero-synch.out ++ stat -c %s copy-allocated-destination-zero-synch.out + size=33554432 ++ stat -c %b copy-allocated-destination-zero-synch.out + balloc=1 ++ stat -c %B copy-allocated-destination-zero-synch.out + bsize=512 + alloc=512 + '[' 33554432 -ne 33554432 ']' + '[' 512 -ne 33554432 ']' + echo '/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh: allocated size is not 32M' /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh: allocated size is not 32M + exit 1 + _run_cleanup_hooks + local _status=1 _i + set +e + trap '' INT QUIT TERM EXIT ERR + echo /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh: run cleanup hooks: exit code 1 /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh: run cleanup hooks: exit code 1 + (( _i = 0 )) + (( _i < 1 )) + eval '"${_cleanup_hook0[@]}"' ++ rm -f copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out + (( ++_i )) + (( _i < 1 )) + exit 1 FAIL copy-allocated-destination-zero-synch.sh (exit status: 1) I suspect you might find the following interesting: $ truncate -s 33554432 copy-allocated-destination-zero-synch.out $ truncate -r copy-allocated-destination-zero-synch.out copy-allocated-destination-zero-synch.in $ ls -lsh copy* 512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:10 copy-allocated-destination-zero-synch.in 512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:10 copy-allocated-destination-zero-synch.out $ sudo /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/nbdcopy --allocated --destination-is-zero --synchronous --request-size=32K copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out $ ls -lsh copy* 512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:13 copy-allocated-destination-zero-synch.in 512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:13 copy-allocated-destination-zero-synch.out (forgive the 'sudo', portage has a strict umask so I can't access that path from my user) On tmpfs, I get the following instead: /tmp/foo$ ls -lsh * 0 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:12 copy-allocated-destination-zero-synch.in 0 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:12 copy-allocated-destination-zero-synch.out ... and, after running nbdcopy: /tmp/foo$ ls -lsh * 0 -rw-r--r-- 1 arsen arsen 32M Oct 23 13:58 copy-allocated-destination-zero-synch.in 32M -rw-r--r-- 1 arsen arsen 32M Oct 23 13:59 copy-allocated-destination-zero-synch.out -- Arsen Arsenović
On Thu, Oct 23, 2025 at 03:15:09PM +0200, Arsen Arsenović wrote:
"Richard W.M. Jones" <rjones@redhat.com> writes:
I wonder if you can grab 'copy/copy-allocated-destination-zero-asynch.sh.log' and/or 'copy/copy-allocated-destination-zero-synch.sh.log' from the failing tests?
Sure thing. The contents of the logs of the synchronous version are:
+ requires cmp --version + requires truncate --version + requires stat --version + requires test -r /dev/zero + inp=copy-allocated-destination-zero-synch.in + out=copy-allocated-destination-zero-synch.out + cleanup_fn rm -f copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out + local _hook=_cleanup_hook0 + declare -ag _cleanup_hook0 + eval '_cleanup_hook0=("$@")' ++ _cleanup_hook0=("$@") + rm -f copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out + truncate -s 33554432 copy-allocated-destination-zero-synch.out + truncate -r copy-allocated-destination-zero-synch.out copy-allocated-destination-zero-synch.in + nbdcopy --allocated --destination-is-zero --synchronous --request-size=32K copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out libnbd: debug: nbd1: nbd_create: allocated the handle (9134 bytes) libnbd: debug: nbd1: nbd_is_uri: enter: uri="copy-allocated-destination-zero-synch.in" libnbd: debug: nbd1: nbd_is_uri: leave: ret=0 libnbd: debug: nbd1: nbd_close: closing handle libnbd: debug: nbd2: nbd_create: allocated the handle (9134 bytes) libnbd: debug: nbd2: nbd_is_uri: enter: uri="copy-allocated-destination-zero-synch.out" libnbd: debug: nbd2: nbd_is_uri: leave: ret=0 libnbd: debug: nbd2: nbd_close: closing handle + echo Output: Output: + ls -lsh copy-allocated-destination-zero-synch.out 512 -rw-r--r-- 1 portage portage 32M Oct 23 13:49 copy-allocated-destination-zero-synch.out ++ stat -c %s copy-allocated-destination-zero-synch.out + size=33554432 ++ stat -c %b copy-allocated-destination-zero-synch.out + balloc=1 ++ stat -c %B copy-allocated-destination-zero-synch.out + bsize=512 + alloc=512 + '[' 33554432 -ne 33554432 ']' + '[' 512 -ne 33554432 ']' + echo '/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh: allocated size is not 32M' /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh: allocated size is not 32M + exit 1 + _run_cleanup_hooks + local _status=1 _i + set +e + trap '' INT QUIT TERM EXIT ERR + echo /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh: run cleanup hooks: exit code 1 /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh: run cleanup hooks: exit code 1 + (( _i = 0 )) + (( _i < 1 )) + eval '"${_cleanup_hook0[@]}"' ++ rm -f copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out + (( ++_i )) + (( _i < 1 )) + exit 1 FAIL copy-allocated-destination-zero-synch.sh (exit status: 1)
I suspect you might find the following interesting:
$ truncate -s 33554432 copy-allocated-destination-zero-synch.out $ truncate -r copy-allocated-destination-zero-synch.out copy-allocated-destination-zero-synch.in $ ls -lsh copy* 512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:10 copy-allocated-destination-zero-synch.in 512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:10 copy-allocated-destination-zero-synch.out $ sudo /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/nbdcopy --allocated --destination-is-zero --synchronous --request-size=32K copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out $ ls -lsh copy* 512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:13 copy-allocated-destination-zero-synch.in 512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:13 copy-allocated-destination-zero-synch.out
(forgive the 'sudo', portage has a strict umask so I can't access that path from my user)
On tmpfs, I get the following instead:
/tmp/foo$ ls -lsh * 0 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:12 copy-allocated-destination-zero-synch.in 0 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:12 copy-allocated-destination-zero-synch.out
... and, after running nbdcopy:
/tmp/foo$ ls -lsh * 0 -rw-r--r-- 1 arsen arsen 32M Oct 23 13:58 copy-allocated-destination-zero-synch.in 32M -rw-r--r-- 1 arsen arsen 32M Oct 23 13:59 copy-allocated-destination-zero-synch.out
I don't have OpenZFS to test, but what seems to be happening on regular filesystems is we call truncate on the destination before doing the copy: https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53668... which eventually calls fallocate with mode == 0 here: https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53668... Confirmed in strace: 2745865 ftruncate(4, 0) = 0 2745865 ftruncate(4, 33554432) = 0 2745865 fallocate(4, 0, 0, 33554432) = 0 In Linux this is documented as: Allocating disk space The default operation (i.e., mode is zero) of fallocate() allocates the disk space within the range specified by offset and len. [...] which is the behaviour that we want here, allocating the space. We make no further writes to the output disk in this test, but end up with a 32M fully allocated, zeroed disk (what we want). ZoL is probably ignoring fallocate and not allocating the space. It may be returning ENOTSUP / EOPNOTSUPP, which we deliberately ignore. I'm fairly sure this is a libnbd bug however, since we shouldn't ignore unsupported fallocate (if ZoL returns that), if the --allocated flag is set. Eric, any thoughts? The code path that this pair of flags takes is very convoluted ... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
"Richard W.M. Jones" <rjones@redhat.com> writes:
I don't have OpenZFS to test, but what seems to be happening on regular filesystems is we call truncate on the destination before doing the copy:
https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53668...
which eventually calls fallocate with mode == 0 here:
https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53668...
Confirmed in strace:
2745865 ftruncate(4, 0) = 0 2745865 ftruncate(4, 33554432) = 0 2745865 fallocate(4, 0, 0, 33554432) = 0
In Linux this is documented as:
Allocating disk space The default operation (i.e., mode is zero) of fallocate() allocates the disk space within the range specified by offset and len. [...]
which is the behaviour that we want here, allocating the space. We make no further writes to the output disk in this test, but end up with a 32M fully allocated, zeroed disk (what we want).
ZoL is probably ignoring fallocate and not allocating the space. It may be returning ENOTSUP / EOPNOTSUPP, which we deliberately ignore.
FWIW, that does not appear to be the case: ~/tmp$ strace -e trace=ftruncate,fallocate truncate -s 33554432 copy-allocated-destination-zero-synch.out ftruncate(3, 33554432) = 0 +++ exited with 0 +++ ~/tmp$ strace -e trace=ftruncate,fallocate truncate -r copy-allocated-destination-zero-synch.out copy-allocated-destination-zero-synch.inftruncate(3, 33554432) = 0 +++ exited with 0 +++ ~/tmp$ strace -e trace=ftruncate,fallocate nbdcopy --allocated --destination-is-zero --synchronous --request-size=32K copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971755, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971757, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971758, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971761, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971764, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971765, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- ftruncate(4, 0) = 0 ftruncate(4, 33554432) = 0 fallocate(4, 0, 0, 33554432) = 0 +++ exited with 0 +++ ~/tmp$ -- Arsen Arsenović
On Thu, Oct 23, 2025 at 04:34:36PM +0200, Arsen Arsenović wrote:
~/tmp$ strace -e trace=ftruncate,fallocate nbdcopy --allocated --destination-is-zero --synchronous --request-size=32K copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971755, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971757, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971758, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971761, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971764, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971765, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- ftruncate(4, 0) = 0 ftruncate(4, 33554432) = 0 fallocate(4, 0, 0, 33554432) = 0
I was definitely expecting this one to fail with ENOTSUP or similar. That's with the file copy-allocated-destination-zero-synch.out on ZFS? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Thu, Oct 23, 2025 at 06:45:33PM +0200, Arsen Arsenović wrote:
"Richard W.M. Jones" <rjones@redhat.com> writes:
I was definitely expecting this one to fail with ENOTSUP or similar. That's with the file copy-allocated-destination-zero-synch.out on ZFS?
Yes - my home directory is on ZoL, and so is ~/tmp.
It appears this is a limitation of OpenZFS: https://github.com/openzfs/zfs/blob/fc519b2c1108b52aaad63e356645f63c2c167f82... <quote> * The original mode=0 (allocate space) behavior can be reasonably emulated * by checking if enough space exists and creating a sparse file, as real * persistent space reservation is not possible due to COW, snapshots, etc. </quote> Looking at the code when mode == 0, I believe we hit the second case (line 707), which seems to sparsify the file and (if that succeeeds) returns no error :-( We should probably add a 'requires' to the test. I think it could still be a bug in libnbd that we don't fail if fallocate fails, but it wouldn't have helped here. Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
"Richard W.M. Jones" <rjones@redhat.com> writes:
On Thu, Oct 23, 2025 at 06:45:33PM +0200, Arsen Arsenović wrote:
"Richard W.M. Jones" <rjones@redhat.com> writes:
I was definitely expecting this one to fail with ENOTSUP or similar. That's with the file copy-allocated-destination-zero-synch.out on ZFS?
Yes - my home directory is on ZoL, and so is ~/tmp.
It appears this is a limitation of OpenZFS:
https://github.com/openzfs/zfs/blob/fc519b2c1108b52aaad63e356645f63c2c167f82...
I figured that might be the case. ZFS has a few instances of similar issues.
<quote> * The original mode=0 (allocate space) behavior can be reasonably emulated * by checking if enough space exists and creating a sparse file, as real * persistent space reservation is not possible due to COW, snapshots, etc. </quote>
Looking at the code when mode == 0, I believe we hit the second case (line 707), which seems to sparsify the file and (if that succeeeds) returns no error :-(
We should probably add a 'requires' to the test. I think it could still be a bug in libnbd that we don't fail if fallocate fails, but it wouldn't have helped here.
A reasonable check might be to check whether a truncated file has zero blocks, I suppose. -- Arsen Arsenović
On Thu, Oct 23, 2025 at 07:18:24PM +0200, Arsen Arsenović wrote:
"Richard W.M. Jones" <rjones@redhat.com> writes:
On Thu, Oct 23, 2025 at 06:45:33PM +0200, Arsen Arsenović wrote:
"Richard W.M. Jones" <rjones@redhat.com> writes:
I was definitely expecting this one to fail with ENOTSUP or similar. That's with the file copy-allocated-destination-zero-synch.out on ZFS?
Yes - my home directory is on ZoL, and so is ~/tmp.
It appears this is a limitation of OpenZFS:
https://github.com/openzfs/zfs/blob/fc519b2c1108b52aaad63e356645f63c2c167f82...
I figured that might be the case. ZFS has a few instances of similar issues.
In QEMU, we ended up skipping several tests on ZFS that rely on measuring sparseness, because of the unpredictability of ZFS: https://gitlab.com/qemu-project/qemu/-/commit/c49dda72 -- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
Eric Blake <eblake@redhat.com> writes:
In QEMU, we ended up skipping several tests on ZFS that rely on measuring sparseness, because of the unpredictability of ZFS:
Yes, that seems like the reasonable option here, too. -- Arsen Arsenović
Merge request for this: https://gitlab.com/nbdkit/libnbd/-/merge_requests/30 -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
participants (3)
-
Arsen Arsenović -
Eric Blake -
Richard W.M. Jones