7:37 a.m.
[Adding the public mailing list]
On Thu, Oct 23, 2025 at 02:13:58PM +0200, Arsen Arsenović wrote:
BTW, partially related; the following tests seem to fail on OpenZFS
(ZoL, specifically):
FAIL: copy-allocated-destination-zero-asynch.sh
FAIL: copy-allocated-destination-zero-synch.sh
I wonder if you can grab 'copy/copy-allocated-destination-zero-asynch.sh.log'
and/or 'copy/copy-allocated-destination-zero-synch.sh.log' from the
failing tests?
The tests themselves check that these functions work respectively:
https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53...
https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53...
In particular, the tests check for the path where target_is_zero == true
and allocated == true, which should write allocated zeroes to the output file.
Looking at the functions now, it seems broken to me to return
immediately from the functions when target_is_zero. However if that
code was wrong then the functions wouldn't work on any filesystem, it
would have nothing to do with ZFS. So I'm not sure ...
Having the logs would help.
I suspect the filesystem is at fault (tmpfs works fine).
Have a lovely day!
--
Arsen Arsenović
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
8:15 a.m.
"Richard W.M. Jones" <rjones(a)redhat.com> writes:
I wonder if you can grab
'copy/copy-allocated-destination-zero-asynch.sh.log'
and/or 'copy/copy-allocated-destination-zero-synch.sh.log' from the
failing tests?
Sure thing. The contents of the logs of the synchronous version are:
+ requires cmp --version
+ requires truncate --version
+ requires stat --version
+ requires test -r /dev/zero
+ inp=copy-allocated-destination-zero-synch.in
+ out=copy-allocated-destination-zero-synch.out
+ cleanup_fn rm -f copy-allocated-destination-zero-synch.in
copy-allocated-destination-zero-synch.out
+ local _hook=_cleanup_hook0
+ declare -ag _cleanup_hook0
+ eval '_cleanup_hook0=("$@")'
++ _cleanup_hook0=("$@")
+ rm -f copy-allocated-destination-zero-synch.in
copy-allocated-destination-zero-synch.out
+ truncate -s 33554432 copy-allocated-destination-zero-synch.out
+ truncate -r copy-allocated-destination-zero-synch.out
copy-allocated-destination-zero-synch.in
+ nbdcopy --allocated --destination-is-zero --synchronous --request-size=32K
copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out
libnbd: debug: nbd1: nbd_create: allocated the handle (9134 bytes)
libnbd: debug: nbd1: nbd_is_uri: enter:
uri="copy-allocated-destination-zero-synch.in"
libnbd: debug: nbd1: nbd_is_uri: leave: ret=0
libnbd: debug: nbd1: nbd_close: closing handle
libnbd: debug: nbd2: nbd_create: allocated the handle (9134 bytes)
libnbd: debug: nbd2: nbd_is_uri: enter:
uri="copy-allocated-destination-zero-synch.out"
libnbd: debug: nbd2: nbd_is_uri: leave: ret=0
libnbd: debug: nbd2: nbd_close: closing handle
+ echo Output:
Output:
+ ls -lsh copy-allocated-destination-zero-synch.out
512 -rw-r--r-- 1 portage portage 32M Oct 23 13:49
copy-allocated-destination-zero-synch.out
++ stat -c %s copy-allocated-destination-zero-synch.out
+ size=33554432
++ stat -c %b copy-allocated-destination-zero-synch.out
+ balloc=1
++ stat -c %B copy-allocated-destination-zero-synch.out
+ bsize=512
+ alloc=512
+ '[' 33554432 -ne 33554432 ']'
+ '[' 512 -ne 33554432 ']'
+ echo
'/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh:
allocated size is not 32M'
/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh:
allocated size is not 32M
+ exit 1
+ _run_cleanup_hooks
+ local _status=1 _i
+ set +e
+ trap '' INT QUIT TERM EXIT ERR
+ echo
/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh:
run cleanup hooks: exit code 1
/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh:
run cleanup hooks: exit code 1
+ (( _i = 0 ))
+ (( _i < 1 ))
+ eval '"${_cleanup_hook0[@]}"'
++ rm -f copy-allocated-destination-zero-synch.in
copy-allocated-destination-zero-synch.out
+ (( ++_i ))
+ (( _i < 1 ))
+ exit 1
FAIL copy-allocated-destination-zero-synch.sh (exit status: 1)
I suspect you might find the following interesting:
$ truncate -s 33554432 copy-allocated-destination-zero-synch.out
$ truncate -r copy-allocated-destination-zero-synch.out
copy-allocated-destination-zero-synch.in
$ ls -lsh copy*
512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:10 copy-allocated-destination-zero-synch.in
512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:10 copy-allocated-destination-zero-synch.out
$ sudo /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/nbdcopy
--allocated --destination-is-zero --synchronous --request-size=32K
copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out
$ ls -lsh copy*
512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:13 copy-allocated-destination-zero-synch.in
512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:13 copy-allocated-destination-zero-synch.out
(forgive the 'sudo', portage has a strict umask so I can't access that
path from my user)
On tmpfs, I get the following instead:
/tmp/foo$ ls -lsh *
0 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:12 copy-allocated-destination-zero-synch.in
0 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:12 copy-allocated-destination-zero-synch.out
... and, after running nbdcopy:
/tmp/foo$ ls -lsh *
0 -rw-r--r-- 1 arsen arsen 32M Oct 23 13:58 copy-allocated-destination-zero-synch.in
32M -rw-r--r-- 1 arsen arsen 32M Oct 23 13:59 copy-allocated-destination-zero-synch.out
--
Arsen Arsenović
9:02 a.m.
On Thu, Oct 23, 2025 at 03:15:09PM +0200, Arsen Arsenović wrote:
"Richard W.M. Jones" <rjones(a)redhat.com> writes:
> I wonder if you can grab
'copy/copy-allocated-destination-zero-asynch.sh.log'
> and/or 'copy/copy-allocated-destination-zero-synch.sh.log' from the
> failing tests?
Sure thing. The contents of the logs of the synchronous version are:
+ requires cmp --version
+ requires truncate --version
+ requires stat --version
+ requires test -r /dev/zero
+ inp=copy-allocated-destination-zero-synch.in
+ out=copy-allocated-destination-zero-synch.out
+ cleanup_fn rm -f copy-allocated-destination-zero-synch.in
copy-allocated-destination-zero-synch.out
+ local _hook=_cleanup_hook0
+ declare -ag _cleanup_hook0
+ eval '_cleanup_hook0=("$@")'
++ _cleanup_hook0=("$@")
+ rm -f copy-allocated-destination-zero-synch.in
copy-allocated-destination-zero-synch.out
+ truncate -s 33554432 copy-allocated-destination-zero-synch.out
+ truncate -r copy-allocated-destination-zero-synch.out
copy-allocated-destination-zero-synch.in
+ nbdcopy --allocated --destination-is-zero --synchronous --request-size=32K
copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out
libnbd: debug: nbd1: nbd_create: allocated the handle (9134 bytes)
libnbd: debug: nbd1: nbd_is_uri: enter:
uri="copy-allocated-destination-zero-synch.in"
libnbd: debug: nbd1: nbd_is_uri: leave: ret=0
libnbd: debug: nbd1: nbd_close: closing handle
libnbd: debug: nbd2: nbd_create: allocated the handle (9134 bytes)
libnbd: debug: nbd2: nbd_is_uri: enter:
uri="copy-allocated-destination-zero-synch.out"
libnbd: debug: nbd2: nbd_is_uri: leave: ret=0
libnbd: debug: nbd2: nbd_close: closing handle
+ echo Output:
Output:
+ ls -lsh copy-allocated-destination-zero-synch.out
512 -rw-r--r-- 1 portage portage 32M Oct 23 13:49
copy-allocated-destination-zero-synch.out
++ stat -c %s copy-allocated-destination-zero-synch.out
+ size=33554432
++ stat -c %b copy-allocated-destination-zero-synch.out
+ balloc=1
++ stat -c %B copy-allocated-destination-zero-synch.out
+ bsize=512
+ alloc=512
+ '[' 33554432 -ne 33554432 ']'
+ '[' 512 -ne 33554432 ']'
+ echo
'/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh:
allocated size is not 32M'
/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh:
allocated size is not 32M
+ exit 1
+ _run_cleanup_hooks
+ local _status=1 _i
+ set +e
+ trap '' INT QUIT TERM EXIT ERR
+ echo
/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh:
run cleanup hooks: exit code 1
/var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/copy-allocated-destination-zero-synch.sh:
run cleanup hooks: exit code 1
+ (( _i = 0 ))
+ (( _i < 1 ))
+ eval '"${_cleanup_hook0[@]}"'
++ rm -f copy-allocated-destination-zero-synch.in
copy-allocated-destination-zero-synch.out
+ (( ++_i ))
+ (( _i < 1 ))
+ exit 1
FAIL copy-allocated-destination-zero-synch.sh (exit status: 1)
I suspect you might find the following interesting:
$ truncate -s 33554432 copy-allocated-destination-zero-synch.out
$ truncate -r copy-allocated-destination-zero-synch.out
copy-allocated-destination-zero-synch.in
$ ls -lsh copy*
512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:10 copy-allocated-destination-zero-synch.in
512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:10
copy-allocated-destination-zero-synch.out
$ sudo /var/tmp/portage/sys-libs/libnbd-1.23.9/work/libnbd-1.23.9/copy/nbdcopy
--allocated --destination-is-zero --synchronous --request-size=32K
copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out
$ ls -lsh copy*
512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:13 copy-allocated-destination-zero-synch.in
512 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:13
copy-allocated-destination-zero-synch.out
(forgive the 'sudo', portage has a strict umask so I can't access that
path from my user)
On tmpfs, I get the following instead:
/tmp/foo$ ls -lsh *
0 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:12 copy-allocated-destination-zero-synch.in
0 -rw-r--r-- 1 arsen arsen 32M Oct 23 15:12 copy-allocated-destination-zero-synch.out
... and, after running nbdcopy:
/tmp/foo$ ls -lsh *
0 -rw-r--r-- 1 arsen arsen 32M Oct 23 13:58 copy-allocated-destination-zero-synch.in
32M -rw-r--r-- 1 arsen arsen 32M Oct 23 13:59 copy-allocated-destination-zero-synch.out
I don't have OpenZFS to test, but what seems to be happening on
regular filesystems is we call truncate on the destination before
doing the copy:
https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53...
which eventually calls fallocate with mode == 0 here:
https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53...
Confirmed in strace:
2745865 ftruncate(4, 0) = 0
2745865 ftruncate(4, 33554432) = 0
2745865 fallocate(4, 0, 0, 33554432) = 0
In Linux this is documented as:
Allocating disk space
The default operation (i.e., mode is zero) of fallocate() allocates the
disk space within the range specified by offset and len. [...]
which is the behaviour that we want here, allocating the space. We
make no further writes to the output disk in this test, but end up
with a 32M fully allocated, zeroed disk (what we want).
ZoL is probably ignoring fallocate and not allocating the space. It
may be returning ENOTSUP / EOPNOTSUPP, which we deliberately ignore.
I'm fairly sure this is a libnbd bug however, since we shouldn't
ignore unsupported fallocate (if ZoL returns that), if the --allocated
flag is set.
Eric, any thoughts?
The code path that this pair of flags takes is very convoluted ...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
9:34 a.m.
"Richard W.M. Jones" <rjones(a)redhat.com> writes:
I don't have OpenZFS to test, but what seems to be happening on
regular filesystems is we call truncate on the destination before
doing the copy:
https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53...
which eventually calls fallocate with mode == 0 here:
https://gitlab.com/nbdkit/libnbd/-/blob/6182a590ced685694574b74884c9cec53...
Confirmed in strace:
2745865 ftruncate(4, 0) = 0
2745865 ftruncate(4, 33554432) = 0
2745865 fallocate(4, 0, 0, 33554432) = 0
In Linux this is documented as:
Allocating disk space
The default operation (i.e., mode is zero) of fallocate() allocates the
disk space within the range specified by offset and len. [...]
which is the behaviour that we want here, allocating the space. We
make no further writes to the output disk in this test, but end up
with a 32M fully allocated, zeroed disk (what we want).
ZoL is probably ignoring fallocate and not allocating the space. It
may be returning ENOTSUP / EOPNOTSUPP, which we deliberately ignore.
FWIW, that does not appear to be the case:
~/tmp$ strace -e trace=ftruncate,fallocate truncate -s 33554432
copy-allocated-destination-zero-synch.out
ftruncate(3, 33554432) = 0
+++ exited with 0 +++
~/tmp$ strace -e trace=ftruncate,fallocate truncate -r
copy-allocated-destination-zero-synch.out
copy-allocated-destination-zero-synch.inftruncate(3, 33554432) = 0
+++ exited with 0 +++
~/tmp$ strace -e trace=ftruncate,fallocate nbdcopy --allocated --destination-is-zero
--synchronous --request-size=32K copy-allocated-destination-zero-synch.in
copy-allocated-destination-zero-synch.out
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971755, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971757, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971758, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971761, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971764, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971765, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
ftruncate(4, 0) = 0
ftruncate(4, 33554432) = 0
fallocate(4, 0, 0, 33554432) = 0
+++ exited with 0 +++
~/tmp$
--
Arsen Arsenović
11:27 a.m.
On Thu, Oct 23, 2025 at 04:34:36PM +0200, Arsen Arsenović wrote:
~/tmp$ strace -e trace=ftruncate,fallocate nbdcopy --allocated
--destination-is-zero --synchronous --request-size=32K
copy-allocated-destination-zero-synch.in copy-allocated-destination-zero-synch.out
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971755, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971757, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971758, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971761, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971764, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1971765, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
ftruncate(4, 0) = 0
ftruncate(4, 33554432) = 0
fallocate(4, 0, 0, 33554432) = 0
I was definitely expecting this one to fail with ENOTSUP or similar.
That's with the file copy-allocated-destination-zero-synch.out on ZFS?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
12:06 p.m.
On Thu, Oct 23, 2025 at 06:45:33PM +0200, Arsen Arsenović wrote:
"Richard W.M. Jones" <rjones(a)redhat.com> writes:
> I was definitely expecting this one to fail with ENOTSUP or similar.
> That's with the file copy-allocated-destination-zero-synch.out on ZFS?
Yes - my home directory is on ZoL, and so is ~/tmp.
It appears this is a limitation of OpenZFS:
https://github.com/openzfs/zfs/blob/fc519b2c1108b52aaad63e356645f63c2c167...
<quote>
* The original mode=0 (allocate space) behavior can be reasonably emulated
* by checking if enough space exists and creating a sparse file, as real
* persistent space reservation is not possible due to COW, snapshots, etc.
</quote>
Looking at the code when mode == 0, I believe we hit the second case
(line 707), which seems to sparsify the file and (if that succeeeds)
returns no error :-(
We should probably add a 'requires' to the test. I think it could
still be a bug in libnbd that we don't fail if fallocate fails, but it
wouldn't have helped here.
Thanks,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
12:18 p.m.
"Richard W.M. Jones" <rjones(a)redhat.com> writes:
On Thu, Oct 23, 2025 at 06:45:33PM +0200, Arsen Arsenović wrote:
> "Richard W.M. Jones" <rjones(a)redhat.com> writes:
>
> > I was definitely expecting this one to fail with ENOTSUP or similar.
> > That's with the file copy-allocated-destination-zero-synch.out on ZFS?
>
> Yes - my home directory is on ZoL, and so is ~/tmp.
It appears this is a limitation of OpenZFS:
https://github.com/openzfs/zfs/blob/fc519b2c1108b52aaad63e356645f63c2c167...
I figured that might be the case. ZFS has a few instances of similar
issues.
<quote>
* The original mode=0 (allocate space) behavior can be reasonably emulated
* by checking if enough space exists and creating a sparse file, as real
* persistent space reservation is not possible due to COW, snapshots, etc.
</quote>
Looking at the code when mode == 0, I believe we hit the second case
(line 707), which seems to sparsify the file and (if that succeeeds)
returns no error :-(
We should probably add a 'requires' to the test. I think it could
still be a bug in libnbd that we don't fail if fallocate fails, but it
wouldn't have helped here.
A reasonable check might be to check whether a truncated file has zero
blocks, I suppose.
--
Arsen Arsenović
3:08 p.m.
On Thu, Oct 23, 2025 at 07:18:24PM +0200, Arsen Arsenović wrote:
"Richard W.M. Jones" <rjones(a)redhat.com> writes:
> On Thu, Oct 23, 2025 at 06:45:33PM +0200, Arsen Arsenović wrote:
>> "Richard W.M. Jones" <rjones(a)redhat.com> writes:
>>
>> > I was definitely expecting this one to fail with ENOTSUP or similar.
>> > That's with the file copy-allocated-destination-zero-synch.out on ZFS?
>>
>> Yes - my home directory is on ZoL, and so is ~/tmp.
>
> It appears this is a limitation of OpenZFS:
>
>
https://github.com/openzfs/zfs/blob/fc519b2c1108b52aaad63e356645f63c2c167...
I figured that might be the case. ZFS has a few instances of similar
issues.
In QEMU, we ended up skipping several tests on ZFS that rely on
measuring sparseness, because of the unpredictability of ZFS:
https://gitlab.com/qemu-project/qemu/-/commit/c49dda72
--
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization: qemu.org | libguestfs.org
4:44 p.m.
Eric Blake <eblake(a)redhat.com> writes:
In QEMU, we ended up skipping several tests on ZFS that rely on
measuring sparseness, because of the unpredictability of ZFS:
https://gitlab.com/qemu-project/qemu/-/commit/c49dda72
Yes, that seems like the reasonable option here, too.
--
Arsen Arsenović
4:10 a.m.
Merge request for this:
https://gitlab.com/nbdkit/libnbd/-/merge_requests/30
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
22
days inactive
23
days old
2 comments
2 participants
participants (2)
-
Arsen Arsenović -
Eric Blake -
Richard W.M. Jones