On Fri, Jul 31, 2020 at 5:12 PM Eric Blake <eblake@redhat.com> wrote:

On 7/31/20 7:07 AM, Richard W.M. Jones wrote:

...

Hi Eric,

I wonder if you have any thoughts about this build failure in tests/test-nozero.sh?

https://koji.fedoraproject.org/koji/taskinfo?taskID=48259627 log: https://kojipkgs.fedoraproject.org//work/tasks/9762/48259762/build.log

The error is “nozero6.img was trimmed by mistake”. I added “set -x” to the script earlier today so we can see exactly what's wrong, and it is that:

++ stat -c %b nozero2.img ++ stat -c %b nozero6.img + test 4096 '!=' 2048 + echo 'nozero6.img was trimmed by mistake'

Hmm, maybe it is file-system dependent (not all filesystems reserve the same amount of space for a sparse file - that's something that qemu iotests keep on hitting).

In qemu iotets we check how much space an empty file is using to avoid issues with filesystem allocating extra blocks. https://github.com/qemu/qemu/blob/d74824cf7c8b352f9045e949dc636c7207a41eee/t... Not sure this is relevant to this case since this grows from 2048 blocks to 4096, not 1 extra block. In ovit we avoid these issues by testing on filesytems we control instead on random filesystem provided by the CI environment. We use this: https://github.com/nirs/userstorage The idea is that you configure some combinations you want to test, like: https://github.com/oVirt/ovirt-imageio/blob/master/storage.py Prepare this storage as root before running the tests: https://github.com/oVirt/ovirt-imageio/blob/7676d97e49eb1399ed5256e08786c006... Then the tests pick some of the available storage as needed: https://github.com/oVirt/ovirt-imageio/blob/7676d97e49eb1399ed5256e08786c006... Tests using the user_file fixture above are run once for every parameter, so we know the code works on 3 different filesystems with 2 logical block size. Nir

...

AFAICT what this means is that nozero2.img is growing during the test (from 2048 to 4096 blocks). When I run the test locally this file stays at 2048 blocks the whole time, and the test does not fail.

Growing a small amount but still being sparse is different from growing a huge amount to be non-sparse altogether. I'll have to double-check what the test is actually doing (the size of the files involved) and see if we can relax the test into allowing a range of sizes that show a file is still reasonably sparse. But knowing what filesystem koji is using may matter (for example, if this is something that shows up on btrfs but not ext4, that would explain why koji fails when I pass locally...)

...

One other unfortunate problem is that Fedora is having lots of toolchain problems right now (see Fedora devel list passim) so we cannot really be sure that *any* other tool we are using has been built correctly :-( I've already disabled LTO in qemu and libguestfs, but possibly there are other toolchain bugs.

Rich.

-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org

_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs

On Sat, Aug 01, 2020 at 08:46:11AM +0100, Richard W.M. Jones wrote:

One thing I noticed which is a bit odd is:

$ rm file; for f in {0..1023}; do printf '%1024s' .; done > file; stat -c "%b %B" file 2048 512 $ rm file; for f in {0..1023}; do printf '%1024s' . >> file; done ; stat -c "%b %B" file 3968 512

The second method is how we currently create the file. Since looking through the history there seems to be no reason for that I'm going to push a commit which changes file creation to the first method, and it may be slightly faster too.

However it makes me wonder if the file is not laid out in a single extent and if that might be causing our problems. Being only able to reproduce this on Koji makes a bit tedious to test theories :-(

I pushed this patch and did another test build in Koji, but the issue is still not fixed. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

On Mon, Aug 03, 2020 at 04:21:13PM -0500, Eric Blake wrote:

On 8/1/20 12:39 PM, Richard W.M. Jones wrote:

...

On Sat, Aug 01, 2020 at 08:46:11AM +0100, Richard W.M. Jones wrote:

...

One thing I noticed which is a bit odd is:

$ rm file; for f in {0..1023}; do printf '%1024s' .; done > file; stat -c "%b %B" file 2048 512 $ rm file; for f in {0..1023}; do printf '%1024s' . >> file; done ; stat -c "%b %B" file 3968 512

The second method is how we currently create the file. Since looking through the history there seems to be no reason for that I'm going to push a commit which changes file creation to the first method, and it may be slightly faster too.

However it makes me wonder if the file is not laid out in a single extent and if that might be causing our problems. Being only able to reproduce this on Koji makes a bit tedious to test theories :-(

I pushed this patch and did another test build in Koji, but the issue is still not fixed.

It looks like the real issue at hand is that stat is indeed reporting allocated size caused by the filesystem pre-emptively over-allocating based on access patterns (more so when creating the first file, especially when reopening the file; less so when copying as the source file size is now stabilized so the copy has less reason to overshoot). But since the real crux of the test is whether we managed to punch holes, would it be sufficient to take note of the original sizes, and merely check that the resulting size has either shrunk (where the file should now be sparser) or remained unchanged?

I'll push a patch along those lines, but as you're the one doing the koji runs, there's obviously another feedback cycle to go through.

Sure I'll give your patch a go once I see it in git, thanks! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

On 8/4/20 4:42 AM, Richard W.M. Jones wrote:

...

...

I'll push a patch along those lines, but as you're the one doing the koji runs, there's obviously another feedback cycle to go through.

Sure I'll give your patch a go once I see it in git, thanks!

Unfortunately no this didn't fix it. The log is:

https://kojipkgs.fedoraproject.org//work/tasks/9782/48609782/build.log

nozero2.img (for example) was originally:

nozero2.img: 2048 allocated blocks of size 512 bytes, total size 1048576

but when we test it later it has grown to 4096 blocks:

+ for i in {2..6} ++ stat -c %b nozero2.img + test 4096 '!=' 2048 + echo 'nozero2.img was trimmed by mistake' nozero2.img was trimmed by mistake + fail=1

So it has been "trimmed" from 2048 to 4096 blocks. Perhaps we should use -lt in that test to detect if the file got smaller?

Well, the point is that it wasn't trimmed - the nozero filter let literal zeroes be written in rather than attempting a trim. But still, it is odd that writing literal zeroes in place made the file occupy more reserved space, even though the file size is unchanged. I wonder if changing the file to be 2 megabytes in size, but only writing zeroes to the first megabyte, would be a more reliable way to avoid the file system thinking that because we just wrote zeroes all the way to the current EOF that we are likely to grow the file size, in order to thwart its preemptive allocations. But you also have a point that merely forbidding a shrinking file is an acceptable test (the fact that the file grew is not nice, but the file did not become sparse, and that's all we really cared about).

Also I was interested in what filesystem the builders are using (since I still cannot reproduce any of this without Koji). So I added ‘stat -f .’ in the %check section. It printed:

File: "." ID: fc0200000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 33538048 Free: 25172957 Available: 25172957 Inodes: Total: 67108864 Free: 66469270

I believe the builder is running Fedora 32, at least going by the kernel version. (This surprised me as I thought they were running RHEL.)

Rich.

-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org