[PATCH libnbd] copy/copy-file-to-qcow2-compressed.sh: Skip test for all zeroes disk
Occasionally this test will choose a random seed which results in an all-zeroes disk. The test tries to convert this to a compressed qcow2 file, and fails because no compressed clusters are detected in the resulting file. This happens because qcow2 stores zero clusters with a special sparse representation, they are never stored compressed, so a disk with only zeroes in it will never contain compressed clusters. To fix this, detect an all-zeroes disk and skip. Reported-by: Eric Blake --- copy/copy-file-to-qcow2-compressed.sh | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/copy/copy-file-to-qcow2-compressed.sh b/copy/copy-file-to-qcow2-compressed.sh index 018c8bba2f..2706eadd66 100755 --- a/copy/copy-file-to-qcow2-compressed.sh +++ b/copy/copy-file-to-qcow2-compressed.sh @@ -25,6 +25,7 @@ requires $QEMU_NBD --version requires nbdkit --exit-with-parent --version requires nbdkit sparse-random --dump-plugin requires qemu-img --version +requires nbdinfo --version #requires stat --version # Check the compress driver is supported by this qemu-nbd. @@ -45,6 +46,15 @@ cleanup_fn rm -f $file1 $file2 $out1 $out2 size=1G seed=$RANDOM +# Occasionally we will choose a seed which results in a completely +# empty file. Skip this case. +if nbdinfo --map --totals -- \ + [ nbdkit --exit-with-parent sparse-random $size seed=$seed ] | + grep -sq '100.0%.*hole,zero'; then + echo "$0: bad seed chosen, skipping the test" + exit 77 +fi + # Create a compressed qcow2 file1. # # sparse-random files should compress easily because by default each -- 2.41.0
After more testing today I pushed a slightly improved version of this patch as: https://gitlab.com/nbdkit/libnbd/-/commit/a333eec17a5ce4b2a4bbbd2f08bf5f4474... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 8/8/23 00:28, Richard W.M. Jones wrote:
Occasionally this test will choose a random seed which results in an all-zeroes disk. The test tries to convert this to a compressed qcow2 file, and fails because no compressed clusters are detected in the resulting file. This happens because qcow2 stores zero clusters with a special sparse representation, they are never stored compressed, so a disk with only zeroes in it will never contain compressed clusters.
To fix this, detect an all-zeroes disk and skip.
Reported-by: Eric Blake --- copy/copy-file-to-qcow2-compressed.sh | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/copy/copy-file-to-qcow2-compressed.sh b/copy/copy-file-to-qcow2-compressed.sh index 018c8bba2f..2706eadd66 100755 --- a/copy/copy-file-to-qcow2-compressed.sh +++ b/copy/copy-file-to-qcow2-compressed.sh @@ -25,6 +25,7 @@ requires $QEMU_NBD --version requires nbdkit --exit-with-parent --version requires nbdkit sparse-random --dump-plugin requires qemu-img --version +requires nbdinfo --version #requires stat --version
# Check the compress driver is supported by this qemu-nbd. @@ -45,6 +46,15 @@ cleanup_fn rm -f $file1 $file2 $out1 $out2 size=1G seed=$RANDOM
+# Occasionally we will choose a seed which results in a completely +# empty file. Skip this case. +if nbdinfo --map --totals -- \ + [ nbdkit --exit-with-parent sparse-random $size seed=$seed ] | + grep -sq '100.0%.*hole,zero'; then + echo "$0: bad seed chosen, skipping the test" + exit 77 +fi + # Create a compressed qcow2 file1. # # sparse-random files should compress easily because by default each
Nbdkit uses its own PRNG from "common/include/random.h", not the one from libc. That means the seed=... parameter entirely determines the random series -- not only does it become time-independent, but also platform-independent. And based on that, could we just eliminate "seed=$RANDOM" from this test case, and use simply "seed=1" (for example)? What we want is a sparse-random disk image, generated both times with the same not-fully-zero contents. seed=1 satisfies that, and makes sure the test is always "armed" (and, not least, deterministic). Laszlo
On Tue, Aug 08, 2023 at 01:38:31PM +0200, Laszlo Ersek wrote:
On 8/8/23 00:28, Richard W.M. Jones wrote:
Occasionally this test will choose a random seed which results in an all-zeroes disk. The test tries to convert this to a compressed qcow2 file, and fails because no compressed clusters are detected in the resulting file. This happens because qcow2 stores zero clusters with a special sparse representation, they are never stored compressed, so a disk with only zeroes in it will never contain compressed clusters.
To fix this, detect an all-zeroes disk and skip.
Reported-by: Eric Blake --- copy/copy-file-to-qcow2-compressed.sh | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/copy/copy-file-to-qcow2-compressed.sh b/copy/copy-file-to-qcow2-compressed.sh index 018c8bba2f..2706eadd66 100755 --- a/copy/copy-file-to-qcow2-compressed.sh +++ b/copy/copy-file-to-qcow2-compressed.sh @@ -25,6 +25,7 @@ requires $QEMU_NBD --version requires nbdkit --exit-with-parent --version requires nbdkit sparse-random --dump-plugin requires qemu-img --version +requires nbdinfo --version #requires stat --version
# Check the compress driver is supported by this qemu-nbd. @@ -45,6 +46,15 @@ cleanup_fn rm -f $file1 $file2 $out1 $out2 size=1G seed=$RANDOM
+# Occasionally we will choose a seed which results in a completely +# empty file. Skip this case. +if nbdinfo --map --totals -- \ + [ nbdkit --exit-with-parent sparse-random $size seed=$seed ] | + grep -sq '100.0%.*hole,zero'; then + echo "$0: bad seed chosen, skipping the test" + exit 77 +fi + # Create a compressed qcow2 file1. # # sparse-random files should compress easily because by default each
Nbdkit uses its own PRNG from "common/include/random.h", not the one from libc. That means the seed=... parameter entirely determines the random series -- not only does it become time-independent, but also platform-independent.
And based on that, could we just eliminate "seed=$RANDOM" from this test case, and use simply "seed=1" (for example)? What we want is a sparse-random disk image, generated both times with the same not-fully-zero contents. seed=1 satisfies that, and makes sure the test is always "armed" (and, not least, deterministic).
There's definitely an argument for this. When writing this test originally I wanted a stochastic test in the hope it would uncover corner-cases in nbdcopy. In practice stochastic tests have been a pain in the backside because they fail (as Eric found here) in ways we didn't anticipate and because they generally appear unreliable. And we have been changing others to be less random. Here is a good example: https://gitlab.com/nbdkit/nbdkit/-/blob/1b67e323e998a5d719f1afe43d5be84e45c6... I think I've patched up this test so it should work for now (famous last words). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Mon, Aug 07, 2023 at 11:28:42PM +0100, Richard W.M. Jones wrote:
Occasionally this test will choose a random seed which results in an all-zeroes disk. The test tries to convert this to a compressed qcow2 file, and fails because no compressed clusters are detected in the resulting file. This happens because qcow2 stores zero clusters with a special sparse representation, they are never stored compressed, so a disk with only zeroes in it will never contain compressed clusters.
To fix this, detect an all-zeroes disk and skip.
Skipping a stochastic test on the cases where a random number set up the corner case is still odd; our testsuite passes, but not always with the same number of tests. I understand benchmarks wanting to use stochastic results, but this particular test seems like one where we aren't buying ourselves any new coverage by using $RANDOM (other than the fact that we found a corner case where nbdkit's sparse-random can produce a 1G empty disk), and where a deterministic test better proves our intent 100% of the time. -- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
participants (3)
-
Eric Blake -
Laszlo Ersek -
Richard W.M. Jones