[libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping
Qemu has unfortunately used two distinct meanings for 'allocated' over time, and that ambiguity has overloaded the word to make it difficult to know which sense is meant in a given context. In one usage, 'allocated' implies that data is present and occupies space on a block device (the opposite of sparse). In another usage, 'allocated' implies that guest contents come from the current layer of a qcow2 backing chain (even if via a sparse zero cluster), rather than deferring to the backing file. It doesn't help that in qemu source code, BDRV_BLOCK_ALLOCATED refers solely to this latter usage, while a preallocated qcow2 image refers to the former usage. But we can help by using better terminology in nbdinfo output. With base:allocation, either a cluster is known to be sparse (a hole), or it (likely) occupies space (remember, NBD block status bits are advisory - if they are set to 1, we know a given property is true; but if they are left 0, we cannot assume the opposite of the property). Instead of calling the latter allocated, we can call it merely "data". And this maps a bit better to 'qemu-img map --output=json' output; a sparse section really is "data":false, and everything that is not known to be sparse is "data":true. Similarly, in qemu:allocation-depth, either a cluster is known to come from the backing chain (either locally, or from a given backing depth), or it is completely absent from the chain (not allocated in any of the layers). Instead of calling it "unallocated", we can call it "absent". Adjust the testsuite to match our improved terminology. Thanks: Nir Soffer --- info/info-map-base-allocation-json.sh | 4 ++-- info/info-map-base-allocation-large.sh | 8 ++++---- info/info-map-base-allocation-weird.sh | 4 ++-- info/info-map-base-allocation.sh | 6 +++--- info/info-map-qemu-allocation-depth.sh | 4 ++-- info/map.c | 4 ++-- 6 files changed, 15 insertions(+), 15 deletions(-) diff --git a/info/info-map-base-allocation-json.sh b/info/info-map-base-allocation-json.sh index 961eac6..b960295 100755 --- a/info/info-map-base-allocation-json.sh +++ b/info/info-map-base-allocation-json.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -40,7 +40,7 @@ jq . < $out test $( jq -r '.[0].offset' < $out ) -eq 0 test $( jq -r '.[0].length' < $out ) -eq 32768 test $( jq -r '.[0].type' < $out ) -eq 0 -test $( jq -r '.[0].description' < $out ) = "allocated" +test $( jq -r '.[0].description' < $out ) = "data" test $( jq -r '.[3].offset' < $out ) -eq 163840 test $( jq -r '.[3].length' < $out ) -eq 884736 diff --git a/info/info-map-base-allocation-large.sh b/info/info-map-base-allocation-large.sh index e7ff904..ab10dc7 100755 --- a/info/info-map-base-allocation-large.sh +++ b/info/info-map-base-allocation-large.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -36,11 +36,11 @@ nbdkit -U - data data='1 @131072 2 @6442450944 3' size=8G \ cat $out -if [ "$(tr -s ' ' < $out)" != " 0 32768 0 allocated +if [ "$(tr -s ' ' < $out)" != " 0 32768 0 data 32768 98304 3 hole,zero - 131072 32768 0 allocated + 131072 32768 0 data 163840 6442287104 3 hole,zero -6442450944 32768 0 allocated +6442450944 32768 0 data 6442483712 2147450880 3 hole,zero" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-base-allocation-weird.sh b/info/info-map-base-allocation-weird.sh index 20ddec6..0b0343a 100755 --- a/info/info-map-base-allocation-weird.sh +++ b/info/info-map-base-allocation-weird.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -45,7 +45,7 @@ EOF cat $out if [ "$(tr -s ' ' < $out)" != " 0 16 1 hole - 16 4 0 allocated + 16 4 0 data 20 12 1 hole" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-base-allocation.sh b/info/info-map-base-allocation.sh index 9b85c97..76c2024 100755 --- a/info/info-map-base-allocation.sh +++ b/info/info-map-base-allocation.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -36,9 +36,9 @@ nbdkit -U - data data='1 @131072 2' size=1M \ cat $out -if [ "$(tr -s ' ' < $out)" != " 0 32768 0 allocated +if [ "$(tr -s ' ' < $out)" != " 0 32768 0 data 32768 98304 3 hole,zero - 131072 32768 0 allocated + 131072 32768 0 data 163840 884736 3 hole,zero" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-qemu-allocation-depth.sh b/info/info-map-qemu-allocation-depth.sh index a5dab0b..9aeffd1 100755 --- a/info/info-map-qemu-allocation-depth.sh +++ b/info/info-map-qemu-allocation-depth.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -84,7 +84,7 @@ if [ "$(tr -s ' ' < $out)" != "\ 3145728 2097152 3 backing depth 3 5242880 1048576 1 local 6291456 1048576 2 backing depth 2 - 7340032 1048576 0 unallocated" ]; then + 7340032 1048576 0 absent" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 fi diff --git a/info/map.c b/info/map.c index ae6d4fe..82c9507 100644 --- a/info/map.c +++ b/info/map.c @@ -202,7 +202,7 @@ extent_description (const char *metacontext, uint32_t type) if (strcmp (metacontext, "base:allocation") == 0) { switch (type) { - case 0: return strdup ("allocated"); + case 0: return strdup ("data"); case 1: return strdup ("hole"); case 2: return strdup ("zero"); case 3: return strdup ("hole,zero"); @@ -216,7 +216,7 @@ extent_description (const char *metacontext, uint32_t type) } else if (strcmp (metacontext, "qemu:allocation-depth") == 0) { switch (type) { - case 0: return strdup ("unallocated"); + case 0: return strdup ("absent"); case 1: return strdup ("local"); default: if (asprintf (&ret, "backing depth %u", type) == -1) { -- 2.31.1
On Thu, Jun 10, 2021 at 6:03 PM Eric Blake <eblake@redhat.com> wrote:
Qemu has unfortunately used two distinct meanings for 'allocated' over time, and that ambiguity has overloaded the word to make it difficult to know which sense is meant in a given context. In one usage, 'allocated' implies that data is present and occupies space on a block device (the opposite of sparse). In another usage, 'allocated' implies that guest contents come from the current layer of a qcow2 backing chain (even if via a sparse zero cluster), rather than deferring to the backing file. It doesn't help that in qemu source code, BDRV_BLOCK_ALLOCATED refers solely to this latter usage, while a preallocated qcow2 image refers to the former usage.
But we can help by using better terminology in nbdinfo output. With base:allocation, either a cluster is known to be sparse (a hole), or it (likely) occupies space (remember, NBD block status bits are advisory - if they are set to 1, we know a given property is true; but if they are left 0, we cannot assume the opposite of the property). Instead of calling the latter allocated, we can call it merely "data". And this maps a bit better to 'qemu-img map --output=json' output; a sparse section really is "data":false, and everything that is not known to be sparse is "data":true.
Similarly, in qemu:allocation-depth, either a cluster is known to come from the backing chain (either locally, or from a given backing depth), or it is completely absent from the chain (not allocated in any of the layers). Instead of calling it "unallocated", we can call it "absent".
Adjust the testsuite to match our improved terminology.
Thanks: Nir Soffer --- info/info-map-base-allocation-json.sh | 4 ++-- info/info-map-base-allocation-large.sh | 8 ++++---- info/info-map-base-allocation-weird.sh | 4 ++-- info/info-map-base-allocation.sh | 6 +++--- info/info-map-qemu-allocation-depth.sh | 4 ++-- info/map.c | 4 ++-- 6 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/info/info-map-base-allocation-json.sh b/info/info-map-base-allocation-json.sh index 961eac6..b960295 100755 --- a/info/info-map-base-allocation-json.sh +++ b/info/info-map-base-allocation-json.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -40,7 +40,7 @@ jq . < $out test $( jq -r '.[0].offset' < $out ) -eq 0 test $( jq -r '.[0].length' < $out ) -eq 32768 test $( jq -r '.[0].type' < $out ) -eq 0 -test $( jq -r '.[0].description' < $out ) = "allocated" +test $( jq -r '.[0].description' < $out ) = "data"
test $( jq -r '.[3].offset' < $out ) -eq 163840 test $( jq -r '.[3].length' < $out ) -eq 884736 diff --git a/info/info-map-base-allocation-large.sh b/info/info-map-base-allocation-large.sh index e7ff904..ab10dc7 100755 --- a/info/info-map-base-allocation-large.sh +++ b/info/info-map-base-allocation-large.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -36,11 +36,11 @@ nbdkit -U - data data='1 @131072 2 @6442450944 3' size=8G \
cat $out
-if [ "$(tr -s ' ' < $out)" != " 0 32768 0 allocated +if [ "$(tr -s ' ' < $out)" != " 0 32768 0 data 32768 98304 3 hole,zero - 131072 32768 0 allocated + 131072 32768 0 data 163840 6442287104 3 hole,zero -6442450944 32768 0 allocated +6442450944 32768 0 data 6442483712 2147450880 3 hole,zero" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-base-allocation-weird.sh b/info/info-map-base-allocation-weird.sh index 20ddec6..0b0343a 100755 --- a/info/info-map-base-allocation-weird.sh +++ b/info/info-map-base-allocation-weird.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -45,7 +45,7 @@ EOF cat $out
if [ "$(tr -s ' ' < $out)" != " 0 16 1 hole - 16 4 0 allocated + 16 4 0 data 20 12 1 hole" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-base-allocation.sh b/info/info-map-base-allocation.sh index 9b85c97..76c2024 100755 --- a/info/info-map-base-allocation.sh +++ b/info/info-map-base-allocation.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -36,9 +36,9 @@ nbdkit -U - data data='1 @131072 2' size=1M \
cat $out
-if [ "$(tr -s ' ' < $out)" != " 0 32768 0 allocated +if [ "$(tr -s ' ' < $out)" != " 0 32768 0 data 32768 98304 3 hole,zero - 131072 32768 0 allocated + 131072 32768 0 data 163840 884736 3 hole,zero" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 diff --git a/info/info-map-qemu-allocation-depth.sh b/info/info-map-qemu-allocation-depth.sh index a5dab0b..9aeffd1 100755 --- a/info/info-map-qemu-allocation-depth.sh +++ b/info/info-map-qemu-allocation-depth.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # nbd client library in userspace -# Copyright (C) 2020 Red Hat Inc. +# Copyright (C) 2020-2021 Red Hat Inc. # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public @@ -84,7 +84,7 @@ if [ "$(tr -s ' ' < $out)" != "\ 3145728 2097152 3 backing depth 3 5242880 1048576 1 local 6291456 1048576 2 backing depth 2 - 7340032 1048576 0 unallocated" ]; then + 7340032 1048576 0 absent" ]; then echo "$0: unexpected output from nbdinfo --map" exit 1 fi diff --git a/info/map.c b/info/map.c index ae6d4fe..82c9507 100644 --- a/info/map.c +++ b/info/map.c @@ -202,7 +202,7 @@ extent_description (const char *metacontext, uint32_t type)
if (strcmp (metacontext, "base:allocation") == 0) { switch (type) { - case 0: return strdup ("allocated"); + case 0: return strdup ("data"); case 1: return strdup ("hole"); case 2: return strdup ("zero"); case 3: return strdup ("hole,zero"); @@ -216,7 +216,7 @@ extent_description (const char *metacontext, uint32_t type) } else if (strcmp (metacontext, "qemu:allocation-depth") == 0) { switch (type) { - case 0: return strdup ("unallocated"); + case 0: return strdup ("absent"); case 1: return strdup ("local"); default: if (asprintf (&ret, "backing depth %u", type) == -1) { -- 2.31.1
Looks better, but can break scripts depending on nbdinfo output text. I hope that nbdino is new enough that this is not a problem. Nir
I think you need to update the nbdinfo.pod file as well, since there are some references there. As Nir says, it's unfortunate, but I guess we've got to do it. If you push it (with docs fixed) soon then I can put it into the libnbd 1.8.1 stable release. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Thu, Jun 10, 2021 at 06:20:31PM +0100, Richard W.M. Jones wrote:
I think you need to update the nbdinfo.pod file as well, since there are some references there.
Yep, thanks for catching.
As Nir says, it's unfortunate, but I guess we've got to do it. If you push it (with docs fixed) soon then I can put it into the libnbd 1.8.1 stable release.
Pushed now. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
participants (3)
-
Eric Blake -
Nir Soffer -
Richard W.M. Jones