[PATCH libnbd] info: Add --map --totals sub-mode to display summary
This adds an nbdinfo sub-option --map --totals to display the total of each type of mapping: $ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals $uri' 1226113024 19.0 0 data 5216337920 81.0 3 hole,zero I need this kind of functionality for virt-v2v in order to estimate how much space is required on the target, although for virt-v2v I will probably use the API directly. Nevertheless it seems like a useful feature and the implementation does not complicate nbdinfo. It can also be used for other map types, eg. for showing how much of the dirty-bitmap is dirty. The implementation could be done in a single pass although for the vast majority of use cases I doubt that would be any faster since the table easily fits in L1 cache, and it would require a complex data structure. But it's worth noting that a malicious server could try to make nbdinfo use O(n^2) time (with a very small constant). Rich.
This is similar to "qemu-img measure". Some examples: $ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals $uri' 1226113024 19.0 0 data 5216337920 81.0 3 hole,zero $ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals --json $uri | jq' [ { "size": 1226113024, "percent": 19.0318, "type": 0, "description": "data" }, { "size": 5216337920, "percent": 80.9682, "type": 3, "description": "hole,zero" } ] $ nbdkit sparse-random 6G --run 'nbdinfo --map --totals $uri' 941551616 14.6 0 data 5500899328 85.4 3 hole,zero --- info/Makefile.am | 2 + info/info-map-totals-json.sh | 48 ++++++++++++++++++++++++ info/info-map-totals.sh | 43 +++++++++++++++++++++ info/main.c | 15 +++++++- info/map.c | 73 +++++++++++++++++++++++++++++++++++- info/nbdinfo.h | 1 + info/nbdinfo.pod | 38 ++++++++++++++++++- 7 files changed, 216 insertions(+), 4 deletions(-) diff --git a/info/Makefile.am b/info/Makefile.am index 5c717c7..75c6a75 100644 --- a/info/Makefile.am +++ b/info/Makefile.am @@ -38,6 +38,8 @@ info_sh_files = \ info-map-base-allocation-zero.sh \ info-map-qemu-dirty-bitmap.sh \ info-map-qemu-allocation-depth.sh \ + info-map-totals.sh \ + info-map-totals-json.sh \ info-atomic-output.sh \ $(NULL) diff --git a/info/info-map-totals-json.sh b/info/info-map-totals-json.sh new file mode 100755 index 0000000..dc386ef --- /dev/null +++ b/info/info-map-totals-json.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright (C) 2020-2021 Red Hat Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +. ../tests/functions.sh + +set -e +set -x + +requires nbdkit --version +requires nbdkit -U - null --run 'test "$uri" != ""' +requires jq --version + +out=info-map-totals-json.out +cleanup_fn rm -f $out +rm -f $out + +# The sparse allocator used by nbdkit-data-plugin uses a 32K page +# size, and extents are always aligned with this. +nbdkit -U - data data='1 @131072 2' size=1M \ + --run '$VG nbdinfo --map --totals --json "$uri"' > $out + +cat $out +jq . < $out + +test $( jq -r '.[0].size' < $out ) -eq 65536 +test $( jq -r '.[0].percent' < $out ) = "6.25" +test $( jq -r '.[0].type' < $out ) -eq 0 +test $( jq -r '.[0].description' < $out ) = "data" + +test $( jq -r '.[1].size' < $out ) -eq 983040 +test $( jq -r '.[1].percent' < $out ) = "93.75" +test $( jq -r '.[1].type' < $out ) -eq 3 +test $( jq -r '.[1].description' < $out ) = "hole,zero" diff --git a/info/info-map-totals.sh b/info/info-map-totals.sh new file mode 100755 index 0000000..12c1263 --- /dev/null +++ b/info/info-map-totals.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright (C) 2020-2021 Red Hat Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +. ../tests/functions.sh + +set -e +set -x + +requires nbdkit --version +requires nbdkit -U - null --run 'test "$uri" != ""' +requires tr --version + +out=info-map-totals.out +cleanup_fn rm -f $out +rm -f $out + +# The sparse allocator used by nbdkit-data-plugin uses a 32K page +# size, and extents are always aligned with this. +nbdkit -U - data data='1 @131072 2' size=1M \ + --run '$VG nbdinfo --map --totals "$uri"' > $out + +cat $out + +if [ "$(tr -s ' ' < $out)" != " 65536 6.2 0 data + 983040 93.8 3 hole,zero" ]; then + echo "$0: unexpected output from nbdinfo --map" + exit 1 +fi diff --git a/info/main.c b/info/main.c index 15e504c..614b924 100644 --- a/info/main.c +++ b/info/main.c @@ -42,6 +42,7 @@ bool probe_content = false; /* --content / --no-content option */ bool json_output = false; /* --json option */ const char *map = NULL; /* --map option */ bool size_only = false; /* --size option */ +bool totals = false; /* --totals option */ static void __attribute__((noreturn)) usage (FILE *fp, int exitcode) @@ -52,7 +53,7 @@ usage (FILE *fp, int exitcode) "\n" " nbdinfo [--json] NBD-URI\n" " nbdinfo --size [--json] NBD-URI\n" -" nbdinfo --map [--json] NBD-URI\n" +" nbdinfo --map [--totals] [--json] NBD-URI\n" " nbdinfo -L|--list [--json] NBD-URI\n" "\n" "Other options:\n" @@ -87,6 +88,7 @@ main (int argc, char *argv[]) JSON_OPTION, MAP_OPTION, SIZE_OPTION, + TOTALS_OPTION, }; const char *short_options = "LV"; const struct option long_options[] = { @@ -99,6 +101,8 @@ main (int argc, char *argv[]) { "map", optional_argument, NULL, MAP_OPTION }, { "short-options", no_argument, NULL, SHORT_OPTIONS }, { "size", no_argument, NULL, SIZE_OPTION }, + { "total", no_argument, NULL, TOTALS_OPTION }, + { "totals", no_argument, NULL, TOTALS_OPTION }, { "version", no_argument, NULL, 'V' }, { NULL } }; @@ -155,6 +159,10 @@ main (int argc, char *argv[]) size_only = true; break; + case TOTALS_OPTION: + totals = true; + break; + case 'L': list_all = true; break; @@ -184,6 +192,11 @@ main (int argc, char *argv[]) progname, "--content", "--no-content"); exit (EXIT_FAILURE); } + if (totals && !map) { + fprintf (stderr, "%s: you must use --totals only with --map option.\n", + progname); + exit (EXIT_FAILURE); + } /* Work out if we should probe content. */ probe_content = !list_all; diff --git a/info/map.c b/info/map.c index 82c9507..9d41e2a 100644 --- a/info/map.c +++ b/info/map.c @@ -38,6 +38,7 @@ DEFINE_VECTOR_TYPE (uint32_vector, uint32_t) static void print_extents (uint32_vector *entries); +static void print_totals (uint32_vector *entries, int64_t size); static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, uint32_t *entries, size_t nr_entries, @@ -89,7 +90,10 @@ do_map (void) offset += entries.ptr[i]; } - print_extents (&entries); + if (!totals) + print_extents (&entries); + else + print_totals (&entries, size); free (entries.ptr); } @@ -195,6 +199,73 @@ print_one_extent (uint64_t offset, uint64_t len, uint32_t type) free (descr); } +/* --map --totals suboption */ +static void +print_totals (uint32_vector *entries, int64_t size) +{ + uint32_t type; + bool comma = false; + + if (json_output) fprintf (fp, "[\n"); + + /* In the outer loop assume we have already printed all entries with + * entry type < type. Count all instances of type and at the same + * time find the next type that exists > type. + */ + type = 0; + for (;;) { + uint64_t next_type = (uint64_t)UINT32_MAX + 1; + uint64_t c = 0; + size_t i; + + for (i = 0; i < entries->size; i += 2) { + uint32_t t = entries->ptr[i+1]; + + if (t == type) + c += entries->ptr[i]; + else if (type < t && t < next_type) + next_type = t; + } + + if (c > 0) { + char *descr = extent_description (map, type); + double percent = 100.0 * c / size; + + if (!json_output) { + fprintf (fp, "%10" PRIu64 " %5.1f %3" PRIu32, + c, percent, type); + if (descr) + fprintf (fp, " %s", descr); + fprintf (fp, "\n"); + } + else { + if (comma) + fprintf (fp, ",\n"); + + fprintf (fp, + "{ \"size\": %" PRIu64 ", " + "\"percent\": %g, " + "\"type\": %" PRIu32, + c, percent, type); + if (descr) { + fprintf (fp, ", \"description\": "); + print_json_string (descr); + } + fprintf (fp, " }"); + comma = true; + } + + free (descr); + } + + if (next_type == (uint64_t)UINT32_MAX + 1) + break; + type = next_type; + } + + if (json_output) fprintf (fp, "\n]\n"); +} + static char * extent_description (const char *metacontext, uint32_t type) { diff --git a/info/nbdinfo.h b/info/nbdinfo.h index ff13e37..6ff73af 100644 --- a/info/nbdinfo.h +++ b/info/nbdinfo.h @@ -32,6 +32,7 @@ extern bool probe_content; extern bool json_output; extern const char *map; extern bool size_only; +extern bool totals; /* list.c */ extern void collect_exports (void); diff --git a/info/nbdinfo.pod b/info/nbdinfo.pod index f1344d4..0c03bcd 100644 --- a/info/nbdinfo.pod +++ b/info/nbdinfo.pod @@ -8,7 +8,7 @@ nbdinfo - display information and metadata about NBD servers and exports nbdinfo --size [--json] NBD-URI - nbdinfo --map [--json] NBD-URI + nbdinfo --map [--totals] [--json] NBD-URI nbdinfo -L|--list [--json] NBD-URI @@ -119,6 +119,35 @@ other maps too: For more information on NBD maps, see I<Metadata querying> in the NBD protocol. +=head2 Map totals + +Using S<I<--map --totals>> performs the same operation as I<--map> but +displays a summary of the total size of each type of allocation, in +bytes and as a percentage (of the virtual size of the export). This +is useful for estimating how much real storage is used on the server, +or might be required when copying a sparse image with L<nbdcopy(1)>. + +In the example below, half (50.0%) of the disk is allocated data and +half is unallocated: + + $ nbdinfo --map --totals nbd://localhost/ + 1048576 50.0 0 data + 1048576 50.0 3 hole,zero + +The fields are: total size in bytes, percentage of the virtual size, +type, description (optional). + +You can also get the same information in parseable form using I<--json>: + + $ nbdinfo --map --totals --json nbd://localhost/ + [{ "size": 1048576, "percent": 50, + "type": 0, "description": "data" }, + { "size": 1048576, "percent": 50, + "type": 3, "description": "hole,zero" }] + +As with the I<--map> option, by default this shows the +C<"base:allocation"> map, but you can show the summary for other maps. + =head2 List all exports To list all the exports available on an NBD server use the I<--list> @@ -179,7 +208,7 @@ The output is displayed in JSON format. Display the map (usually whether parts of the disk are allocated or sparse) of the given export. This displays the C<"base:allocation"> map by default, you can choose a different map with the optional -parameter. +parameter. Using S<I<--map --totals>> displays a summary. =item B<-L> @@ -188,6 +217,11 @@ parameter. List all the exports on an NBD server. The export name in the NBD URI is ignored. +=item B<--totals> + +Use S<I<--map --totals>> to display a summary. See L</Map totals> +above. + =item B<--size> Display only the size in bytes of the export. -- 2.32.0
On Sat, Jun 26, 2021 at 05:19:47PM +0100, Richard W.M. Jones wrote:
This is similar to "qemu-img measure". Some examples:
$ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals $uri' 1226113024 19.0 0 data 5216337920 81.0 3 hole,zero
Is it worth a human-legible header line? Or maybe a % character in the second column?
$ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals --json $uri | jq' [ { "size": 1226113024, "percent": 19.0318, "type": 0, "description": "data" }, { "size": 5216337920, "percent": 80.9682, "type": 3, "description": "hole,zero" } ]
Output looks nice.
$ nbdkit sparse-random 6G --run 'nbdinfo --map --totals $uri' 941551616 14.6 0 data 5500899328 85.4 3 hole,zero ---
...
+++ b/info/info-map-totals-json.sh + +test $( jq -r '.[0].size' < $out ) -eq 65536 +test $( jq -r '.[0].percent' < $out ) = "6.25" +test $( jq -r '.[0].type' < $out ) -eq 0 +test $( jq -r '.[0].description' < $out ) = "data" + +test $( jq -r '.[1].size' < $out ) -eq 983040 +test $( jq -r '.[1].percent' < $out ) = "93.75" +test $( jq -r '.[1].type' < $out ) -eq 3 +test $( jq -r '.[1].description' < $out ) = "hole,zero" ... diff --git a/info/info-map-totals.sh b/info/info-map-totals.sh new file mode 100755 index 0000000..12c1263 --- /dev/null +++ b/info/info-map-totals.sh @@ -0,0 +1,43 @@
+ +cat $out + +if [ "$(tr -s ' ' < $out)" != " 65536 6.2 0 data + 983040 93.8 3 hole,zero" ]; then
So this is the human-readable counterpart of info-map-totals-json.sh. I'm trying to figure out why 6.25 rounded down and 93.75 rounded up. I guess it's related to round-to-even? But it's also fine if the rounded values don't sum back up to 100%; I don't know if it's worth a documentation disclaimer.
+++ b/info/map.c
+/* --map --totals suboption */ +static void +print_totals (uint32_vector *entries, int64_t size) +{ + uint32_t type; + bool comma = false; + + if (json_output) fprintf (fp, "[\n"); + + /* In the outer loop assume we have already printed all entries with + * entry type < type. Count all instances of type and at the same + * time find the next type that exists > type. + */ + type = 0; + for (;;) { + uint64_t next_type = (uint64_t)UINT32_MAX + 1; + uint64_t c = 0; + size_t i; + + for (i = 0; i < entries->size; i += 2) { + uint32_t t = entries->ptr[i+1]; + + if (t == type) + c += entries->ptr[i]; + else if (type < t && t < next_type) + next_type = t; + }
I see what you mean about a malicious server being able to force us to do a LOT of work, but I think this is fine for a first cut (we can worry about rewriting it to do a single pass utilizing a hash table if it proves to be noticeable in practice).
+++ b/info/nbdinfo.pod @@ -8,7 +8,7 @@ nbdinfo - display information and metadata about NBD servers and exports
nbdinfo --size [--json] NBD-URI
- nbdinfo --map [--json] NBD-URI + nbdinfo --map [--totals] [--json] NBD-URI
nbdinfo -L|--list [--json] NBD-URI
@@ -119,6 +119,35 @@ other maps too: For more information on NBD maps, see I<Metadata querying> in the NBD protocol.
+=head2 Map totals + +Using S<I<--map --totals>> performs the same operation as I<--map> but +displays a summary of the total size of each type of allocation, in +bytes and as a percentage (of the virtual size of the export). This +is useful for estimating how much real storage is used on the server, +or might be required when copying a sparse image with L<nbdcopy(1)>. + +In the example below, half (50.0%) of the disk is allocated data and +half is unallocated: + + $ nbdinfo --map --totals nbd://localhost/ + 1048576 50.0 0 data + 1048576 50.0 3 hole,zero + +The fields are: total size in bytes, percentage of the virtual size, +type, description (optional). + +You can also get the same information in parseable form using I<--json>: + + $ nbdinfo --map --totals --json nbd://localhost/ + [{ "size": 1048576, "percent": 50, + "type": 0, "description": "data" }, + { "size": 1048576, "percent": 50, + "type": 3, "description": "hole,zero" }] + +As with the I<--map> option, by default this shows the +C<"base:allocation"> map, but you can show the summary for other maps.
I'd still like to someday have a mode where we can read all advertised maps at once, rather than making the command-line user have to manually invoke nbdinfo once per map spelled by the user, but that can still go on top of this. Overall, looks good to me! -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Mon, Jun 28, 2021 at 12:05:03PM -0500, Eric Blake wrote:
On Sat, Jun 26, 2021 at 05:19:47PM +0100, Richard W.M. Jones wrote:
This is similar to "qemu-img measure". Some examples:
$ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals $uri' 1226113024 19.0 0 data 5216337920 81.0 3 hole,zero
Is it worth a human-legible header line? Or maybe a % character in the second column?
We don't have headers on the other table and it may be too late to add those now. But I think you're right it would be better to add a % character after the percent field (ie. "%5.1f%%"). I think I'll make this change.
+++ b/info/info-map-totals-json.sh + +test $( jq -r '.[0].size' < $out ) -eq 65536 +test $( jq -r '.[0].percent' < $out ) = "6.25" +test $( jq -r '.[0].type' < $out ) -eq 0 +test $( jq -r '.[0].description' < $out ) = "data" + +test $( jq -r '.[1].size' < $out ) -eq 983040 +test $( jq -r '.[1].percent' < $out ) = "93.75" +test $( jq -r '.[1].type' < $out ) -eq 3 +test $( jq -r '.[1].description' < $out ) = "hole,zero" ... diff --git a/info/info-map-totals.sh b/info/info-map-totals.sh new file mode 100755 index 0000000..12c1263 --- /dev/null +++ b/info/info-map-totals.sh @@ -0,0 +1,43 @@
+ +cat $out + +if [ "$(tr -s ' ' < $out)" != " 65536 6.2 0 data + 983040 93.8 3 hole,zero" ]; then
So this is the human-readable counterpart of info-map-totals-json.sh. I'm trying to figure out why 6.25 rounded down and 93.75 rounded up. I guess it's related to round-to-even? But it's also fine if the rounded values don't sum back up to 100%; I don't know if it's worth a documentation disclaimer.
I don't know either.
+ $ nbdinfo --map --totals --json nbd://localhost/ + [{ "size": 1048576, "percent": 50, + "type": 0, "description": "data" }, + { "size": 1048576, "percent": 50, + "type": 3, "description": "hole,zero" }] + +As with the I<--map> option, by default this shows the +C<"base:allocation"> map, but you can show the summary for other maps.
I'd still like to someday have a mode where we can read all advertised maps at once, rather than making the command-line user have to manually invoke nbdinfo once per map spelled by the user, but that can still go on top of this.
Yes it should be possible using nbd_opt_list_meta_context.
Overall, looks good to me!
Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
participants (2)
-
Eric Blake -
Richard W.M. Jones