May 2020 - Libguestfs - libguestfs List Archives

[PATCH nbdkit] ddrescue: Miscellaneous fixes.
by Richard W.M. Jones 22 May '20

22 May '20

A few fixes and a possible enhancement to the ddrescue filter. If you think these are all OK, I will squash it into your patch and push it. Rich.

3 5

[PATCH] [v3] ddrescue mapfile filter
by François Revol 22 May '20

22 May '20

This allows to overlay bad sectors according to the mapfile generated by ddrescue, to then see where sectors are used using fsck and trying to copy files around. Should now be mature enough to be useful to someone else. And we have a proper test also. Signed-off-by: François Revol <revol(a)free.fr> --- configure.ac | 2 + filters/ddrescue/Makefile.am | 75 +++++++ filters/ddrescue/ddrescue.c | 212 ++++++++++++++++++++ filters/ddrescue/nbdkit-ddrescue-filter.pod | 74 +++++++ tests/Makefile.am | 2 + tests/test-ddrescue-filter.sh | 83 ++++++++ 6 files changed, 448 insertions(+) create mode 100644 filters/ddrescue/Makefile.am create mode 100644 filters/ddrescue/ddrescue.c create mode 100644 filters/ddrescue/nbdkit-ddrescue-filter.pod create mode 100755 tests/test-ddrescue-filter.sh diff --git a/configure.ac b/configure.ac index 2a70d90a..2727c20f 100644 --- a/configure.ac +++ b/configure.ac @@ -97,6 +97,7 @@ filters="\ cache \ cacheextents \ cow \ + ddrescue \ delay \ error \ exitlast \ @@ -1089,6 +1090,7 @@ AC_CONFIG_FILES([Makefile filters/cache/Makefile filters/cacheextents/Makefile filters/cow/Makefile + filters/ddrescue/Makefile filters/delay/Makefile filters/error/Makefile filters/exitlast/Makefile diff --git a/filters/ddrescue/Makefile.am b/filters/ddrescue/Makefile.am new file mode 100644 index 00000000..66b9d238 --- /dev/null +++ b/filters/ddrescue/Makefile.am @@ -0,0 +1,75 @@ +# nbdkit +# Copyright (C) 2018 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +include $(top_srcdir)/common-rules.mk + +EXTRA_DIST = \ + nbdkit-ddrescue-filter.pod \ + $(NULL) + +filter_LTLIBRARIES = nbdkit-ddrescue-filter.la + +nbdkit_ddrescue_filter_la_SOURCES = \ + ddrescue.c \ + $(top_srcdir)/include/nbdkit-filter.h \ + $(NULL) + +nbdkit_ddrescue_filter_la_CPPFLAGS = \ + -I$(top_srcdir)/include \ + -I$(top_srcdir)/common/include \ + -I$(top_srcdir)/common/sparse \ + -I$(top_srcdir)/common/utils \ + $(NULL) +nbdkit_ddrescue_filter_la_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(GNUTLS_CFLAGS) \ + $(NULL) +nbdkit_ddrescue_filter_la_LDFLAGS = \ + -module -avoid-version -shared $(SHARED_LDFLAGS) \ + -Wl,--version-script=$(top_srcdir)/filters/filters.syms \ + $(NULL) +nbdkit_ddrescue_filter_la_LIBADD = \ + $(top_builddir)/common/sparse/libsparse.la \ + $(top_builddir)/common/utils/libutils.la \ + $(GNUTLS_LIBS) \ + $(NULL) + +if HAVE_POD + +man_MANS = nbdkit-ddrescue-filter.1 +CLEANFILES += $(man_MANS) + +nbdkit-ddrescue-filter.1: nbdkit-ddrescue-filter.pod + $(PODWRAPPER) --section=1 --man $@ \ + --html $(top_builddir)/html/$@.html \ + $< + +endif HAVE_POD diff --git a/filters/ddrescue/ddrescue.c b/filters/ddrescue/ddrescue.c new file mode 100644 index 00000000..e4b51de4 --- /dev/null +++ b/filters/ddrescue/ddrescue.c @@ -0,0 +1,212 @@ +/* nbdkit + * Copyright (C) 2018-2020 François Revol. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> + +#include <pthread.h> + +#include <nbdkit-filter.h> + +#include "cleanup.h" + +struct range { + int64_t start; + int64_t end; + int64_t size; + char status; +}; + +struct mapfile { + int ranges_count; + struct range *ranges; +}; + +static struct mapfile map = { 0, NULL }; + +static int +parse_mapfile (const char *filename) +{ + FILE *fp = NULL; + CLEANUP_FREE char *line = NULL; + size_t linelen = 0; + ssize_t len; + int ret = -1; + int status_seen = 0; + + fp = fopen (filename, "r"); + if (!fp) { + nbdkit_error ("%s: ddrescue: fopen: %m", filename); + goto out; + } + + while ((len = getline (&line, &linelen, fp)) != -1) { + const char *delim = " \t"; + char *sp, *p; + int64_t offset, length; + char status; + + if (len > 0 && line[len-1] == '\n') { + line[len-1] = '\0'; + len--; + } + + if (len > 0 && line[0] == '#') + continue; + + if (len > 0 && !status_seen) { + /* status line, ignore it for now */ + status_seen = 1; + nbdkit_debug ("%s: skipping status line: '%s'", filename, line); + continue; + } + + if (sscanf (line, "%" SCNi64 "\t%" SCNi64 "\t%c", &offset, &length, &status) == 3) { + if (offset < 0) { + nbdkit_error ("block offset must not be negative"); + return -1; + } + if (length < 0) { + nbdkit_error ("block length must not be negative"); + return -1; + } + if (status == '+') { + int i = map.ranges_count++; + map.ranges = realloc(map.ranges, map.ranges_count * sizeof(struct range)); + if (map.ranges == NULL) { + nbdkit_error ("%s: ddrescue: realloc: %m", filename); + goto out; + } + map.ranges[i].start = offset; + map.ranges[i].end = offset + length - 1; + map.ranges[i].size = length; + map.ranges[i].status = status; + } + + nbdkit_debug ("%s: range: 0x%" PRIx64 " 0x%" PRIx64 " '%c'", filename, offset, length, status); + } + } + + ret = 0; + + out: + if (fp) + fclose (fp); + return ret; +} + +/* On unload, free the mapfile data. */ +static void +ddrescue_unload (void) +{ + free (map.ranges); + map.ranges = NULL; + map.ranges_count = 0; +} + +static int +ddrescue_config (nbdkit_next_config *next, void *nxdata, + const char *key, const char *value) +{ + if (strcmp (key, "ddrescue-mapfile") == 0) { + if (parse_mapfile (value) == -1) + return -1; + return 0; + } + + else + return next (nxdata, key, value); +} + +#define ddrescue_config_help \ + "ddrescue-mapfile=... Specify ddrescue mapfile to use" + +/* We need this because otherwise the layer below can_write is called + * and that might return true (eg. if the plugin has a pwrite method + * at all), resulting in writes being passed through to the layer + * below. + */ +static int +ddrescue_can_write (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle) +{ + return 0; +} + +static int +ddrescue_can_cache (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle) +{ + return 0; +} + +/* Read data. */ +static int +ddrescue_pread (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle, void *buf, uint32_t count, uint64_t offset, + uint32_t flags, int *err) +{ + int i; + + for (i = 0; i < map.ranges_count; i++) { + if (map.ranges[i].status != '+') + continue; + if (offset >= map.ranges[i].start && offset <= map.ranges[i].end) { + if (offset + count - 1 <= map.ranges[i].end) { + /* entirely contained within this range */ + return next_ops->pread (nxdata, buf, count, offset, flags, err); + } + } + } + /* read was not fully covered */ + nbdkit_debug ("ddrescue: pread: range: 0x%" PRIx64 " 0x%" PRIx32 " failing with EIO", offset, count); + *err = EIO; + return -1; +} + +static struct nbdkit_filter filter = { + .name = "ddrescue", + .longname = "nbdkit ddrescue mapfile filter", + .unload = ddrescue_unload, + .config = ddrescue_config, + .config_help = ddrescue_config_help, + .can_write = ddrescue_can_write, + .can_cache = ddrescue_can_cache, + .pread = ddrescue_pread, +}; + +NBDKIT_REGISTER_FILTER(filter) diff --git a/filters/ddrescue/nbdkit-ddrescue-filter.pod b/filters/ddrescue/nbdkit-ddrescue-filter.pod new file mode 100644 index 00000000..8210866b --- /dev/null +++ b/filters/ddrescue/nbdkit-ddrescue-filter.pod @@ -0,0 +1,74 @@ +=head1 NAME + +nbdkit-ddrescue-filter - nbdkit filter for serving from ddrescue dump + +=head1 SYNOPSIS + + nbdkit --filter=ddrescue plugin [plugin-args...] ddrescue-mapfile=file.map + + nbdkit --filter=ddrescue file file=file.img ddrescue-mapfile=file.map [plugin-args...] + +=head1 DESCRIPTION + +C<nbdkit-ddrescue-filter> is a filter for L<nbdkit(1)> which overlays +bad blocks according to a GNU L<ddrescue(1)> mapfile. This is mainly useful +for testing disk images recovered with ddrescue, to detect which files +or filesystem structures are impacted, or attempting fsck on them. + +Note that the current implementation is read-only. + +=head1 EXAMPLES + +=over 4 + +=item Expose a rescued disk image with detected bad sectors: + + nbdkit --filter=ddrescue file file=disk.img ddrescue-mapfile=disk.map + +The above command serves the disk image disk.img and maps the bad +sectors listed in disk.img so that read attempts on them do not return +a valid block full of zeroes. + +=back + +=head1 PARAMETERS + +The C<ddrescue-mapfile> parameter must point to a valid GNU ddrescue +mapfile. + +=head1 DATA FORMAT + +The file pointed to by the C<ddrescue-mapfile> parameter should +conform to the format of a GNU L<ddrescue(1)> mapfile. + +=head1 FILES + +=over 4 + +=item F<$filterdir/nbdkit-ddrescue-filter.so> + +The filter. + +Use C<nbdkit --dump-config> to find the location of C<$filterdir>. + +=back + +=head1 VERSION + +C<nbdkit-ddrescue-filter> first appeared in nbdkit 1.22. + +=head1 SEE ALSO + +L<nbdkit(1)>, +L<nbdkit-file-plugin(1)>, +L<nbdkit-filter(3)>, +L<ddrescue(1)>, +L<https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html>. + +=head1 AUTHORS + +François Revol + +=head1 COPYRIGHT + +Copyright (C) 2020 François Revol diff --git a/tests/Makefile.am b/tests/Makefile.am index dbc6aa66..747e8e4e 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -131,6 +131,7 @@ TESTS += \ test-dump-plugin-name.sh \ test-dump-plugin-and-single.sh \ test-dump-plugin-thread-model.sh \ + test-ddrescue-filter.sh \ test-probe-filter.sh \ test-probe-plugin.sh \ test-start.sh \ @@ -155,6 +156,7 @@ TESTS += \ $(NULL) EXTRA_DIST += \ test-captive.sh \ + test-ddrescue-filter.sh \ test-debug-flags.sh \ test-dump-plugin-and-single.sh \ test-dump-plugin-example1.sh \ diff --git a/tests/test-ddrescue-filter.sh b/tests/test-ddrescue-filter.sh new file mode 100755 index 00000000..f08bbd4e --- /dev/null +++ b/tests/test-ddrescue-filter.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 François Revol. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +# Test the ip filter. Necessarily this is rather limited because we +# are only able to use the loopback connection. + +source ./functions.sh +set -e +set -x + +requires nbdsh --version + +sock=`mktemp -u` +files="data-file.pid $sock data-ddrescue.txt ddrescue-test1.map" +rm -f $files +cleanup_fn rm -f $files + +rm -f data-ddrescue.txt +for i in {0..1000}; do + printf "ddrescue" >> data-ddrescue.txt +done + +echo "# +# current_pos current_status current_pass +0x00000000 + 1 +# pos size status +0x00000000 0x00000200 + +0x00000200 0x00000200 - +0x00000400 0x00000200 +" > ddrescue-test1.map + + +# Run nbdkit. +start_nbdkit -P data-file.pid -U $sock \ + --filter=ddrescue data \ + ddrescue-mapfile="ddrescue-test1.map"\ + size=1M \ + data=" + @0x000 <data-ddrescue.txt + " + +nbdsh --connect "nbd+unix://?socket=$sock" \ + -c ' +buf = h.pread (512, 0) +assert buf == b"ddrescue" * 64 +try: + h.pread (512, 512) + # This should not happen. + exit (1) +except nbd.Error as ex: + # Check the errno is expected. + assert ex.errno == "EIO" +buf = h.pread (512, 2 * 512) +assert buf == b"ddrescue" * 64 +' -- 2.27.0.rc0

1 0

[PATCH] [v2] WIP: ddrescue mapfile filter
by François Revol 22 May '20

22 May '20

This allows to overlay bad sectors according to the mapfile generated by ddrescue, to then see where sectors are used using fsck and trying to copy files around. Signed-off-by: François Revol <revol(a)free.fr> --- configure.ac | 2 + filters/ddrescue/Makefile.am | 75 +++++++ filters/ddrescue/ddrescue.c | 211 ++++++++++++++++++++ filters/ddrescue/nbdkit-ddrescue-filter.pod | 74 +++++++ 4 files changed, 362 insertions(+) create mode 100644 filters/ddrescue/Makefile.am create mode 100644 filters/ddrescue/ddrescue.c create mode 100644 filters/ddrescue/nbdkit-ddrescue-filter.pod diff --git a/configure.ac b/configure.ac index 6c25226e..21e1013f 100644 --- a/configure.ac +++ b/configure.ac @@ -97,6 +97,7 @@ filters="\ cache \ cacheextents \ cow \ + ddrescue \ delay \ error \ exitlast \ @@ -1089,6 +1090,7 @@ AC_CONFIG_FILES([Makefile filters/cache/Makefile filters/cacheextents/Makefile filters/cow/Makefile + filters/ddrescue/Makefile filters/delay/Makefile filters/error/Makefile filters/exitlast/Makefile diff --git a/filters/ddrescue/Makefile.am b/filters/ddrescue/Makefile.am new file mode 100644 index 00000000..2498074c --- /dev/null +++ b/filters/ddrescue/Makefile.am @@ -0,0 +1,75 @@ +# nbdkit +# Copyright (C) 2018 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +include $(top_srcdir)/common-rules.mk + +EXTRA_DIST = \ + nbdkit-ddrescue-filter.pod \ + $(NULL) + +filter_LTLIBRARIES = nbdkit-ddrescue-filter.la + +nbdkit_ddrescue_filter_la_SOURCES = \ + ddrescue.c \ + $(top_srcdir)/include/nbdkit-filter.h \ + $(NULL) + +nbdkit_ddrescue_filter_la_CPPFLAGS = \ + -I$(top_srcdir)/include \ + -I$(top_srcdir)/common/include \ + -I$(top_srcdir)/common/sparse \ + -I$(top_srcdir)/common/utils \ + $(NULL) +nbdkit_ddrescue_filter_la_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(GNUTLS_CFLAGS) \ + $(NULL) +nbdkit_ddrescue_filter_la_LDFLAGS = \ + -module -avoid-version -shared \ + -Wl,--version-script=$(top_srcdir)/filters/filters.syms \ + $(NULL) +nbdkit_ddrescue_filter_la_LIBADD = \ + $(top_builddir)/common/sparse/libsparse.la \ + $(top_builddir)/common/utils/libutils.la \ + $(GNUTLS_LIBS) \ + $(NULL) + +if HAVE_POD + +man_MANS = nbdkit-ddrescue-filter.1 +CLEANFILES += $(man_MANS) + +nbdkit-ddrescue-filter.1: nbdkit-ddrescue-filter.pod + $(PODWRAPPER) --section=1 --man $@ \ + --html $(top_builddir)/html/$@.html \ + $< + +endif HAVE_POD diff --git a/filters/ddrescue/ddrescue.c b/filters/ddrescue/ddrescue.c new file mode 100644 index 00000000..f6dd9382 --- /dev/null +++ b/filters/ddrescue/ddrescue.c @@ -0,0 +1,211 @@ +/* nbdkit + * Copyright (C) 2018-2020 François Revol. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> + +#include <pthread.h> + +#include <nbdkit-filter.h> + +#include "cleanup.h" + +struct range { + int64_t start; + int64_t end; + int64_t size; + char status; +}; + +struct mapfile { + int ranges_count; + struct range *ranges; +}; + +static struct mapfile map = { 0, NULL }; + +static int +parse_mapfile (const char *filename) +{ + FILE *fp = NULL; + CLEANUP_FREE char *line = NULL; + size_t linelen = 0; + ssize_t len; + int ret = -1; + int status_seen = 0; + + fp = fopen (filename, "r"); + if (!fp) { + nbdkit_error ("%s: ddrescue: fopen: %m", filename); + goto out; + } + + while ((len = getline (&line, &linelen, fp)) != -1) { + const char *delim = " \t"; + char *sp, *p; + int64_t offset, length; + char status; + + if (len > 0 && line[len-1] == '\n') { + line[len-1] = '\0'; + len--; + } + + if (len > 0 && line[0] == '#') + continue; + + if (len > 0 && !status_seen) { + /* status line, ignore it for now */ + status_seen = 1; + nbdkit_debug ("%s: skipping status line: '%s'", filename, line); + continue; + } + + if (sscanf (line, "%" SCNi64 "\t%" SCNi64 "\t%c", &offset, &length, &status) == 3) { + if (offset < 0) { + nbdkit_error ("block offset must not be negative"); + return -1; + } + if (length < 0) { + nbdkit_error ("block length must not be negative"); + return -1; + } + if (status == '+') { + int i = map.ranges_count++; + map.ranges = realloc(map.ranges, map.ranges_count * sizeof(struct range)); + if (map.ranges == NULL) { + nbdkit_error ("%s: ddrescue: realloc: %m", filename); + goto out; + } + map.ranges[i].start = offset; + map.ranges[i].end = offset + length - 1; + map.ranges[i].size = length; + map.ranges[i].status = status; + } + + nbdkit_debug ("%s: range: 0x%" PRIx64 " 0x%" PRIx64 " '%c'", filename, offset, length, status); + } + } + + ret = 0; + + out: + if (fp) + fclose (fp); + return ret; +} + +/* On unload, free the sparse array. */ +static void +ddrescue_unload (void) +{ + free (map.ranges); + map.ranges = NULL; + map.ranges_count = 0; +} + +static int +ddrescue_config (nbdkit_next_config *next, void *nxdata, + const char *key, const char *value) +{ + if (strcmp (key, "ddrescue-mapfile") == 0) { + if (parse_mapfile (value) == -1) + return -1; + return 0; + } + + else + return next (nxdata, key, value); +} + +#define ddrescue_config_help \ + "ddrescue-mapfile=... Specify ddrescue mapfile to use" + +/* We need this because otherwise the layer below can_write is called + * and that might return true (eg. if the plugin has a pwrite method + * at all), resulting in writes being passed through to the layer + * below. + */ +static int +ddrescue_can_write (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle) +{ + return 0; +} + +static int +ddrescue_can_cache (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle) +{ + return 0; +} + +/* Read data. */ +static int +ddrescue_pread (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle, void *buf, uint32_t count, uint64_t offset, + uint32_t flags, int *err) +{ + int i; + + for (i = 0; i < map.ranges_count; i++) { + if (map.ranges[i].status != '+') + continue; + if (offset >= map.ranges[i].start && offset <= map.ranges[i].end) { + if (offset + count - 1 <= map.ranges[i].end) { + /* entirely contained within this range */ + return next_ops->pread (nxdata, buf, count, offset, flags, err); + } + } + } + /* read was not fully covered */ + *err = EIO; + return -1; +} + +static struct nbdkit_filter filter = { + .name = "ddrescue", + .longname = "nbdkit ddrescue mapfile filter", + .unload = ddrescue_unload, + .config = ddrescue_config, + .config_help = ddrescue_config_help, + .can_write = ddrescue_can_write, + .can_cache = ddrescue_can_cache, + .pread = ddrescue_pread, +}; + +NBDKIT_REGISTER_FILTER(filter) diff --git a/filters/ddrescue/nbdkit-ddrescue-filter.pod b/filters/ddrescue/nbdkit-ddrescue-filter.pod new file mode 100644 index 00000000..8210866b --- /dev/null +++ b/filters/ddrescue/nbdkit-ddrescue-filter.pod @@ -0,0 +1,74 @@ +=head1 NAME + +nbdkit-ddrescue-filter - nbdkit filter for serving from ddrescue dump + +=head1 SYNOPSIS + + nbdkit --filter=ddrescue plugin [plugin-args...] ddrescue-mapfile=file.map + + nbdkit --filter=ddrescue file file=file.img ddrescue-mapfile=file.map [plugin-args...] + +=head1 DESCRIPTION + +C<nbdkit-ddrescue-filter> is a filter for L<nbdkit(1)> which overlays +bad blocks according to a GNU L<ddrescue(1)> mapfile. This is mainly useful +for testing disk images recovered with ddrescue, to detect which files +or filesystem structures are impacted, or attempting fsck on them. + +Note that the current implementation is read-only. + +=head1 EXAMPLES + +=over 4 + +=item Expose a rescued disk image with detected bad sectors: + + nbdkit --filter=ddrescue file file=disk.img ddrescue-mapfile=disk.map + +The above command serves the disk image disk.img and maps the bad +sectors listed in disk.img so that read attempts on them do not return +a valid block full of zeroes. + +=back + +=head1 PARAMETERS + +The C<ddrescue-mapfile> parameter must point to a valid GNU ddrescue +mapfile. + +=head1 DATA FORMAT + +The file pointed to by the C<ddrescue-mapfile> parameter should +conform to the format of a GNU L<ddrescue(1)> mapfile. + +=head1 FILES + +=over 4 + +=item F<$filterdir/nbdkit-ddrescue-filter.so> + +The filter. + +Use C<nbdkit --dump-config> to find the location of C<$filterdir>. + +=back + +=head1 VERSION + +C<nbdkit-ddrescue-filter> first appeared in nbdkit 1.22. + +=head1 SEE ALSO + +L<nbdkit(1)>, +L<nbdkit-file-plugin(1)>, +L<nbdkit-filter(3)>, +L<ddrescue(1)>, +L<https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html>. + +=head1 AUTHORS + +François Revol + +=head1 COPYRIGHT + +Copyright (C) 2020 François Revol -- 2.26.2

3 7

[v2v PATCH] libvirt: make use of libvirt's default auth handler (RHBZ#1838425)
by Pino Toscano 21 May '20

21 May '20

Use the default libvirt authentication handler as base for ours, overriding it with our callback only in case we have a password to supply. --- v2v/libvirt_utils.ml | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/v2v/libvirt_utils.ml b/v2v/libvirt_utils.ml index 7df17b29..4d0b8639 100644 --- a/v2v/libvirt_utils.ml +++ b/v2v/libvirt_utils.ml @@ -33,10 +33,14 @@ let auth_for_password_file ?password_file () = ) creds in - { - Libvirt.Connect.credtype = [ Libvirt.Connect.CredentialPassphrase ]; - cb = auth_fn; - } + let base_auth = Libvirt.Connect.get_auth_default () in + + if password_file = None then + base_auth + else + { base_auth with + cb = auth_fn; + } let get_domain conn name = let dom = -- 2.25.4

2 1

[PATCH nbdkit] tests/test-truncate4.sh: Rewrite to use nbdsh.
by Richard W.M. Jones 20 May '20

20 May '20

This commit works for me, but TBH I'm not clear on what the actual purpose of this test is supposed to be. Rich.

2 3

[PATCH nbdkit] common/include: Add locale-safe ascii_strcasecmp and ascii_strncasecmp.
by Richard W.M. Jones 19 May '20

19 May '20

These are derived from the FreeBSD functions here: https://github.com/freebsd/freebsd/blob/master/sys/libkern/strcasecmp.c Thanks: Eric Blake. --- common/include/Makefile.am | 6 +++ common/include/ascii-ctype.h | 6 +++ common/include/ascii-string.h | 77 ++++++++++++++++++++++++++++ common/include/test-ascii-string.c | 79 +++++++++++++++++++++++++++++ server/main.c | 8 +-- server/public.c | 21 ++++---- plugins/curl/curl.c | 7 +-- plugins/info/info.c | 17 ++++--- plugins/nbd/nbd.c | 8 +-- plugins/partitioning/partitioning.c | 10 ++-- plugins/sh/call.c | 25 ++++----- plugins/sh/methods.c | 27 +++++----- filters/ip/ip.c | 13 ++--- .gitignore | 1 + 14 files changed, 243 insertions(+), 62 deletions(-) diff --git a/common/include/Makefile.am b/common/include/Makefile.am index aa515702..dd1f871a 100644 --- a/common/include/Makefile.am +++ b/common/include/Makefile.am @@ -35,6 +35,7 @@ include $(top_srcdir)/common-rules.mk # plugins and/or filters. They are not installed. EXTRA_DIST = \ ascii-ctype.h \ + ascii-string.h \ byte-swapping.h \ exit-with-parent.h \ get-current-dir-name.h \ @@ -52,6 +53,7 @@ EXTRA_DIST = \ TESTS = \ test-ascii-ctype \ + test-ascii-string \ test-byte-swapping \ test-current-dir-name \ test-isaligned \ @@ -68,6 +70,10 @@ test_ascii_ctype_SOURCES = test-ascii-ctype.c ascii-ctype.h test_ascii_ctype_CPPFLAGS = -I$(srcdir) test_ascii_ctype_CFLAGS = $(WARNINGS_CFLAGS) +test_ascii_string_SOURCES = test-ascii-string.c ascii-string.h +test_ascii_string_CPPFLAGS = -I$(srcdir) +test_ascii_string_CFLAGS = $(WARNINGS_CFLAGS) + test_byte_swapping_SOURCES = test-byte-swapping.c byte-swapping.h test_byte_swapping_CPPFLAGS = -I$(srcdir) test_byte_swapping_CFLAGS = $(WARNINGS_CFLAGS) diff --git a/common/include/ascii-ctype.h b/common/include/ascii-ctype.h index 5e8bf237..a700563e 100644 --- a/common/include/ascii-ctype.h +++ b/common/include/ascii-ctype.h @@ -49,6 +49,9 @@ #define ascii_isspace(c) \ ((c) == '\t' || (c) == '\n' || (c) == '\f' || (c) == '\r' || (c) == ' ') +#define ascii_isupper(c) \ + ((c) >= 'A' && (c) <= 'Z') + #define ascii_isxdigit(c) \ ((c) == '0' || (c) == '1' || (c) == '2' || (c) == '3' || (c) == '4' || \ (c) == '5' || (c) == '6' || (c) == '7' || (c) == '8' || (c) == '9' || \ @@ -57,4 +60,7 @@ (c) == 'A' || (c) == 'B' || (c) == 'C' || \ (c) == 'D' || (c) == 'E' || (c) == 'F') +#define ascii_tolower(c) \ + (ascii_isupper ((c)) ? (c) - 'A' + 'a' : (c)) + #endif /* NBDKIT_ASCII_CTYPE_H */ diff --git a/common/include/ascii-string.h b/common/include/ascii-string.h new file mode 100644 index 00000000..0a60d5f4 --- /dev/null +++ b/common/include/ascii-string.h @@ -0,0 +1,77 @@ +/* nbdkit + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Case insensitive string comparison functions (like strcasecmp, + * strncasecmp) which work correctly in any locale. They can only be + * used for comparison when one or both strings is 7 bit ASCII. + */ + +#ifndef NBDKIT_ASCII_STRING_H +#define NBDKIT_ASCII_STRING_H + +#include "ascii-ctype.h" + +static inline int +ascii_strcasecmp (const char *s1, const char *s2) +{ + const unsigned char *us1 = (const unsigned char *)s1; + const unsigned char *us2 = (const unsigned char *)s2; + + while (ascii_tolower (*us1) == ascii_tolower (*us2)) { + if (*us1++ == '\0') + return 0; + us2++; + } + + return ascii_tolower (*us1) - ascii_tolower (*us2); +} + +static inline int +ascii_strncasecmp (const char *s1, const char *s2, size_t n) +{ + if (n != 0) { + const unsigned char *us1 = (const unsigned char *)s1; + const unsigned char *us2 = (const unsigned char *)s2; + + do { + if (ascii_tolower (*us1) != ascii_tolower (*us2)) + return ascii_tolower (*us1) - ascii_tolower (*us2); + if (*us1++ == '\0') + break; + us2++; + } while (--n != 0); + } + + return 0; +} + +#endif /* NBDKIT_ASCII_STRING_H */ diff --git a/common/include/test-ascii-string.c b/common/include/test-ascii-string.c new file mode 100644 index 00000000..0fa4a483 --- /dev/null +++ b/common/include/test-ascii-string.c @@ -0,0 +1,79 @@ +/* nbdkit + * Copyright (C) 2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <assert.h> + +#include "ascii-string.h" + +int +main (void) +{ + assert (ascii_strcasecmp ("", "") == 0); + assert (ascii_strcasecmp ("a", "a") == 0); + assert (ascii_strcasecmp ("abc", "abc") == 0); + assert (ascii_strcasecmp ("a", "b") < 0); + assert (ascii_strcasecmp ("b", "a") > 0); + assert (ascii_strcasecmp ("aa", "a") > 0); + + /* Second string contains Turkish dotless lowercase letter ı. */ + assert (ascii_strcasecmp ("hi", "hı") != 0); + + /* Check that we got our rounding behaviour correct. */ + assert (ascii_strcasecmp ("\x1", "\x7f") < 0); + assert (ascii_strcasecmp ("\x1", "\x80") < 0); + assert (ascii_strcasecmp ("\x1", "\x81") < 0); + assert (ascii_strcasecmp ("\x1", "\xff") < 0); + + assert (ascii_strncasecmp ("", "", 0) == 0); + assert (ascii_strncasecmp ("a", "a", 1) == 0); + assert (ascii_strncasecmp ("abc", "abc", 3) == 0); + assert (ascii_strncasecmp ("abc", "def", 0) == 0); + assert (ascii_strncasecmp ("abc", "abd", 2) == 0); + assert (ascii_strncasecmp ("a", "b", 1) < 0); + assert (ascii_strncasecmp ("b", "a", 1) > 0); + assert (ascii_strncasecmp ("aa", "a", 2) > 0); + assert (ascii_strncasecmp ("aa", "a", 100) > 0); + + assert (ascii_strncasecmp ("hi", "hı", 1) == 0); + assert (ascii_strncasecmp ("hi", "hı", 2) != 0); + + assert (ascii_strncasecmp ("\x1", "\x7f", 1) < 0); + assert (ascii_strncasecmp ("\x1", "\x80", 1) < 0); + assert (ascii_strncasecmp ("\x1", "\x81", 1) < 0); + assert (ascii_strncasecmp ("\x1", "\xff", 1) < 0); + + exit (EXIT_SUCCESS); +} diff --git a/server/main.c b/server/main.c index 9dd9d400..feff57a3 100644 --- a/server/main.c +++ b/server/main.c @@ -59,6 +59,8 @@ #include <dlfcn.h> +#include "ascii-string.h" + #include "internal.h" #include "nbd-protocol.h" #include "options.h" @@ -300,9 +302,9 @@ main (int argc, char *argv[]) case TLS_OPTION: tls_set_on_cli = true; - if (strcasecmp (optarg, "require") == 0 || - strcasecmp (optarg, "required") == 0 || - strcasecmp (optarg, "force") == 0) + if (ascii_strcasecmp (optarg, "require") == 0 || + ascii_strcasecmp (optarg, "required") == 0 || + ascii_strcasecmp (optarg, "force") == 0) tls = 2; else { tls = nbdkit_parse_bool (optarg); diff --git a/server/public.c b/server/public.c index d1925c9f..bcf1a3a2 100644 --- a/server/public.c +++ b/server/public.c @@ -52,6 +52,7 @@ #include <sys/socket.h> #include "ascii-ctype.h" +#include "ascii-string.h" #include "get-current-dir-name.h" #include "internal.h" @@ -385,19 +386,19 @@ int nbdkit_parse_bool (const char *str) { if (!strcmp (str, "1") || - !strcasecmp (str, "true") || - !strcasecmp (str, "t") || - !strcasecmp (str, "yes") || - !strcasecmp (str, "y") || - !strcasecmp (str, "on")) + !ascii_strcasecmp (str, "true") || + !ascii_strcasecmp (str, "t") || + !ascii_strcasecmp (str, "yes") || + !ascii_strcasecmp (str, "y") || + !ascii_strcasecmp (str, "on")) return 1; if (!strcmp (str, "0") || - !strcasecmp (str, "false") || - !strcasecmp (str, "f") || - !strcasecmp (str, "no") || - !strcasecmp (str, "n") || - !strcasecmp (str, "off")) + !ascii_strcasecmp (str, "false") || + !ascii_strcasecmp (str, "f") || + !ascii_strcasecmp (str, "no") || + !ascii_strcasecmp (str, "n") || + !ascii_strcasecmp (str, "off")) return 0; nbdkit_error ("could not decipher boolean (%s)", str); diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c index f4afc826..b3059ed4 100644 --- a/plugins/curl/curl.c +++ b/plugins/curl/curl.c @@ -58,6 +58,7 @@ #include "cleanup.h" #include "ascii-ctype.h" +#include "ascii-string.h" static const char *url = NULL; /* required */ @@ -477,8 +478,8 @@ curl_open (int readonly) #endif nbdkit_debug ("content length: %" PRIi64, h->exportsize); - if (strncasecmp (url, "http://", strlen ("http://")) == 0 || - strncasecmp (url, "https://", strlen ("https://")) == 0) { + if (ascii_strncasecmp (url, "http://", strlen ("http://")) == 0 || + ascii_strncasecmp (url, "https://", strlen ("https://")) == 0) { if (!h->accept_range) { nbdkit_error ("server does not support 'range' (byte range) requests"); goto err; @@ -562,7 +563,7 @@ header_cb (void *ptr, size_t size, size_t nmemb, void *opaque) const char *bytes = "bytes"; if (realsize >= strlen (accept_ranges) && - strncasecmp (header, accept_ranges, strlen (accept_ranges)) == 0) { + ascii_strncasecmp (header, accept_ranges, strlen (accept_ranges)) == 0) { const char *p = strchr (header, ':') + 1; /* Skip whitespace between the header name and value. */ diff --git a/plugins/info/info.c b/plugins/info/info.c index 6505ffbb..33c4facd 100644 --- a/plugins/info/info.c +++ b/plugins/info/info.c @@ -49,6 +49,7 @@ #include <nbdkit-plugin.h> +#include "ascii-string.h" #include "byte-swapping.h" #include "tvdiff.h" @@ -76,12 +77,12 @@ static int info_config (const char *key, const char *value) { if (strcmp (key, "mode") == 0) { - if (strcasecmp (value, "exportname") == 0 || - strcasecmp (value, "export-name") == 0) { + if (ascii_strcasecmp (value, "exportname") == 0 || + ascii_strcasecmp (value, "export-name") == 0) { mode = MODE_EXPORTNAME; } - else if (strcasecmp (value, "base64exportname") == 0 || - strcasecmp (value, "base64-export-name") == 0) { + else if (ascii_strcasecmp (value, "base64exportname") == 0 || + ascii_strcasecmp (value, "base64-export-name") == 0) { #ifdef HAVE_BASE64 mode = MODE_BASE64EXPORTNAME; #else @@ -89,13 +90,13 @@ info_config (const char *key, const char *value) return -1; #endif } - else if (strcasecmp (value, "address") == 0) + else if (ascii_strcasecmp (value, "address") == 0) mode = MODE_ADDRESS; - else if (strcasecmp (value, "time") == 0) + else if (ascii_strcasecmp (value, "time") == 0) mode = MODE_TIME; - else if (strcasecmp (value, "uptime") == 0) + else if (ascii_strcasecmp (value, "uptime") == 0) mode = MODE_UPTIME; - else if (strcasecmp (value, "conntime") == 0) + else if (ascii_strcasecmp (value, "conntime") == 0) mode = MODE_CONNTIME; else { nbdkit_error ("unknown mode: '%s'", value); diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c index 0ea2a4a2..b21590e6 100644 --- a/plugins/nbd/nbd.c +++ b/plugins/nbd/nbd.c @@ -52,6 +52,8 @@ #define NBDKIT_API_VERSION 2 #include <nbdkit-plugin.h> + +#include "ascii-string.h" #include "byte-swapping.h" #include "cleanup.h" #include "utils.h" @@ -152,9 +154,9 @@ nbdplug_config (const char *key, const char *value) shared = r; } else if (strcmp (key, "tls") == 0) { - if (strcasecmp (value, "require") == 0 || - strcasecmp (value, "required") == 0 || - strcasecmp (value, "force") == 0) + if (ascii_strcasecmp (value, "require") == 0 || + ascii_strcasecmp (value, "required") == 0 || + ascii_strcasecmp (value, "force") == 0) tls = LIBNBD_TLS_REQUIRE; else { r = nbdkit_parse_bool (value); diff --git a/plugins/partitioning/partitioning.c b/plugins/partitioning/partitioning.c index 09e17e3a..5e963026 100644 --- a/plugins/partitioning/partitioning.c +++ b/plugins/partitioning/partitioning.c @@ -48,6 +48,7 @@ #include <nbdkit-plugin.h> +#include "ascii-string.h" #include "byte-swapping.h" #include "isaligned.h" #include "iszero.h" @@ -172,9 +173,10 @@ partitioning_config (const char *key, const char *value) } } else if (strcmp (key, "partition-type") == 0) { - if (strcasecmp (value, "mbr") == 0 || strcasecmp (value, "dos") == 0) + if (ascii_strcasecmp (value, "mbr") == 0 || + ascii_strcasecmp (value, "dos") == 0) parttype = PARTTYPE_MBR; - else if (strcasecmp (value, "gpt") == 0) + else if (ascii_strcasecmp (value, "gpt") == 0) parttype = PARTTYPE_GPT; else { nbdkit_error ("unknown partition-type: %s", value); @@ -205,13 +207,13 @@ partitioning_config (const char *key, const char *value) alignment = r; } else if (strcmp (key, "mbr-id") == 0) { - if (strcasecmp (value, "default") == 0) + if (ascii_strcasecmp (value, "default") == 0) mbr_id = DEFAULT_MBR_ID; else if (nbdkit_parse_uint8_t ("mbr-id", value, &mbr_id) == -1) return -1; } else if (strcmp (key, "type-guid") == 0) { - if (strcasecmp (value, "default") == 0) + if (ascii_strcasecmp (value, "default") == 0) parse_guid (DEFAULT_TYPE_GUID, type_guid); else if (parse_guid (value, type_guid) == -1) { nbdkit_error ("could not validate GUID: %s", value); diff --git a/plugins/sh/call.c b/plugins/sh/call.c index 741022b6..aa0fe8a0 100644 --- a/plugins/sh/call.c +++ b/plugins/sh/call.c @@ -48,6 +48,7 @@ #include <nbdkit-plugin.h> #include "ascii-ctype.h" +#include "ascii-string.h" #include "cleanup.h" #include "utils.h" @@ -383,48 +384,48 @@ handle_script_error (const char *argv0, char *ebuf, size_t len) } /* Recognize the errno values that match NBD protocol errors */ - if (strncasecmp (ebuf, "EPERM", 5) == 0) { + if (ascii_strncasecmp (ebuf, "EPERM", 5) == 0) { err = EPERM; skip = 5; } - else if (strncasecmp (ebuf, "EIO", 3) == 0) { + else if (ascii_strncasecmp (ebuf, "EIO", 3) == 0) { err = EIO; skip = 3; } - else if (strncasecmp (ebuf, "ENOMEM", 6) == 0) { + else if (ascii_strncasecmp (ebuf, "ENOMEM", 6) == 0) { err = ENOMEM; skip = 6; } - else if (strncasecmp (ebuf, "EINVAL", 6) == 0) { + else if (ascii_strncasecmp (ebuf, "EINVAL", 6) == 0) { err = EINVAL; skip = 6; } - else if (strncasecmp (ebuf, "ENOSPC", 6) == 0) { + else if (ascii_strncasecmp (ebuf, "ENOSPC", 6) == 0) { err = ENOSPC; skip = 6; } - else if (strncasecmp (ebuf, "EOVERFLOW", 9) == 0) { + else if (ascii_strncasecmp (ebuf, "EOVERFLOW", 9) == 0) { err = EOVERFLOW; skip = 9; } - else if (strncasecmp (ebuf, "ESHUTDOWN", 9) == 0) { + else if (ascii_strncasecmp (ebuf, "ESHUTDOWN", 9) == 0) { err = ESHUTDOWN; skip = 9; } - else if (strncasecmp (ebuf, "ENOTSUP", 7) == 0) { + else if (ascii_strncasecmp (ebuf, "ENOTSUP", 7) == 0) { err = ENOTSUP; skip = 7; } - else if (strncasecmp (ebuf, "EOPNOTSUPP", 10) == 0) { + else if (ascii_strncasecmp (ebuf, "EOPNOTSUPP", 10) == 0) { err = EOPNOTSUPP; skip = 10; } /* Other errno values that server/protocol.c treats specially */ - else if (strncasecmp (ebuf, "EROFS", 5) == 0) { + else if (ascii_strncasecmp (ebuf, "EROFS", 5) == 0) { err = EROFS; skip = 5; } - else if (strncasecmp (ebuf, "EDQUOT", 6) == 0) { + else if (ascii_strncasecmp (ebuf, "EDQUOT", 6) == 0) { #ifdef EDQUOT err = EDQUOT; #else @@ -432,7 +433,7 @@ handle_script_error (const char *argv0, char *ebuf, size_t len) #endif skip = 6; } - else if (strncasecmp (ebuf, "EFBIG", 5) == 0) { + else if (ascii_strncasecmp (ebuf, "EFBIG", 5) == 0) { err = EFBIG; skip = 5; } diff --git a/plugins/sh/methods.c b/plugins/sh/methods.c index 5a7dfe05..10cd4100 100644 --- a/plugins/sh/methods.c +++ b/plugins/sh/methods.c @@ -45,6 +45,7 @@ #include <nbdkit-plugin.h> #include "cleanup.h" +#include "ascii-string.h" #include "call.h" #include "methods.h" @@ -105,16 +106,16 @@ sh_thread_model (void) case OK: if (slen > 0 && s[slen-1] == '\n') s[slen-1] = '\0'; - if (strcasecmp (s, "parallel") == 0) + if (ascii_strcasecmp (s, "parallel") == 0) r = NBDKIT_THREAD_MODEL_PARALLEL; - else if (strcasecmp (s, "serialize_requests") == 0 || - strcasecmp (s, "serialize-requests") == 0) + else if (ascii_strcasecmp (s, "serialize_requests") == 0 || + ascii_strcasecmp (s, "serialize-requests") == 0) r = NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS; - else if (strcasecmp (s, "serialize_all_requests") == 0 || - strcasecmp (s, "serialize-all-requests") == 0) + else if (ascii_strcasecmp (s, "serialize_all_requests") == 0 || + ascii_strcasecmp (s, "serialize-all-requests") == 0) r = NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS; - else if (strcasecmp (s, "serialize_connections") == 0 || - strcasecmp (s, "serialize-connections") == 0) + else if (ascii_strcasecmp (s, "serialize_connections") == 0 || + ascii_strcasecmp (s, "serialize-connections") == 0) r = NBDKIT_THREAD_MODEL_SERIALIZE_CONNECTIONS; else { nbdkit_debug ("%s: ignoring unrecognized thread model: %s", @@ -545,11 +546,11 @@ sh_can_fua (void *handle) case OK: if (slen > 0 && s[slen-1] == '\n') s[slen-1] = '\0'; - if (strcasecmp (s, "none") == 0) + if (ascii_strcasecmp (s, "none") == 0) r = NBDKIT_FUA_NONE; - else if (strcasecmp (s, "emulate") == 0) + else if (ascii_strcasecmp (s, "emulate") == 0) r = NBDKIT_FUA_EMULATE; - else if (strcasecmp (s, "native") == 0) + else if (ascii_strcasecmp (s, "native") == 0) r = NBDKIT_FUA_NATIVE; else { nbdkit_error ("%s: could not parse output from %s method: %s", @@ -600,11 +601,11 @@ sh_can_cache (void *handle) case OK: if (slen > 0 && s[slen-1] == '\n') s[slen-1] = '\0'; - if (strcasecmp (s, "none") == 0) + if (ascii_strcasecmp (s, "none") == 0) r = NBDKIT_CACHE_NONE; - else if (strcasecmp (s, "emulate") == 0) + else if (ascii_strcasecmp (s, "emulate") == 0) r = NBDKIT_CACHE_EMULATE; - else if (strcasecmp (s, "native") == 0) + else if (ascii_strcasecmp (s, "native") == 0) r = NBDKIT_CACHE_NATIVE; else { nbdkit_error ("%s: could not parse output from %s method: %s", diff --git a/filters/ip/ip.c b/filters/ip/ip.c index 4f6e32e4..26a34d6e 100644 --- a/filters/ip/ip.c +++ b/filters/ip/ip.c @@ -45,6 +45,7 @@ #include <nbdkit-filter.h> +#include "ascii-string.h" #include "cleanup.h" /* -D ip.rules=1 to enable debugging of rules and rule matching. */ @@ -199,20 +200,20 @@ parse_rule (const char *paramname, assert (n > 0); - if (n == 3 && (strncasecmp (value, "all", 3) == 0 || - strncasecmp (value, "any", 3) == 0)) { + if (n == 3 && (ascii_strncasecmp (value, "all", 3) == 0 || + ascii_strncasecmp (value, "any", 3) == 0)) { new_rule->type = ANY; return 0; } - if (n == 7 && (strncasecmp (value, "allipv4", 7) == 0 || - strncasecmp (value, "anyipv4", 7) == 0)) { + if (n == 7 && (ascii_strncasecmp (value, "allipv4", 7) == 0 || + ascii_strncasecmp (value, "anyipv4", 7) == 0)) { new_rule->type = ANYV4; return 0; } - if (n == 7 && (strncasecmp (value, "allipv6", 7) == 0 || - strncasecmp (value, "anyipv6", 7) == 0)) { + if (n == 7 && (ascii_strncasecmp (value, "allipv6", 7) == 0 || + ascii_strncasecmp (value, "anyipv6", 7) == 0)) { new_rule->type = ANYV6; return 0; } diff --git a/.gitignore b/.gitignore index dd510e46..3157a8d3 100644 --- a/.gitignore +++ b/.gitignore @@ -34,6 +34,7 @@ plugins/*/*.3 /autom4te.cache /common/bitmap/test-bitmap /common/include/test-ascii-ctype +/common/include/test-ascii-string /common/include/test-byte-swapping /common/include/test-current-dir-name /common/include/test-isaligned -- 2.25.0

2 1

[PATCH virt-v2v] v2v: -o libvirt: Remove cache=none.
by Richard W.M. Jones 19 May '20

19 May '20

Traditionally if you did live migration (KVM to KVM), you had to ensure that cache=none was set on all disks of the guest up front. This was because of quirks in how NFS works (I think the close-to-open consistency and the fact that during live migration both qemus have the file open), and we had to assume the worst case that a guest might be backed by NFS. Because of this when virt-v2v converts a guest to run on KVM using libvirt it sets cache=none. This is not necessary with modern qemu. If qemu supports the drop-cache property of the file block driver, which libvirt will automatically detect for us, then libvirt live migration is able to tell qemu to drop cached data at the right time even if the backing is NFS. It also had a significant performance impact. In some synthetic benchmarks it could show 2 or 3 times slower performance. Thanks: Ming Xie, Peter Krempa. --- v2v/create_libvirt_xml.ml | 1 - 1 file changed, 1 deletion(-) diff --git a/v2v/create_libvirt_xml.ml b/v2v/create_libvirt_xml.ml index 05553c4f7d..5a1fba0fd6 100644 --- a/v2v/create_libvirt_xml.ml +++ b/v2v/create_libvirt_xml.ml @@ -336,7 +336,6 @@ let create_libvirt_xml ?pool source targets target_buses guestcaps e "driver" [ "name", "qemu"; "type", t.target_format; - "cache", "none" ] []; (match pool with | None -> -- 2.26.2

1 1

[PATCH nbdkit] sh: Don't need to cast parameter of ascii_is* to (unsigned char).
by Richard W.M. Jones 19 May '20

19 May '20

Our replacements for these functions are not undefined for negative values. Thanks: Eric Blake Fixes: commit 9f34db74786fdc92b290a7d47e4b003bd84fec69. --- plugins/sh/call.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/sh/call.c b/plugins/sh/call.c index b2d4a794..741022b6 100644 --- a/plugins/sh/call.c +++ b/plugins/sh/call.c @@ -443,7 +443,7 @@ handle_script_error (const char *argv0, char *ebuf, size_t len) } if (skip && ebuf[skip]) { - if (!ascii_isspace ((unsigned char) ebuf[skip])) { + if (!ascii_isspace (ebuf[skip])) { /* Treat 'EINVALID' as EIO, not EINVAL */ err = EIO; skip = 0; @@ -451,7 +451,7 @@ handle_script_error (const char *argv0, char *ebuf, size_t len) else do skip++; - while (ascii_isspace ((unsigned char) ebuf[skip])); + while (ascii_isspace (ebuf[skip])); } while (len > 0 && ebuf[len-1] == '\n') -- 2.25.0

2 1

[v2v PATCH] vCenter: fix parsing of HTTP status string (RHBZ#1837328)
by Pino Toscano 19 May '20

19 May '20

vCenter 7 answers with an HTTP/2 status string, so we cannot extract the status code from it by using fixed positions in that string. Hence, pick the status code by reading what's after the whitespace. Tested with vCenter 6.5 and 7. --- v2v/vCenter.ml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/v2v/vCenter.ml b/v2v/vCenter.ml index c28a4ced..4c128b0c 100644 --- a/v2v/vCenter.ml +++ b/v2v/vCenter.ml @@ -190,7 +190,9 @@ and fetch_headers_from_url password_file uri sslverify https_url = | [] -> dump_response stderr; error (f_"vcenter: no status code in output of ‘curl’ command. Is ‘curl’ installed?") - | ss -> String.sub (List.hd (List.rev ss)) 9 3 in + | ss -> + let s = List.hd (List.rev ss) in + String.sub s (String.index s ' ' + 1) 3 in let headers = List.map ( -- 2.25.4

2 1

[PATCH v4 00/34] Configurable policy for handling deprecated interfaces
by Markus Armbruster 08 May '20

08 May '20

This series extends QMP introspection to cover deprecation. Additionally, new option -compat lets you configure what to do when deprecated interfaces get used. This is intended for testing users of the management interfaces. It is experimental. -compat deprecated-input=<in-policy> configures what to do when deprecated input is received. Available policies: * accept: Accept deprecated commands and arguments (default) * reject: Reject them * crash: Crash -compat deprecated-output=<out-policy> configures what to do when deprecated output is sent. Available output policies: * accept: Emit deprecated command results and events (default) * hide: Suppress them For now, -compat covers only deprecated syntactic aspects of QMP. We may want to extend it to cover semantic aspects, CLI, and experimental features. PATCH 01-04: Documentation fixes PATCH 05-10: Test improvements PATCH 11-24: Add feature flags to remaining user-defined types and to struct members PATCH 25-26: New special feature 'deprecated', visible in introspection PATCH 27-34: New -compat to set policy for handling stuff marked with feature 'deprecated' v4: PATCH 05+07: Temporary memory leak plugged [Marc-André] PATCH 23: Rewritten [Marc-André] PATCH 24: Comment typo [Marc-André] PATCH 30: Memory leaks plugged v3: * Rebased, non-trivial conflicts in PATCH 01+26+27+34 due to RST conversion and code motion * PATCH 28-29: Old PATCH 28 split up to ease review * PATCH 30-31: New * PATCH 32-33: Old PATCH 29 split up to ease review Comparison to RFC (24 Oct 2019): * Cover arguments and results in addition to commands and events * Half-baked "[RFC PATCH 18/19] qapi: Include a warning in the response to a deprecated command" dropped See also last item of Subject: Minutes of KVM Forum BoF on deprecating stuff Date: Fri, 26 Oct 2018 16:03:51 +0200 Message-ID: <87mur0ls8o.fsf(a)dusky.pond.sub.org> https://lists.nongnu.org/archive/html/qemu-devel/2018-10/msg05828.html Cc: Lukáš Doktor <ldoktor(a)redhat.com> Cc: libguestfs(a)redhat.com Cc: libvir-list(a)redhat.com Cc: Daniel P. Berrange <berrange(a)redhat.com> Cc: Peter Krempa <pkrempa(a)redhat.com> Cc: Kevin Wolf <kwolf(a)redhat.com> Markus Armbruster (34): qemu-doc: Belatedly document QMP command arg & result deprecation qapi: Belatedly update doc comment for @wait deprecation docs/devel/qapi-code-gen: Clarify allow-oob introspection docs/devel/qapi-code-gen: Document 'features' introspection tests/test-qmp-cmds: Factor out qmp_dispatch() test helpers tests/test-qmp-cmds: Check responses more thoroughly tests/test-qmp-cmds: Simplify test data setup tests/test-qmp-event: Simplify test data setup tests/test-qmp-event: Use qobject_is_equal() tests/test-qmp-event: Check event is actually emitted qapi/schema: Clean up around QAPISchemaEntity.connect_doc() qapi: Add feature flags to remaining definitions qapi: Consistently put @features parameter right after @ifcond qapi/introspect: Rename *qlit* to reduce confusion qapi/introspect: Factor out _make_tree() qapi/schema: Change _make_features() to a take feature list qapi/schema: Reorder classes so related ones are together qapi/schema: Rename QAPISchemaObjectType{Variant,Variants} qapi/schema: Call QAPIDoc.connect_member() in just one place qapi: Add feature flags to struct members qapi: Inline do_qmp_dispatch() into qmp_dispatch() qapi: Simplify how qmp_dispatch() deals with QCO_NO_SUCCESS_RESP qapi: Simplify how qmp_dispatch() gets the request ID qapi: Replace qmp_dispatch()'s TODO comment by an explanation qapi: New special feature flag "deprecated" qapi: Mark deprecated QMP parts with feature 'deprecated' qemu-options: New -compat to set policy for deprecated interfaces qapi: Implement deprecated-output=hide for QMP command results qapi: Implement deprecated-output=hide for QMP events qapi: Implement deprecated-output=hide for QMP event data qapi: Implement deprecated-output=hide for QMP introspection qapi: Implement deprecated-input=reject for QMP commands qapi: Implement deprecated-input=reject for QMP command arguments qapi: New -compat deprecated-input=crash docs/devel/qapi-code-gen.txt | 79 ++- docs/system/deprecated.rst | 48 +- tests/qapi-schema/doc-good.texi | 32 ++ qapi/block-core.json | 48 +- qapi/block.json | 30 +- qapi/char.json | 1 + qapi/compat.json | 52 ++ qapi/control.json | 11 +- qapi/introspect.json | 28 +- qapi/machine.json | 34 +- qapi/migration.json | 36 +- qapi/misc.json | 13 +- qapi/qapi-schema.json | 1 + include/qapi/compat-policy.h | 20 + include/qapi/qmp/dispatch.h | 1 + include/qapi/qobject-input-visitor.h | 9 + include/qapi/qobject-output-visitor.h | 9 + include/qapi/visitor-impl.h | 3 + include/qapi/visitor.h | 9 + monitor/monitor-internal.h | 3 - monitor/misc.c | 2 - monitor/qmp-cmds-control.c | 102 +++- qapi/qapi-visit-core.c | 9 + qapi/qmp-dispatch.c | 149 +++--- qapi/qobject-input-visitor.c | 29 ++ qapi/qobject-output-visitor.c | 20 + qemu-storage-daemon.c | 2 - softmmu/vl.c | 17 + tests/test-qmp-cmds.c | 249 +++++---- tests/test-qmp-event.c | 203 +++----- qapi/Makefile.objs | 8 +- qapi/trace-events | 1 + qemu-options.hx | 22 + scripts/qapi/commands.py | 20 +- scripts/qapi/doc.py | 16 +- scripts/qapi/events.py | 24 +- scripts/qapi/expr.py | 14 +- scripts/qapi/introspect.py | 104 ++-- scripts/qapi/schema.py | 488 ++++++++++-------- scripts/qapi/types.py | 8 +- scripts/qapi/visit.py | 28 +- tests/Makefile.include | 1 + tests/qapi-schema/alternate-base.err | 2 +- tests/qapi-schema/doc-good.json | 22 +- tests/qapi-schema/doc-good.out | 18 + .../qapi-schema/features-deprecated-type.err | 2 + .../qapi-schema/features-deprecated-type.json | 3 + .../qapi-schema/features-deprecated-type.out | 0 tests/qapi-schema/qapi-schema-test.json | 51 +- tests/qapi-schema/qapi-schema-test.out | 48 +- tests/qapi-schema/test-qapi.py | 26 +- 51 files changed, 1393 insertions(+), 762 deletions(-) create mode 100644 qapi/compat.json create mode 100644 include/qapi/compat-policy.h create mode 100644 tests/qapi-schema/features-deprecated-type.err create mode 100644 tests/qapi-schema/features-deprecated-type.json create mode 100644 tests/qapi-schema/features-deprecated-type.out -- 2.21.1

3 5