May 2019 - Libguestfs - libguestfs List Archives

nbdkit problem with cache/cow and unaligned sizes
by Eric Blake 17 May '19

17 May '19

Right now, the cache and cow filters always round up requests to blksize boundaries (blksize for cache is dynamically determined at connection start, for cow is fixed as BLKSIZE). Which is fine for the bulk of the underlying file, but can cause problems when reading past EOF for a partial tail of an underlying plugin. We aren't validating that filter calls to next_ops are within bounds; and even if the plugin tolerates the past-EOF read, we aren't guaranteeing that the client will always read 0 bytes in the past-EOF tail. Several ideas of fixing it, each with some drawbacks: + in cache/cow_get_size(), truncate the plugin's size down to blksize prior to calling blk_set_size() (renders the plugin's tail unusable) + reject serving images that aren't already aligned to blksize (avoids missing bytes or worrying about past-EOF slop, but can be mean, unless...) + document that for unaligned images, you can use --filter=cache --filter=truncate round-up=BLKSIZE, to let the truncate filter take care of our slop (doesn't play nicely with the fact that we can only use a filter once, if a user wants to also use --filter=truncate prior to --filter=cache) + rewrite both the cache/blk.c and cow/blk.c handlers to pay more attention to unaligned EOF (code duplication) + teach filters.c next_ops to auto-cap filter requests into valid ranges prior to calling into the next layer (trickier than it looks, especially if we later add NBD resize extension support) + others? -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org

2 2

[nbdkit PATCH] Introduce cacheextents filter
by Martin Kletzander 15 May '19

15 May '19

This filter caches the last result of the extents() call and offers a nice speed-up for clients that only support req_on=1 in combination with plugins like vddk, which has no overhead for returning information for multiple extents in one call, but that call is very time-consuming. Quick test showed that on a fast connection and a sparsely allocated 16G disk with a OS installed `qemu-img map` runs 16s instead of 33s (out of which it takes 5s to the first extents request). For 100G disk with no data on it, that is one hole extent spanning the whole disk (where there is no space for improvement) this does not add any noticeable overhead. Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com> --- Yes, I hate the filter name, so if anyone has a better idea, feel free to suggest (or rename) it. configure.ac | 2 + filters/cacheextents/Makefile.am | 64 ++++++ filters/cacheextents/cacheextents.c | 200 ++++++++++++++++++ .../nbdkit-cacheextents-filter.pod | 47 ++++ 4 files changed, 313 insertions(+) create mode 100644 filters/cacheextents/Makefile.am create mode 100644 filters/cacheextents/cacheextents.c create mode 100644 filters/cacheextents/nbdkit-cacheextents-filter.pod diff --git a/configure.ac b/configure.ac index 58031f3c6d86..8db5a4cac3c5 100644 --- a/configure.ac +++ b/configure.ac @@ -825,6 +825,7 @@ plugins="$(echo $(printf %s\\n $lang_plugins $non_lang_plugins | sort -u))" filters="\ blocksize \ cache \ + cacheextents \ cow \ delay \ error \ @@ -900,6 +901,7 @@ AC_CONFIG_FILES([Makefile filters/Makefile filters/blocksize/Makefile filters/cache/Makefile + filters/cacheextents/Makefile filters/cow/Makefile filters/delay/Makefile filters/error/Makefile diff --git a/filters/cacheextents/Makefile.am b/filters/cacheextents/Makefile.am new file mode 100644 index 000000000000..9e5b59c09c59 --- /dev/null +++ b/filters/cacheextents/Makefile.am @@ -0,0 +1,64 @@ +# nbdkit +# Copyright (C) 2019 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +include $(top_srcdir)/common-rules.mk + +EXTRA_DIST = nbdkit-cacheextents-filter.pod + +filter_LTLIBRARIES = nbdkit-cacheextents-filter.la + +nbdkit_cacheextents_filter_la_SOURCES = \ + cacheextents.c \ + $(top_srcdir)/include/nbdkit-filter.h + +nbdkit_cacheextents_filter_la_CPPFLAGS = \ + -I$(top_srcdir)/include \ + -I$(top_srcdir)/common/include \ + -I$(top_srcdir)/common/utils +nbdkit_cacheextents_filter_la_CFLAGS = \ + $(WARNINGS_CFLAGS) +nbdkit_cacheextents_filter_la_LDFLAGS = \ + -module -avoid-version -shared \ + -Wl,--version-script=$(top_srcdir)/filters/filters.syms +nbdkit_cacheextents_filter_la_LIBADD = \ + $(top_builddir)/common/utils/libutils.la + +if HAVE_POD + +man_MANS = nbdkit-cacheextents-filter.1 +CLEANFILES += $(man_MANS) + +nbdkit-cacheextents-filter.1: nbdkit-cacheextents-filter.pod + $(PODWRAPPER) --section=1 --man $@ \ + --html $(top_builddir)/html/$@.html \ + $< + +endif HAVE_POD diff --git a/filters/cacheextents/cacheextents.c b/filters/cacheextents/cacheextents.c new file mode 100644 index 000000000000..89ac2e7a0767 --- /dev/null +++ b/filters/cacheextents/cacheextents.c @@ -0,0 +1,200 @@ +/* nbdkit + * Copyright (C) 2019 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <errno.h> +#include <inttypes.h> + +#include <pthread.h> + +#include <nbdkit-filter.h> + +#include "cleanup.h" + +#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL + +/* This lock protects the global state. */ +static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; + +/* The real size of the underlying plugin. */ +static uint64_t size; + +/* Cached extents from the last extents () call and its start and end for the + sake of simplicity. */ +struct nbdkit_extents *cache_extents; +static uint64_t cache_start; +static uint64_t cache_end; + +static void +cacheextents_unload (void) +{ + nbdkit_extents_free (cache_extents); +} + +static int +cacheextents_add (struct nbdkit_extents *extents) +{ + size_t i = 0; + + for (i = 0; i < nbdkit_extents_count (cache_extents); i++) { + struct nbdkit_extent ex = nbdkit_get_extent (cache_extents, i); + if (nbdkit_add_extent (extents, ex.offset, ex.length, ex.type) < 0) + return -1; + } + + return 0; +} + +static int +cacheextents_fill (struct nbdkit_extents *extents) +{ + size_t i = 0; + size_t count = nbdkit_extents_count (extents); + struct nbdkit_extent first = nbdkit_get_extent (extents, 0); + struct nbdkit_extent last = nbdkit_get_extent (extents, count - 1); + + nbdkit_extents_free (cache_extents); + cache_start = first.offset; + cache_end = last.offset + last.length; + cache_extents = nbdkit_extents_new (cache_start, cache_end); + + for (i = 0; i < count; i++) { + struct nbdkit_extent ex = nbdkit_get_extent (extents, i); + nbdkit_debug ("cacheextents: updating cache with" + ": offset=%" PRIu64 + ": length=%" PRIu64 + "; type=%x", + ex.offset, ex.length, ex.type); + if (nbdkit_add_extent (cache_extents, ex.offset, ex.length, ex.type) < 0) + return -1; + } + + return 0; +} + +static int +cacheextents_extents (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle, uint32_t count, uint64_t offset, uint32_t flags, + struct nbdkit_extents *extents, + int *err) +{ + nbdkit_debug ("cacheextents:" + " cache_start=%" PRIu64 + " cache_end=%" PRIu64 + " cache_extents=%p", + cache_start, cache_end, cache_extents); + + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + + if (cache_extents && + offset >= cache_start && offset < cache_end) { + nbdkit_debug ("cacheextents: returning from cache"); + int r = cacheextents_add (extents); + if (r < 0) + *err = errno; + return r; + } + + nbdkit_debug ("cacheextents: cache miss"); + int r = next_ops->extents (nxdata, count, offset, flags, extents, err); + if (r < 0) + return r; + + if (cacheextents_fill (extents) < 0) { + *err = errno; + return -1; + } + + return r; +} + +/* Any changes to the data need to clean the cache. + * + * Similarly to readahead filter this could be more intelligent, but there would + * be very little benefit. + */ + +static void +kill_cacheextents (void) +{ + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + nbdkit_extents_free (cache_extents); + cache_extents = NULL; +} + +static int +cacheextents_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle, + const void *buf, uint32_t count, uint64_t offset, + uint32_t flags, int *err) +{ + kill_cacheextents (); + return next_ops->pwrite (nxdata, buf, count, offset, flags, err); +} + +static int +cacheextents_trim (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle, + uint32_t count, uint64_t offset, uint32_t flags, + int *err) +{ + kill_cacheextents (); + return next_ops->trim (nxdata, count, offset, flags, err); +} + +static int +cacheextents_zero (struct nbdkit_next_ops *next_ops, void *nxdata, + void *handle, + uint32_t count, uint64_t offset, uint32_t flags, + int *err) +{ + kill_cacheextents (); + return next_ops->zero (nxdata, count, offset, flags, err); +} + +static struct nbdkit_filter filter = { + .name = "cacheextents", + .longname = "nbdkit cacheextents filter", + .version = PACKAGE_VERSION, + .unload = cacheextents_unload, + .pwrite = cacheextents_pwrite, + .trim = cacheextents_trim, + .zero = cacheextents_zero, + .extents = cacheextents_extents, +}; + +NBDKIT_REGISTER_FILTER (filter) diff --git a/filters/cacheextents/nbdkit-cacheextents-filter.pod b/filters/cacheextents/nbdkit-cacheextents-filter.pod new file mode 100644 index 000000000000..88f3173b9375 --- /dev/null +++ b/filters/cacheextents/nbdkit-cacheextents-filter.pod @@ -0,0 +1,47 @@ +=head1 NAME + +nbdkit-cacheextents-filter - cache extents + +=head1 SYNOPSIS + + nbdkit --filter=cacheextents plugin + +=head1 DESCRIPTION + +C<nbdkit-cacheextents-filter> is a filter that caches the result of last +extents() call. + +A common use for this filter is to accelerate returning extents data for +clients which ask for extents with the REQ_ONE flag set (like +S<C<qemu-img convert>>) and is especially useful in combination with +plugins that report multiple extents in one call, but with high latency +for each of those calls (like L<nbdkit-vddk-plugin(1)>). For example: + + nbdkit -U - --filter=cacheextents --run 'qemu-img map $nbd' vddk ... + +For files with big extents (when it is unlikely for one extents() call +to return multiple different extents) this does not slow down the +access. + +=head1 PARAMETERS + +There are no parameters specific to nbdkit-cacheextents-filter. Any +parameters are passed through to and processed by the underlying +plugin in the normal way. + +=head1 SEE ALSO + +L<nbdkit(1)>, +L<nbdkit-cache-filter(1)>, +L<nbdkit-readahead-filter(1)>, +L<nbdkit-vddk-plugin(1)>, +L<nbdkit-filter(3)>, +L<qemu-img(1)>. + +=head1 AUTHORS + +Martin Kletzander + +=head1 COPYRIGHT + +Copyright (C) 2019 Red Hat Inc. -- 2.21.0

2 2

[PATCH nbdkit 0/7] server: Implement NBD_FLAG_CAN_MULTI_CONN.
by Richard W.M. Jones 15 May '19

15 May '19

First thing to say is that I need to do a *lot* more testing on this, so this is just an early peek. In particular, although it passed ‘make check && make check-valgrind’ I have *not* tested it against a multi-conn-aware client such as the Linux kernel >= 4.9. This implements NBD_FLAG_CAN_MULTI_CONN, described in the protocol doc as: "NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to the client. In the absence of this flag, clients SHOULD NOT multiplex their commands over more than one connection to the export." This is necessary to support the Linux nbd client -C option. The only plugin which sets the flag so far is file. The ocaml, sh and nbd plugins allow the flag to be controlled or passed through. Notable also is that the blocksize filter has to filter out this flag, because I'm not convinced that the bounce buffer is safe. However I believe the other filters *are* safe (although not really certain about the fua filter, and the new cache filter is tricky too). My feeling is that we should set the flag unconditionally for all readonly connections, but I can think of nasty corner cases where it might not work. We should most probably set it in all plugins that are readonly (eg. nbdkit-pattern-plugin). Rich.

2 15

[nbdkit PATCH] extents: Do not shorten overlaps by 0
by Martin Kletzander 15 May '19

15 May '19

When offset + length == exts->end the code which is shortening the extents is subtracting 0 from length. And it is not optimized out. Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com> --- server/extents.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server/extents.c b/server/extents.c index d3d1a15ab97c..c422491601f0 100644 --- a/server/extents.c +++ b/server/extents.c @@ -168,7 +168,7 @@ nbdkit_add_extent (struct nbdkit_extents *exts, return 0; /* Shorten extents that overlap the end of the range. */ - if (offset + length >= exts->end) { + if (offset + length > exts->end) { overlap = offset + length - exts->end; length -= overlap; } -- 2.21.0

3 2

[nbdkit PATCH v2 0/2] Bounce buffer cleanups
by Eric Blake 14 May '19

14 May '19

Based on Rich's review of my v1 that touched only cache.c, I have now tried to bring all three filters with alignment rounding in line with one another. There is definitely room for future improvements once we teach nbdkit to let filters and plugins advertise block sizes, but I'm hoping to get NBD_CMD_CACHE implemented first. Eric Blake (2): blocksize: Process requests in linear order cache, cow: Reduce use of bounce-buffer filters/blocksize/blocksize.c | 81 +++++++-------- filters/cache/cache.c | 173 +++++++++++++++++++++++--------- filters/cow/cow.c | 180 +++++++++++++++++++++++++--------- 3 files changed, 299 insertions(+), 135 deletions(-) -- 2.20.1

2 3

[nbdkit PATCH 0/7] Implement structured replies in nbd plugin
by Eric Blake 14 May '19

14 May '19

I'm hoping to implement .extents for the nbd plugin; this is a prerequisite. I'm not sure about patch 3 - if we like it, I'll squash it to 2, if we don't, I think we are okay just dropping it. I'm also wondering if we have to worry about malicious plugins that don't populate the entire .pread buffer in an effort to get nbdkit to expose portions of the heap; my patch 7 loses some time on extra memset()s to avoid that, although it's probably in the noise compared to the time spent talking over sockets to the remote server. Eric Blake (7): protocol: Send correct structured reply error for non-Linux protocol: Support EOVERFLOW RFC: protocol: Only send EOVERFLOW when valid nbd: Honor server global flags on little-endian protocol: Add helpers for error response handling nbd: Implement NBD_OPT_GO client request nbd: Implement structured replies common/protocol/protocol.h | 21 +- plugins/nbd/nbd.c | 434 +++++++++++++++++++++++++++++++++---- server/protocol.c | 16 +- 3 files changed, 422 insertions(+), 49 deletions(-) -- 2.20.1

3 17

[nbdkit PATCH 0/9] RFC: implement NBD_CMD_CACHE
by Eric Blake 13 May '19

13 May '19

I'm still working my way through the filters before this series will be complete, but this is enough of a start to at least get some feedback on the idea of implementing another NBD protocol extension. Eric Blake (9): server: Internal hooks for implementing NBD_CMD_CACHE plugins: Add .cache callback file, split: Implement .cache with posix_fadvise nbd: Implement NBD_CMD_CACHE passthrough plugins: Implement no-op .cache for in-memory plugins sh: Implement .cache script callback filters: Add .cache callback test-layers: Test .cache usage blocksize: Implement .cache rounding docs/nbdkit-filter.pod | 27 ++++++++++++++- docs/nbdkit-plugin.pod | 52 ++++++++++++++++++++++++++++ docs/nbdkit-protocol.pod | 10 ++++++ plugins/sh/nbdkit-sh-plugin.pod | 12 ++++++- configure.ac | 3 +- common/protocol/protocol.h | 2 ++ include/nbdkit-filter.h | 8 +++++ include/nbdkit-plugin.h | 2 ++ server/internal.h | 4 +++ filters/blocksize/blocksize.c | 31 ++++++++++++++++- plugins/data/data.c | 10 ++++++ plugins/file/file.c | 22 ++++++++++++ plugins/full/full.c | 12 ++++++- plugins/memory/memory.c | 10 ++++++ plugins/nbd/nbd.c | 22 ++++++++++++ plugins/null/null.c | 12 ++++++- plugins/pattern/pattern.c | 12 ++++++- plugins/random/random.c | 12 ++++++- plugins/sh/sh.c | 40 ++++++++++++++++++++++ plugins/split/split.c | 37 +++++++++++++++++++- plugins/streaming/streaming.c | 11 ++++++ plugins/zero/zero.c | 12 ++++++- server/filters.c | 60 ++++++++++++++++++++++++++++++++- server/plugins.c | 39 +++++++++++++++++++++ server/protocol-handshake.c | 10 +++++- server/protocol.c | 25 ++++++++++++++ tests/test-layers-filter.c | 22 +++++++++++- tests/test-layers-plugin.c | 17 ++++++++++ tests/test-layers.c | 36 ++++++++++++++++++++ tests/test-eflags.sh | 53 +++++++++++++++-------------- 30 files changed, 587 insertions(+), 38 deletions(-) -- 2.20.1

2 12

[nbdkit PATCH] cache: Reduce use of bounce-buffer
by Eric Blake 13 May '19

13 May '19

Although the time spent in memcpy/memset probably pales in comparison to time spent in socket I/O, it's still worth worth reducing the number of times we have to utilize a bounce buffer when we already have aligned data. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- filters/cache/cache.c | 60 ++++++++++++++++++++++++++++--------------- 1 file changed, 39 insertions(+), 21 deletions(-) diff --git a/filters/cache/cache.c b/filters/cache/cache.c index 19ce555..98786b5 100644 --- a/filters/cache/cache.c +++ b/filters/cache/cache.c @@ -1,5 +1,5 @@ /* nbdkit - * Copyright (C) 2018 Red Hat Inc. + * Copyright (C) 2018-2019 Red Hat Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are @@ -58,6 +58,7 @@ #include "cache.h" #include "blk.h" #include "reclaim.h" +#include "isaligned.h" #define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL @@ -233,11 +234,13 @@ cache_pread (struct nbdkit_next_ops *next_ops, void *nxdata, CLEANUP_FREE uint8_t *block = NULL; assert (!flags); - block = malloc (blksize); - if (block == NULL) { - *err = errno; - nbdkit_error ("malloc: %m"); - return -1; + if (!IS_ALIGNED (count | offset, blksize)) { + block = malloc (blksize); + if (block == NULL) { + *err = errno; + nbdkit_error ("malloc: %m"); + return -1; + } } /* XXX This breaks up large read requests into smaller ones, which @@ -258,12 +261,14 @@ cache_pread (struct nbdkit_next_ops *next_ops, void *nxdata, { ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); - r = blk_read (next_ops, nxdata, blknum, block, err); + r = blk_read (next_ops, nxdata, blknum, + blkoffs || n < blksize ? block : buf, err); } if (r == -1) return -1; - memcpy (buf, &block[blkoffs], n); + if (blkoffs || n < blksize) + memcpy (buf, &block[blkoffs], n); buf += n; count -= n; @@ -282,11 +287,13 @@ cache_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata, CLEANUP_FREE uint8_t *block = NULL; bool need_flush = false; - block = malloc (blksize); - if (block == NULL) { - *err = errno; - nbdkit_error ("malloc: %m"); - return -1; + if (!IS_ALIGNED (count | offset, blksize)) { + block = malloc (blksize); + if (block == NULL) { + *err = errno; + nbdkit_error ("malloc: %m"); + return -1; + } } if ((flags & NBDKIT_FLAG_FUA) && @@ -308,11 +315,15 @@ cache_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata, * Hold the lock over the whole operation. */ ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); - r = blk_read (next_ops, nxdata, blknum, block, err); - if (r != -1) { - memcpy (&block[blkoffs], buf, n); - r = blk_write (next_ops, nxdata, blknum, block, flags, err); + if (blkoffs || n < blksize) { + r = blk_read (next_ops, nxdata, blknum, block, err); + if (r != -1) { + memcpy (&block[blkoffs], buf, n); + r = blk_write (next_ops, nxdata, blknum, block, flags, err); + } } + else + r = blk_write (next_ops, nxdata, blknum, buf, flags, err); if (r == -1) return -1; @@ -334,6 +345,7 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata, { CLEANUP_FREE uint8_t *block = NULL; bool need_flush = false; + bool clean = false; block = malloc (blksize); if (block == NULL) { @@ -350,7 +362,7 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata, } while (count > 0) { uint64_t blknum, blkoffs, n; - int r; + int r = 0; blknum = offset / blksize; /* block number */ blkoffs = offset % blksize; /* offset within the block */ @@ -362,11 +374,17 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata, * Hold the lock over the whole operation. */ ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); - r = blk_read (next_ops, nxdata, blknum, block, err); - if (r != -1) { + if (blkoffs || n < blksize) { + r = blk_read (next_ops, nxdata, blknum, block, err); memset (&block[blkoffs], 0, n); + clean = false; + } + else if (!clean) { + memset (block, 0, blksize); + clean = true; + } + if (r != -1) r = blk_write (next_ops, nxdata, blknum, block, flags, err); - } if (r == -1) return -1; -- 2.20.1

2 2

Re: [Libguestfs] Welcome Hiroyuki Katsura as Google Summer of Code 2019 student
by Hiroyuki Katsura 13 May '19

13 May '19

Thanks. I'll summarize the project and issue an outline of the implementation to the mailing list soon. (I'm not familiar with mailing lists so I'm sorry if this message is passed to a wrong place.) Hiroyuki Katsura

1 0

[PATCH 43 0/7] v2v: switch to ocaml-libvirt
by Pino Toscano 13 May '19

13 May '19

Hi, this series switches virt-2v to ocaml-libvirt, embedding the latest version of it from git. This way, it is possible to improve the way v2v connects to libvirt for both input, and output modules, and interacts with libvirt (e.g. no more virsh calls needed in virt-v2v). As side effect, virt-v2v now requires libvirt, as keeping it optional would create too much burden. I could not test all the libvirt input modes (like VDDK, and Xen), but VMware and libvirtxml work fine as before. Changes from v3: - rebase on master - update libvirt-ocaml from its master branch - use $(srcdir) in common/mllibvirt/Makefile.am - parse_libvirt_xml: open a libvirt connection only when needed (and only once) Changes from v2: - rebase on master Changes from v1: - rebase on master - update ocaml-libvirt from libvirt-ocaml.git on libvirt.org, and adjust the code to it - pass again the URI to input_libvirt_vddk, so an error message is preserved Pino Toscano (7): v2v: require libvirt common: Bundle the libvirt-ocaml library for use by virt-v2v v2v: switch to ocaml-libvirt v2v: -o libvirt: use a Lazy for the connection v2v: -o libvirt: switch away from virsh v2v: test-harness: stop using the external ocaml-libvirt build: stop looking for ocaml-libvirt .gitignore | 2 + Makefile.am | 5 +- common/mllibvirt/Makefile.am | 102 ++ common/mllibvirt/generator.pl | 908 +++++++++++++ common/mllibvirt/libvirt.README | 9 + common/mllibvirt/libvirt.ml | 1673 ++++++++++++++++++++++++ common/mllibvirt/libvirt.mli | 1647 ++++++++++++++++++++++++ common/mllibvirt/libvirt_c_epilogue.c | 462 +++++++ common/mllibvirt/libvirt_c_oneoffs.c | 1714 +++++++++++++++++++++++++ common/mllibvirt/libvirt_c_prologue.c | 134 ++ configure.ac | 1 + docs/C_SOURCE_FILES | 1 - m4/guestfs-ocaml.m4 | 4 - po/POTFILES | 1 - v2v/Makefile.am | 26 +- v2v/copy_to_local.ml | 7 +- v2v/input_libvirt.ml | 20 +- v2v/input_libvirt_other.ml | 27 +- v2v/input_libvirt_other.mli | 5 +- v2v/input_libvirt_vcenter_https.ml | 13 +- v2v/input_libvirt_vcenter_https.mli | 2 +- v2v/input_libvirt_vddk.ml | 15 +- v2v/input_libvirt_vddk.mli | 4 +- v2v/input_libvirt_xen_ssh.ml | 13 +- v2v/input_libvirt_xen_ssh.mli | 2 +- v2v/libvirt_utils-c.c | 539 -------- v2v/libvirt_utils.ml | 95 +- v2v/libvirt_utils.mli | 51 +- v2v/output_libvirt.ml | 56 +- v2v/parse_libvirt_xml.ml | 23 +- v2v/parse_libvirt_xml.mli | 11 +- v2v/test-harness/Makefile.am | 5 +- v2v/v2v.ml | 12 +- 33 files changed, 6893 insertions(+), 696 deletions(-) create mode 100644 common/mllibvirt/Makefile.am create mode 100755 common/mllibvirt/generator.pl create mode 100644 common/mllibvirt/libvirt.README create mode 100644 common/mllibvirt/libvirt.ml create mode 100644 common/mllibvirt/libvirt.mli create mode 100644 common/mllibvirt/libvirt_c_epilogue.c create mode 100644 common/mllibvirt/libvirt_c_oneoffs.c create mode 100644 common/mllibvirt/libvirt_c_prologue.c delete mode 100644 v2v/libvirt_utils-c.c -- 2.20.1

2 14