nbdkit problem with cache/cow and unaligned sizes
by Eric Blake
Right now, the cache and cow filters always round up requests to blksize
boundaries (blksize for cache is dynamically determined at connection
start, for cow is fixed as BLKSIZE). Which is fine for the bulk of the
underlying file, but can cause problems when reading past EOF for a
partial tail of an underlying plugin. We aren't validating that filter
calls to next_ops are within bounds; and even if the plugin tolerates
the past-EOF read, we aren't guaranteeing that the client will always
read 0 bytes in the past-EOF tail.
Several ideas of fixing it, each with some drawbacks:
+ in cache/cow_get_size(), truncate the plugin's size down to blksize
prior to calling blk_set_size() (renders the plugin's tail unusable)
+ reject serving images that aren't already aligned to blksize (avoids
missing bytes or worrying about past-EOF slop, but can be mean, unless...)
+ document that for unaligned images, you can use --filter=cache
--filter=truncate round-up=BLKSIZE, to let the truncate filter take care
of our slop (doesn't play nicely with the fact that we can only use a
filter once, if a user wants to also use --filter=truncate prior to
--filter=cache)
+ rewrite both the cache/blk.c and cow/blk.c handlers to pay more
attention to unaligned EOF (code duplication)
+ teach filters.c next_ops to auto-cap filter requests into valid ranges
prior to calling into the next layer (trickier than it looks, especially
if we later add NBD resize extension support)
+ others?
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
5 years, 5 months
[nbdkit PATCH] Introduce cacheextents filter
by Martin Kletzander
This filter caches the last result of the extents() call and offers a nice
speed-up for clients that only support req_on=1 in combination with plugins like
vddk, which has no overhead for returning information for multiple extents in
one call, but that call is very time-consuming.
Quick test showed that on a fast connection and a sparsely allocated 16G disk
with a OS installed `qemu-img map` runs 16s instead of 33s (out of which it
takes 5s to the first extents request). For 100G disk with no data on it, that
is one hole extent spanning the whole disk (where there is no space for
improvement) this does not add any noticeable overhead.
Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com>
---
Yes, I hate the filter name, so if anyone has a better idea, feel free to
suggest (or rename) it.
configure.ac | 2 +
filters/cacheextents/Makefile.am | 64 ++++++
filters/cacheextents/cacheextents.c | 200 ++++++++++++++++++
.../nbdkit-cacheextents-filter.pod | 47 ++++
4 files changed, 313 insertions(+)
create mode 100644 filters/cacheextents/Makefile.am
create mode 100644 filters/cacheextents/cacheextents.c
create mode 100644 filters/cacheextents/nbdkit-cacheextents-filter.pod
diff --git a/configure.ac b/configure.ac
index 58031f3c6d86..8db5a4cac3c5 100644
--- a/configure.ac
+++ b/configure.ac
@@ -825,6 +825,7 @@ plugins="$(echo $(printf %s\\n $lang_plugins $non_lang_plugins | sort -u))"
filters="\
blocksize \
cache \
+ cacheextents \
cow \
delay \
error \
@@ -900,6 +901,7 @@ AC_CONFIG_FILES([Makefile
filters/Makefile
filters/blocksize/Makefile
filters/cache/Makefile
+ filters/cacheextents/Makefile
filters/cow/Makefile
filters/delay/Makefile
filters/error/Makefile
diff --git a/filters/cacheextents/Makefile.am b/filters/cacheextents/Makefile.am
new file mode 100644
index 000000000000..9e5b59c09c59
--- /dev/null
+++ b/filters/cacheextents/Makefile.am
@@ -0,0 +1,64 @@
+# nbdkit
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+include $(top_srcdir)/common-rules.mk
+
+EXTRA_DIST = nbdkit-cacheextents-filter.pod
+
+filter_LTLIBRARIES = nbdkit-cacheextents-filter.la
+
+nbdkit_cacheextents_filter_la_SOURCES = \
+ cacheextents.c \
+ $(top_srcdir)/include/nbdkit-filter.h
+
+nbdkit_cacheextents_filter_la_CPPFLAGS = \
+ -I$(top_srcdir)/include \
+ -I$(top_srcdir)/common/include \
+ -I$(top_srcdir)/common/utils
+nbdkit_cacheextents_filter_la_CFLAGS = \
+ $(WARNINGS_CFLAGS)
+nbdkit_cacheextents_filter_la_LDFLAGS = \
+ -module -avoid-version -shared \
+ -Wl,--version-script=$(top_srcdir)/filters/filters.syms
+nbdkit_cacheextents_filter_la_LIBADD = \
+ $(top_builddir)/common/utils/libutils.la
+
+if HAVE_POD
+
+man_MANS = nbdkit-cacheextents-filter.1
+CLEANFILES += $(man_MANS)
+
+nbdkit-cacheextents-filter.1: nbdkit-cacheextents-filter.pod
+ $(PODWRAPPER) --section=1 --man $@ \
+ --html $(top_builddir)/html/$@.html \
+ $<
+
+endif HAVE_POD
diff --git a/filters/cacheextents/cacheextents.c b/filters/cacheextents/cacheextents.c
new file mode 100644
index 000000000000..89ac2e7a0767
--- /dev/null
+++ b/filters/cacheextents/cacheextents.c
@@ -0,0 +1,200 @@
+/* nbdkit
+ * Copyright (C) 2019 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <errno.h>
+#include <inttypes.h>
+
+#include <pthread.h>
+
+#include <nbdkit-filter.h>
+
+#include "cleanup.h"
+
+#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL
+
+/* This lock protects the global state. */
+static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
+
+/* The real size of the underlying plugin. */
+static uint64_t size;
+
+/* Cached extents from the last extents () call and its start and end for the
+ sake of simplicity. */
+struct nbdkit_extents *cache_extents;
+static uint64_t cache_start;
+static uint64_t cache_end;
+
+static void
+cacheextents_unload (void)
+{
+ nbdkit_extents_free (cache_extents);
+}
+
+static int
+cacheextents_add (struct nbdkit_extents *extents)
+{
+ size_t i = 0;
+
+ for (i = 0; i < nbdkit_extents_count (cache_extents); i++) {
+ struct nbdkit_extent ex = nbdkit_get_extent (cache_extents, i);
+ if (nbdkit_add_extent (extents, ex.offset, ex.length, ex.type) < 0)
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cacheextents_fill (struct nbdkit_extents *extents)
+{
+ size_t i = 0;
+ size_t count = nbdkit_extents_count (extents);
+ struct nbdkit_extent first = nbdkit_get_extent (extents, 0);
+ struct nbdkit_extent last = nbdkit_get_extent (extents, count - 1);
+
+ nbdkit_extents_free (cache_extents);
+ cache_start = first.offset;
+ cache_end = last.offset + last.length;
+ cache_extents = nbdkit_extents_new (cache_start, cache_end);
+
+ for (i = 0; i < count; i++) {
+ struct nbdkit_extent ex = nbdkit_get_extent (extents, i);
+ nbdkit_debug ("cacheextents: updating cache with"
+ ": offset=%" PRIu64
+ ": length=%" PRIu64
+ "; type=%x",
+ ex.offset, ex.length, ex.type);
+ if (nbdkit_add_extent (cache_extents, ex.offset, ex.length, ex.type) < 0)
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+cacheextents_extents (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, uint32_t count, uint64_t offset, uint32_t flags,
+ struct nbdkit_extents *extents,
+ int *err)
+{
+ nbdkit_debug ("cacheextents:"
+ " cache_start=%" PRIu64
+ " cache_end=%" PRIu64
+ " cache_extents=%p",
+ cache_start, cache_end, cache_extents);
+
+ ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
+
+ if (cache_extents &&
+ offset >= cache_start && offset < cache_end) {
+ nbdkit_debug ("cacheextents: returning from cache");
+ int r = cacheextents_add (extents);
+ if (r < 0)
+ *err = errno;
+ return r;
+ }
+
+ nbdkit_debug ("cacheextents: cache miss");
+ int r = next_ops->extents (nxdata, count, offset, flags, extents, err);
+ if (r < 0)
+ return r;
+
+ if (cacheextents_fill (extents) < 0) {
+ *err = errno;
+ return -1;
+ }
+
+ return r;
+}
+
+/* Any changes to the data need to clean the cache.
+ *
+ * Similarly to readahead filter this could be more intelligent, but there would
+ * be very little benefit.
+ */
+
+static void
+kill_cacheextents (void)
+{
+ ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
+ nbdkit_extents_free (cache_extents);
+ cache_extents = NULL;
+}
+
+static int
+cacheextents_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle,
+ const void *buf, uint32_t count, uint64_t offset,
+ uint32_t flags, int *err)
+{
+ kill_cacheextents ();
+ return next_ops->pwrite (nxdata, buf, count, offset, flags, err);
+}
+
+static int
+cacheextents_trim (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle,
+ uint32_t count, uint64_t offset, uint32_t flags,
+ int *err)
+{
+ kill_cacheextents ();
+ return next_ops->trim (nxdata, count, offset, flags, err);
+}
+
+static int
+cacheextents_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle,
+ uint32_t count, uint64_t offset, uint32_t flags,
+ int *err)
+{
+ kill_cacheextents ();
+ return next_ops->zero (nxdata, count, offset, flags, err);
+}
+
+static struct nbdkit_filter filter = {
+ .name = "cacheextents",
+ .longname = "nbdkit cacheextents filter",
+ .version = PACKAGE_VERSION,
+ .unload = cacheextents_unload,
+ .pwrite = cacheextents_pwrite,
+ .trim = cacheextents_trim,
+ .zero = cacheextents_zero,
+ .extents = cacheextents_extents,
+};
+
+NBDKIT_REGISTER_FILTER (filter)
diff --git a/filters/cacheextents/nbdkit-cacheextents-filter.pod b/filters/cacheextents/nbdkit-cacheextents-filter.pod
new file mode 100644
index 000000000000..88f3173b9375
--- /dev/null
+++ b/filters/cacheextents/nbdkit-cacheextents-filter.pod
@@ -0,0 +1,47 @@
+=head1 NAME
+
+nbdkit-cacheextents-filter - cache extents
+
+=head1 SYNOPSIS
+
+ nbdkit --filter=cacheextents plugin
+
+=head1 DESCRIPTION
+
+C<nbdkit-cacheextents-filter> is a filter that caches the result of last
+extents() call.
+
+A common use for this filter is to accelerate returning extents data for
+clients which ask for extents with the REQ_ONE flag set (like
+S<C<qemu-img convert>>) and is especially useful in combination with
+plugins that report multiple extents in one call, but with high latency
+for each of those calls (like L<nbdkit-vddk-plugin(1)>). For example:
+
+ nbdkit -U - --filter=cacheextents --run 'qemu-img map $nbd' vddk ...
+
+For files with big extents (when it is unlikely for one extents() call
+to return multiple different extents) this does not slow down the
+access.
+
+=head1 PARAMETERS
+
+There are no parameters specific to nbdkit-cacheextents-filter. Any
+parameters are passed through to and processed by the underlying
+plugin in the normal way.
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-cache-filter(1)>,
+L<nbdkit-readahead-filter(1)>,
+L<nbdkit-vddk-plugin(1)>,
+L<nbdkit-filter(3)>,
+L<qemu-img(1)>.
+
+=head1 AUTHORS
+
+Martin Kletzander
+
+=head1 COPYRIGHT
+
+Copyright (C) 2019 Red Hat Inc.
--
2.21.0
5 years, 5 months
[PATCH nbdkit 0/7] server: Implement NBD_FLAG_CAN_MULTI_CONN.
by Richard W.M. Jones
First thing to say is that I need to do a *lot* more testing on this,
so this is just an early peek. In particular, although it passed
‘make check && make check-valgrind’ I have *not* tested it against a
multi-conn-aware client such as the Linux kernel >= 4.9.
This implements NBD_FLAG_CAN_MULTI_CONN, described in the protocol doc
as:
"NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates
entirely without cache, or that the cache it uses is shared among
all connections to the given device. In particular, if this flag is
present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST
be visible across all connections when the server sends its reply to
that command to the client. In the absence of this flag, clients
SHOULD NOT multiplex their commands over more than one connection to
the export."
This is necessary to support the Linux nbd client -C option.
The only plugin which sets the flag so far is file. The ocaml, sh and
nbd plugins allow the flag to be controlled or passed through.
Notable also is that the blocksize filter has to filter out this flag,
because I'm not convinced that the bounce buffer is safe. However I
believe the other filters *are* safe (although not really certain
about the fua filter, and the new cache filter is tricky too).
My feeling is that we should set the flag unconditionally for all
readonly connections, but I can think of nasty corner cases where it
might not work. We should most probably set it in all plugins that
are readonly (eg. nbdkit-pattern-plugin).
Rich.
5 years, 5 months
[nbdkit PATCH] extents: Do not shorten overlaps by 0
by Martin Kletzander
When offset + length == exts->end the code which is shortening the extents is
subtracting 0 from length. And it is not optimized out.
Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com>
---
server/extents.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/server/extents.c b/server/extents.c
index d3d1a15ab97c..c422491601f0 100644
--- a/server/extents.c
+++ b/server/extents.c
@@ -168,7 +168,7 @@ nbdkit_add_extent (struct nbdkit_extents *exts,
return 0;
/* Shorten extents that overlap the end of the range. */
- if (offset + length >= exts->end) {
+ if (offset + length > exts->end) {
overlap = offset + length - exts->end;
length -= overlap;
}
--
2.21.0
5 years, 5 months
[nbdkit PATCH v2 0/2] Bounce buffer cleanups
by Eric Blake
Based on Rich's review of my v1 that touched only cache.c, I have now
tried to bring all three filters with alignment rounding in line with
one another.
There is definitely room for future improvements once we teach nbdkit
to let filters and plugins advertise block sizes, but I'm hoping to
get NBD_CMD_CACHE implemented first.
Eric Blake (2):
blocksize: Process requests in linear order
cache, cow: Reduce use of bounce-buffer
filters/blocksize/blocksize.c | 81 +++++++--------
filters/cache/cache.c | 173 +++++++++++++++++++++++---------
filters/cow/cow.c | 180 +++++++++++++++++++++++++---------
3 files changed, 299 insertions(+), 135 deletions(-)
--
2.20.1
5 years, 5 months
[nbdkit PATCH 0/7] Implement structured replies in nbd plugin
by Eric Blake
I'm hoping to implement .extents for the nbd plugin; this is a
prerequisite. I'm not sure about patch 3 - if we like it, I'll
squash it to 2, if we don't, I think we are okay just dropping
it. I'm also wondering if we have to worry about malicious
plugins that don't populate the entire .pread buffer in an
effort to get nbdkit to expose portions of the heap; my patch 7
loses some time on extra memset()s to avoid that, although it's
probably in the noise compared to the time spent talking over
sockets to the remote server.
Eric Blake (7):
protocol: Send correct structured reply error for non-Linux
protocol: Support EOVERFLOW
RFC: protocol: Only send EOVERFLOW when valid
nbd: Honor server global flags on little-endian
protocol: Add helpers for error response handling
nbd: Implement NBD_OPT_GO client request
nbd: Implement structured replies
common/protocol/protocol.h | 21 +-
plugins/nbd/nbd.c | 434 +++++++++++++++++++++++++++++++++----
server/protocol.c | 16 +-
3 files changed, 422 insertions(+), 49 deletions(-)
--
2.20.1
5 years, 5 months
[nbdkit PATCH 0/9] RFC: implement NBD_CMD_CACHE
by Eric Blake
I'm still working my way through the filters before this series will
be complete, but this is enough of a start to at least get some
feedback on the idea of implementing another NBD protocol extension.
Eric Blake (9):
server: Internal hooks for implementing NBD_CMD_CACHE
plugins: Add .cache callback
file, split: Implement .cache with posix_fadvise
nbd: Implement NBD_CMD_CACHE passthrough
plugins: Implement no-op .cache for in-memory plugins
sh: Implement .cache script callback
filters: Add .cache callback
test-layers: Test .cache usage
blocksize: Implement .cache rounding
docs/nbdkit-filter.pod | 27 ++++++++++++++-
docs/nbdkit-plugin.pod | 52 ++++++++++++++++++++++++++++
docs/nbdkit-protocol.pod | 10 ++++++
plugins/sh/nbdkit-sh-plugin.pod | 12 ++++++-
configure.ac | 3 +-
common/protocol/protocol.h | 2 ++
include/nbdkit-filter.h | 8 +++++
include/nbdkit-plugin.h | 2 ++
server/internal.h | 4 +++
filters/blocksize/blocksize.c | 31 ++++++++++++++++-
plugins/data/data.c | 10 ++++++
plugins/file/file.c | 22 ++++++++++++
plugins/full/full.c | 12 ++++++-
plugins/memory/memory.c | 10 ++++++
plugins/nbd/nbd.c | 22 ++++++++++++
plugins/null/null.c | 12 ++++++-
plugins/pattern/pattern.c | 12 ++++++-
plugins/random/random.c | 12 ++++++-
plugins/sh/sh.c | 40 ++++++++++++++++++++++
plugins/split/split.c | 37 +++++++++++++++++++-
plugins/streaming/streaming.c | 11 ++++++
plugins/zero/zero.c | 12 ++++++-
server/filters.c | 60 ++++++++++++++++++++++++++++++++-
server/plugins.c | 39 +++++++++++++++++++++
server/protocol-handshake.c | 10 +++++-
server/protocol.c | 25 ++++++++++++++
tests/test-layers-filter.c | 22 +++++++++++-
tests/test-layers-plugin.c | 17 ++++++++++
tests/test-layers.c | 36 ++++++++++++++++++++
tests/test-eflags.sh | 53 +++++++++++++++--------------
30 files changed, 587 insertions(+), 38 deletions(-)
--
2.20.1
5 years, 5 months
[nbdkit PATCH] cache: Reduce use of bounce-buffer
by Eric Blake
Although the time spent in memcpy/memset probably pales in comparison
to time spent in socket I/O, it's still worth worth reducing the
number of times we have to utilize a bounce buffer when we already
have aligned data.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
filters/cache/cache.c | 60 ++++++++++++++++++++++++++++---------------
1 file changed, 39 insertions(+), 21 deletions(-)
diff --git a/filters/cache/cache.c b/filters/cache/cache.c
index 19ce555..98786b5 100644
--- a/filters/cache/cache.c
+++ b/filters/cache/cache.c
@@ -1,5 +1,5 @@
/* nbdkit
- * Copyright (C) 2018 Red Hat Inc.
+ * Copyright (C) 2018-2019 Red Hat Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
@@ -58,6 +58,7 @@
#include "cache.h"
#include "blk.h"
#include "reclaim.h"
+#include "isaligned.h"
#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL
@@ -233,11 +234,13 @@ cache_pread (struct nbdkit_next_ops *next_ops, void *nxdata,
CLEANUP_FREE uint8_t *block = NULL;
assert (!flags);
- block = malloc (blksize);
- if (block == NULL) {
- *err = errno;
- nbdkit_error ("malloc: %m");
- return -1;
+ if (!IS_ALIGNED (count | offset, blksize)) {
+ block = malloc (blksize);
+ if (block == NULL) {
+ *err = errno;
+ nbdkit_error ("malloc: %m");
+ return -1;
+ }
}
/* XXX This breaks up large read requests into smaller ones, which
@@ -258,12 +261,14 @@ cache_pread (struct nbdkit_next_ops *next_ops, void *nxdata,
{
ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
- r = blk_read (next_ops, nxdata, blknum, block, err);
+ r = blk_read (next_ops, nxdata, blknum,
+ blkoffs || n < blksize ? block : buf, err);
}
if (r == -1)
return -1;
- memcpy (buf, &block[blkoffs], n);
+ if (blkoffs || n < blksize)
+ memcpy (buf, &block[blkoffs], n);
buf += n;
count -= n;
@@ -282,11 +287,13 @@ cache_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata,
CLEANUP_FREE uint8_t *block = NULL;
bool need_flush = false;
- block = malloc (blksize);
- if (block == NULL) {
- *err = errno;
- nbdkit_error ("malloc: %m");
- return -1;
+ if (!IS_ALIGNED (count | offset, blksize)) {
+ block = malloc (blksize);
+ if (block == NULL) {
+ *err = errno;
+ nbdkit_error ("malloc: %m");
+ return -1;
+ }
}
if ((flags & NBDKIT_FLAG_FUA) &&
@@ -308,11 +315,15 @@ cache_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata,
* Hold the lock over the whole operation.
*/
ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
- r = blk_read (next_ops, nxdata, blknum, block, err);
- if (r != -1) {
- memcpy (&block[blkoffs], buf, n);
- r = blk_write (next_ops, nxdata, blknum, block, flags, err);
+ if (blkoffs || n < blksize) {
+ r = blk_read (next_ops, nxdata, blknum, block, err);
+ if (r != -1) {
+ memcpy (&block[blkoffs], buf, n);
+ r = blk_write (next_ops, nxdata, blknum, block, flags, err);
+ }
}
+ else
+ r = blk_write (next_ops, nxdata, blknum, buf, flags, err);
if (r == -1)
return -1;
@@ -334,6 +345,7 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
{
CLEANUP_FREE uint8_t *block = NULL;
bool need_flush = false;
+ bool clean = false;
block = malloc (blksize);
if (block == NULL) {
@@ -350,7 +362,7 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
}
while (count > 0) {
uint64_t blknum, blkoffs, n;
- int r;
+ int r = 0;
blknum = offset / blksize; /* block number */
blkoffs = offset % blksize; /* offset within the block */
@@ -362,11 +374,17 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
* Hold the lock over the whole operation.
*/
ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
- r = blk_read (next_ops, nxdata, blknum, block, err);
- if (r != -1) {
+ if (blkoffs || n < blksize) {
+ r = blk_read (next_ops, nxdata, blknum, block, err);
memset (&block[blkoffs], 0, n);
+ clean = false;
+ }
+ else if (!clean) {
+ memset (block, 0, blksize);
+ clean = true;
+ }
+ if (r != -1)
r = blk_write (next_ops, nxdata, blknum, block, flags, err);
- }
if (r == -1)
return -1;
--
2.20.1
5 years, 5 months
[PATCH 43 0/7] v2v: switch to ocaml-libvirt
by Pino Toscano
Hi,
this series switches virt-2v to ocaml-libvirt, embedding the latest
version of it from git. This way, it is possible to improve the way
v2v connects to libvirt for both input, and output modules, and
interacts with libvirt (e.g. no more virsh calls needed in virt-v2v).
As side effect, virt-v2v now requires libvirt, as keeping it optional
would create too much burden.
I could not test all the libvirt input modes (like VDDK, and Xen), but
VMware and libvirtxml work fine as before.
Changes from v3:
- rebase on master
- update libvirt-ocaml from its master branch
- use $(srcdir) in common/mllibvirt/Makefile.am
- parse_libvirt_xml: open a libvirt connection only when needed (and
only once)
Changes from v2:
- rebase on master
Changes from v1:
- rebase on master
- update ocaml-libvirt from libvirt-ocaml.git on libvirt.org, and adjust
the code to it
- pass again the URI to input_libvirt_vddk, so an error message is
preserved
Pino Toscano (7):
v2v: require libvirt
common: Bundle the libvirt-ocaml library for use by virt-v2v
v2v: switch to ocaml-libvirt
v2v: -o libvirt: use a Lazy for the connection
v2v: -o libvirt: switch away from virsh
v2v: test-harness: stop using the external ocaml-libvirt
build: stop looking for ocaml-libvirt
.gitignore | 2 +
Makefile.am | 5 +-
common/mllibvirt/Makefile.am | 102 ++
common/mllibvirt/generator.pl | 908 +++++++++++++
common/mllibvirt/libvirt.README | 9 +
common/mllibvirt/libvirt.ml | 1673 ++++++++++++++++++++++++
common/mllibvirt/libvirt.mli | 1647 ++++++++++++++++++++++++
common/mllibvirt/libvirt_c_epilogue.c | 462 +++++++
common/mllibvirt/libvirt_c_oneoffs.c | 1714 +++++++++++++++++++++++++
common/mllibvirt/libvirt_c_prologue.c | 134 ++
configure.ac | 1 +
docs/C_SOURCE_FILES | 1 -
m4/guestfs-ocaml.m4 | 4 -
po/POTFILES | 1 -
v2v/Makefile.am | 26 +-
v2v/copy_to_local.ml | 7 +-
v2v/input_libvirt.ml | 20 +-
v2v/input_libvirt_other.ml | 27 +-
v2v/input_libvirt_other.mli | 5 +-
v2v/input_libvirt_vcenter_https.ml | 13 +-
v2v/input_libvirt_vcenter_https.mli | 2 +-
v2v/input_libvirt_vddk.ml | 15 +-
v2v/input_libvirt_vddk.mli | 4 +-
v2v/input_libvirt_xen_ssh.ml | 13 +-
v2v/input_libvirt_xen_ssh.mli | 2 +-
v2v/libvirt_utils-c.c | 539 --------
v2v/libvirt_utils.ml | 95 +-
v2v/libvirt_utils.mli | 51 +-
v2v/output_libvirt.ml | 56 +-
v2v/parse_libvirt_xml.ml | 23 +-
v2v/parse_libvirt_xml.mli | 11 +-
v2v/test-harness/Makefile.am | 5 +-
v2v/v2v.ml | 12 +-
33 files changed, 6893 insertions(+), 696 deletions(-)
create mode 100644 common/mllibvirt/Makefile.am
create mode 100755 common/mllibvirt/generator.pl
create mode 100644 common/mllibvirt/libvirt.README
create mode 100644 common/mllibvirt/libvirt.ml
create mode 100644 common/mllibvirt/libvirt.mli
create mode 100644 common/mllibvirt/libvirt_c_epilogue.c
create mode 100644 common/mllibvirt/libvirt_c_oneoffs.c
create mode 100644 common/mllibvirt/libvirt_c_prologue.c
delete mode 100644 v2v/libvirt_utils-c.c
--
2.20.1
5 years, 5 months
