August 2019 - Libguestfs - libguestfs List Archives

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones 29 Nov '19

29 Nov '19

My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 18

cross-project patches: Add NBD Fast Zero support
by Eric Blake 04 Sep '19

04 Sep '19

This is a cover letter to a series of patches being proposed in tandem to four different projects: - nbd: Document a new NBD_CMD_FLAG_FAST_ZERO command flag - qemu: Implement the flag for both clients and server - libnbd: Implement the flag for clients - nbdkit: Implement the flag for servers, including the nbd passthrough client If you want to test the patches together, I've pushed a 'fast-zero' branch to each of: https://repo.or.cz/nbd/ericb.git/shortlog/refs/heads/fast-zero https://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/fast-zero https://repo.or.cz/libnbd/ericb.git/shortlog/refs/heads/fast-zero https://repo.or.cz/nbdkit/ericb.git/shortlog/refs/heads/fast-zero I've run several tests to demonstrate why this is useful, as well as prove that because I have multiple interoperable projects, it is worth including in the NBD standard. The original proposal was here: https://lists.debian.org/nbd/2019/03/msg00004.html where I stated: > I will not push this without both: > - a positive review (for example, we may decide that burning another > NBD_FLAG_* is undesirable, and that we should instead have some sort > of NBD_OPT_ handshake for determining when the server supports > NBD_CMF_FLAG_FAST_ZERO) > - a reference client and server implementation (probably both via qemu, > since it was qemu that raised the problem in the first place) Consensus on that thread seemed to be that a new NBD_FLAG was okay; and this thread solves the second bullet of having reference implementations. Here's what I did for testing full-path interoperability: nbdkit memory -> qemu-nbd -> nbdkit nbd -> nbdsh $ nbdkit -p 10810 --filter=nozero --filter=delay memory 1m delay-write=3 zeromode=emulate $ qemu-nbd -p 10811 -f raw nbd://localhost:10810 $ nbdkit -p 10812 nbd nbd://localhost:10811 $ time nbdsh --connect nbd://localhost:10812 -c 'buf = h.zero(512, 0)' # takes more than 3 seconds, but succeeds $ time nbdsh --connect nbd://localhost:10812 -c 'buf = h.zero(512, 0, nbd.CMD_FLAG_FAST_ZERO)' # takes less than 1 second to fail with ENOTSUP And here's some demonstrations on why the feature matters, starting with this qemu thread as justification: https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg06389.html First, I had to create a scenario where falling back to writes is noticeably slower than performing a zero operation, and where pre-zeroing also shows an effect. My choice: let's test 'qemu-img convert' on an image that is half-sparse (every other megabyte is a hole) to an in-memory nbd destination. Then I use a series of nbdkit filters to force the destination to behave in various manners: log logfile=>(sed ...|uniq -c) (track how many normal/fast zero requests the client makes) nozero $params (fine-tune how zero requests behave - the parameters zeromode and fastzeromode are the real drivers of my various tests) blocksize maxdata=256k (allows large zero requests, but forces large writes into smaller chunks, to magnify the effects of write delays and allow testing to provide obvious results with a smaller image) delay delay-write=20ms delay-zero=5ms (also to magnify the effects on a smaller image, with writes penalized more than zeroing) stats statsfile=/dev/stderr (to track overall time and a decent summary of how much I/O occurred). noextents (forces the entire image to report that it is allocated, which eliminates any testing variability based on whether qemu-img uses that to bypass a zeroing operation [1]) So here's my one-time setup, followed by repetitions of the nbdkit command with different parameters to the nozero filter to explore different behaviors. $ qemu-img create -f qcow2 src 100m $ for i in `seq 0 2 99`; do qemu-io -f qcow2 -c "w ${i}m 1m" src; done $ nbdkit -U - --filter=log --filter=nozero --filter=blocksize \ --filter=delay --filter=stats --filter=noextents memory 100m \ logfile=>(sed -n '/Zero.*\.\./ s/.*$fast=.$.*/\1/p' |sort|uniq -c) \ statsfile=/dev/stderr delay-write=20ms delay-zero=5s maxdata=256k \ --run 'qemu-img convert -n -f qcow2 -O raw src $nbd' $params Establish a baseline: when qemu-img does not see write zero support at all (such as when talking to /dev/nbd0, because the kernel NBD implementation still does not support write zeroes), qemu is forced to write the entire disk, including the holes, but doesn't waste any time pre-zeroing or checking block status for whether the disk is zero (the default of the nozero filter is to turn off write zero advertisement): params= elapsed time: 8.54488 s write: 400 ops, 104857600 bytes, 9.81712e+07 bits/s Next, let's emulate what qemu 3.1 was like, with a blind pre-zeroing pass of the entire image without regards to whether that pass is fast or slow. For this test, it was easier to use modern qemu and merely ignore the fast zero bit in nbdkit, but the numbers should be similar when actually using older qemu. If qemu guessed right that pre-zeroing is fast, we see: params='zeromode=plugin fastzeromode=ignore' elapsed time: 4.30183 s write: 200 ops, 52428800 bytes, 9.75005e+07 bits/s zero: 4 ops, 104857600 bytes, 1.95001e+08 bits/s 4 fast=1 which is definite win - instead of having to write the half of the image that was zero on the source, the fast pre-zeroing pass already cleared it (qemu-img currently breaks write zeroes into 32M chunks [1], and thus requires 4 zero requests to pre-zero the image). But if qemu guesses wrong: params='zeromode=emulate fastzeromode=ignore' elapsed time: 12.5065 s write: 600 ops, 157286400 bytes, 1.00611e+08 bits/s 4 fast=1 Ouch - that is actually slower than the case when zeroing is not used at all, because the zeroes turned into writes result in performing double the I/O over the data portions of the file (once during the pre-zero pass, then again during the data). The qemu 3.1 behavior is very bi-polar in nature, and we don't like that. So qemu 4.0 introduced BDRV_REQ_NO_FALLBACK, which qemu uses during the pre-zero request to fail quickly if pre-zeroing is not viable. At the time, NBD did not have a way to support fast zero requests, so qemu blindly assumes that pre-zeroing is not viable over NBD: params='zeromode=emulate fastzeromode=none' elapsed time: 8.32433 s write: 400 ops, 104857600 bytes, 1.00772e+08 bits/s 50 fast=0 When zeroing is slow, our time actually beats the baseline by about 0.2 seconds (although zeroing still turned into writes, the use of zero requests results in less network traffic; you also see that there are 50 zero requests, one per hole, rather than 4 requests for pre-zeroing the image). So we've avoided the pre-zeroing penalty. However: params='zeromode=plugin fastzeromode=none' elapsed time: 4.53951 s write: 200 ops, 52428800 bytes, 9.23955e+07 bits/s zero: 50 ops, 52428800 bytes, 9.23955e+07 bits/s 50 fast=0 when zeroing is fast, we're still 0.2 seconds slower than the pre-zeroing behavior (zeroing runs fast, but one request per hole is still more transactions than pre-zeroing used to use). The qemu 4.0 decision thus regained the worst degradation seen in 3.1 when zeroing is slow, but at a penalty to the case when zeroing is fast. Since guessing is never as nice as knowing, let's repeat the test, but now exploiting the new NBD fast zero: params='zeromode=emulate' elapsed time: 8.41174 s write: 400 ops, 104857600 bytes, 9.9725e+07 bits/s 50 fast=0 1 fast=1 Good: when zeroes are not fast, qemu-img's initial fast-zero request immediately fails, and then it switches back to writing the entire image using regular zeroing for the holes; performance is comparable to the baseline and to the qemu 4.0 behavior. params='zeromode=plugin' elapsed time: 4.31356 s write: 200 ops, 52428800 bytes, 9.72354e+07 bits/s zero: 4 ops, 104857600 bytes, 1.94471e+08 bits/s 4 fast=1 Good: when zeroes are fast, qemu-img is able to use pre-zeroing on the entire image, resulting in fewer zero transactions overall, getting us back to the qemu 3.1 maximum performance (and better than the 4.0 behavior). I hope you enjoyed reading this far, and agree with my interpretation of the numbers about why this feature is useful! [1] Orthogonal to these patches are other ideas I have for improving the NBD protocol in its effects to qemu-img convert, which will result in later cross-project patches: - NBD should have a way to advertise (probably via NBD_INFO_ during NBD_OPT_GO) if the initial image is known to begin life with all zeroes (if that is the case, qemu-img can skip the extents calls and pre-zeroing pass altogether) - improving support to allow NBD to pass larger zero requests (qemu is currently capping zero requests at 32m based on NBD_INFO_BLOCK_SIZE, but could easily go up to ~4G with proper info advertisement of maximum zero request sizing, or if we introduce 64-bit commands to the NBD protocol) Given that NBD extensions need not be present in every server, each orthogonal improvement should be tested in isolation to show that it helps, even though qemu-img will probably use all of the extensions at once when the server supports all of them. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org

5 40

[p2v PATCH 0/3] Small build fixes
by Pino Toscano 02 Sep '19

02 Sep '19

This series for virt-p2v removes the usage of GTK deprecated stuff, and checks for a required Perl module used during build. Tested on: - RHEL 6 (GTK 2) - RHEL 7 (GTK 3) - Fedora 30 (GTK 2 & 3) - Fedora Rawhide/32 (GTK 2 & 3) Pino Toscano (3): Copy GtkAttachOptions from GTK >= 3.4 build: disable deprecated GTK stuff build: require List::MoreUtils Makefile.am | 1 + gui-gtk3-compat.h | 9 +++++++++ m4/p2v-progs.m4 | 8 ++++++++ 3 files changed, 18 insertions(+) -- 2.21.0

2 7

[nbdkit PATCH v2 0/2] caching .can_write
by Eric Blake 31 Aug '19

31 Aug '19

This is a subset of the last half of the larger 9-patch series. The uncontroversial first half of that series is pushed, but here, I tried to reduce the size of the patches by splitting out some of the more complex changes, so that the rest of the changes remaining in the series are more mechanical. In turn, it forced me to write timing tests, which let me spot another spot where we are wasting time, so patch 2 is completely new. As we discussed on IRC, we're early enough in the 1.16 release cycle that I'll probably just push these when I'm happy with them, whether or not a review turns up something (we can always do followup patches). But forcing me to reread the patches for legibility has been a good exercise, because I really like how it forced me to spend a good hour on the commit message of patch 1 explaining things, and how patch 2 then fell out in 10 minutes of hacking to fix the problem identified in passing in the patch 1 commit message. Eric Blake (2): server: Cache per-connection can_write server: Remember .open(readonly) status server/internal.h | 5 ++- server/backend.c | 43 ++++++++++++++++++++- server/connections.c | 5 ++- server/filters.c | 6 +-- server/plugins.c | 4 +- server/protocol-handshake.c | 74 ++++++++++++++++++------------------- 6 files changed, 88 insertions(+), 49 deletions(-) -- 2.21.0

2 3

[nbdkit PATCH v2] filters: Stronger version match requirements
by Eric Blake 31 Aug '19

31 Aug '19

We documented our intent of only allowing a filter to run with the same version of nbdkit it was compiled against, but up to now, were not actually enforcing that - we had only been insisting on the looser notion of a matching ._api_version, which doesn't help when we've forgotten to bump that macro when making incompatible API/ABI changes (see commit 6934d4c1). However, we can't use .version (it was documented as optional, so even though all our in-tree filters set it, a theoretical out-of-tree filter could use NULL; and it has changed offsets in the ABI during previous API changes, such as commit ee61d232). Thus, we have to bump the API version one final time; but now, our API guaarantees enough stable ABI to check the version string as part of our API sanity checking, so we are set to avoid future maintenance snafus. All in-tree filters are affected now that .version is no longer available as an optional member (although it so happens that we are setting the new mandatory ._version to the same value that they all used). And thanks to the previous patch adding nbdkit-version.h, even a theoretical out-of-tree filter will get the correct version string without having to maintain it by hand. Note that the offset of ._version differs between 32- and 64-bit platform ABIs; but on that front, we are safe: Rich and I tested (at least on Linux where ELF executables encode ABI) that attempts to load a 32-bit .so from 64-bit nbdkit, or vice versa, gracefully fail dlopen() for being an incompatible ABI even before we get far enough to worry about inspecting ._api_version or ._version. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- This is what I ended up committing for filter API/ABI sanity checking (a lot nicer than playing ABI games with the unused bytes in ._api_version). docs/nbdkit-filter.pod | 7 ------- server/filters.c | 20 +++++++++++++++----- include/nbdkit-filter.h | 18 ++++++++++-------- filters/blocksize/blocksize.c | 1 - filters/cache/cache.c | 1 - filters/cacheextents/cacheextents.c | 1 - filters/cow/cow.c | 1 - filters/delay/delay.c | 1 - filters/error/error.c | 1 - filters/fua/fua.c | 1 - filters/log/log.c | 1 - filters/nocache/nocache.c | 1 - filters/noextents/noextents.c | 1 - filters/noparallel/noparallel.c | 1 - filters/nozero/nozero.c | 1 - filters/offset/offset.c | 1 - filters/partition/partition.c | 1 - filters/rate/rate.c | 1 - filters/readahead/readahead.c | 1 - filters/stats/stats.c | 1 - filters/truncate/truncate.c | 1 - filters/xz/xz.c | 1 - tests/test-layers-filter.c | 1 - tests/test-cxx-filter.cpp | 1 - 24 files changed, 25 insertions(+), 41 deletions(-) diff --git a/docs/nbdkit-filter.pod b/docs/nbdkit-filter.pod index 0cd3e26a..4bf87295 100644 --- a/docs/nbdkit-filter.pod +++ b/docs/nbdkit-filter.pod @@ -178,13 +178,6 @@ methods. This field (a string) is required, and B<must> contain only ASCII alphanumeric characters and be unique amongst all filters. -=head2 C<.version> - - const char *version; - -Filters may optionally set a version string which is displayed in help -and debugging output. - =head2 C<.longname> const char *longname; diff --git a/server/filters.c b/server/filters.c index c67676a3..0e10816f 100644 --- a/server/filters.c +++ b/server/filters.c @@ -102,7 +102,7 @@ filter_version (struct backend *b) { struct backend_filter *f = container_of (b, struct backend_filter, backend); - return f->filter.version; + return f->filter._version; } static void @@ -730,14 +730,24 @@ filter_register (struct backend *next, size_t index, const char *filename, } /* We do not provide API or ABI guarantees for filters, other than - * the ABI position of _api_version that will let us diagnose - * mismatch when the API changes. + * the ABI position and API contents of _api_version and _version to + * diagnose mismatch from the current nbdkit version. */ if (filter->_api_version != NBDKIT_FILTER_API_VERSION) { fprintf (stderr, "%s: %s: filter is incompatible with this version of nbdkit " - "(_api_version = %d)\n", - program_name, filename, filter->_api_version); + "(_api_version = %d, need %d)\n", + program_name, filename, filter->_api_version, + NBDKIT_FILTER_API_VERSION); + exit (EXIT_FAILURE); + } + if (filter->_version == NULL || + strcmp (filter->_version, PACKAGE_VERSION) != 0) { + fprintf (stderr, + "%s: %s: filter is incompatible with this version of nbdkit " + "(_version = %s, need %s)\n", + program_name, filename, filter->_version ?: "<null>", + PACKAGE_VERSION); exit (EXIT_FAILURE); } diff --git a/include/nbdkit-filter.h b/include/nbdkit-filter.h index 4f8f19ab..8423644e 100644 --- a/include/nbdkit-filter.h +++ b/include/nbdkit-filter.h @@ -43,7 +43,7 @@ extern "C" { #endif -#define NBDKIT_FILTER_API_VERSION 5 /* Corresponding to v1.14 */ +#define NBDKIT_FILTER_API_VERSION 6 /* Corresponding to v1.16+ */ struct nbdkit_extent { uint64_t offset; @@ -93,20 +93,21 @@ struct nbdkit_next_ops { }; struct nbdkit_filter { - /* Do not set this field directly; use NBDKIT_REGISTER_FILTER. - * It exists so that we can diagnose filters compiled against - * one version of the header with a runtime compiled against a - * different version. + /* Do not set these fields directly; use NBDKIT_REGISTER_FILTER. + * They exist so that we can diagnose filters compiled against one + * version of the header with a runtime compiled against a different + * version. As of API version 6, _version is also part of the + * guaranteed ABI, so we no longer have to remember to bump API + * versions regardless of other API/ABI changes later in the struct. */ int _api_version; + const char *_version; /* Because there is no ABI guarantee, new fields may be added where - * logically appropriate, as long as we correctly bump - * NBDKIT_FILTER_API_VERSION once per stable release. + * logically appropriate. */ const char *name; const char *longname; - const char *version; const char *description; void (*load) (void); @@ -179,6 +180,7 @@ struct nbdkit_filter { filter_init (void) \ { \ (filter)._api_version = NBDKIT_FILTER_API_VERSION; \ + (filter)._version = NBDKIT_VERSION_STRING; \ return &(filter); \ } diff --git a/filters/blocksize/blocksize.c b/filters/blocksize/blocksize.c index 0978887f..058c236b 100644 --- a/filters/blocksize/blocksize.c +++ b/filters/blocksize/blocksize.c @@ -399,7 +399,6 @@ blocksize_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "blocksize", .longname = "nbdkit blocksize filter", - .version = PACKAGE_VERSION, .config = blocksize_config, .config_complete = blocksize_config_complete, .config_help = blocksize_config_help, diff --git a/filters/cache/cache.c b/filters/cache/cache.c index b5dbccd2..fb852df9 100644 --- a/filters/cache/cache.c +++ b/filters/cache/cache.c @@ -615,7 +615,6 @@ cache_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "cache", .longname = "nbdkit caching filter", - .version = PACKAGE_VERSION, .load = cache_load, .unload = cache_unload, .config = cache_config, diff --git a/filters/cacheextents/cacheextents.c b/filters/cacheextents/cacheextents.c index b3828f41..e89fe580 100644 --- a/filters/cacheextents/cacheextents.c +++ b/filters/cacheextents/cacheextents.c @@ -192,7 +192,6 @@ cacheextents_zero (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "cacheextents", .longname = "nbdkit cacheextents filter", - .version = PACKAGE_VERSION, .unload = cacheextents_unload, .pwrite = cacheextents_pwrite, .trim = cacheextents_trim, diff --git a/filters/cow/cow.c b/filters/cow/cow.c index 9d91d432..491d0009 100644 --- a/filters/cow/cow.c +++ b/filters/cow/cow.c @@ -482,7 +482,6 @@ cow_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "cow", .longname = "nbdkit copy-on-write (COW) filter", - .version = PACKAGE_VERSION, .load = cow_load, .unload = cow_unload, .open = cow_open, diff --git a/filters/delay/delay.c b/filters/delay/delay.c index c92a12d7..6b13d459 100644 --- a/filters/delay/delay.c +++ b/filters/delay/delay.c @@ -266,7 +266,6 @@ delay_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "delay", .longname = "nbdkit delay filter", - .version = PACKAGE_VERSION, .config = delay_config, .config_help = delay_config_help, .pread = delay_pread, diff --git a/filters/error/error.c b/filters/error/error.c index 968f2837..2d39b251 100644 --- a/filters/error/error.c +++ b/filters/error/error.c @@ -375,7 +375,6 @@ error_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "error", .longname = "nbdkit error filter", - .version = PACKAGE_VERSION, .load = error_load, .unload = error_unload, .config = error_config, diff --git a/filters/fua/fua.c b/filters/fua/fua.c index 9d0e561e..83f7a1a7 100644 --- a/filters/fua/fua.c +++ b/filters/fua/fua.c @@ -233,7 +233,6 @@ fua_zero (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "fua", .longname = "nbdkit fua filter", - .version = PACKAGE_VERSION, .config = fua_config, .config_help = fua_config_help, .prepare = fua_prepare, diff --git a/filters/log/log.c b/filters/log/log.c index 7cf741e1..6d37d583 100644 --- a/filters/log/log.c +++ b/filters/log/log.c @@ -435,7 +435,6 @@ log_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "log", .longname = "nbdkit log filter", - .version = PACKAGE_VERSION, .config = log_config, .config_complete = log_config_complete, .config_help = log_config_help, diff --git a/filters/nocache/nocache.c b/filters/nocache/nocache.c index a3f11984..7ddb84bc 100644 --- a/filters/nocache/nocache.c +++ b/filters/nocache/nocache.c @@ -101,7 +101,6 @@ nocache_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "nocache", .longname = "nbdkit nocache filter", - .version = PACKAGE_VERSION, .config = nocache_config, .config_help = nocache_config_help, .can_cache = nocache_can_cache, diff --git a/filters/noextents/noextents.c b/filters/noextents/noextents.c index e39723cd..e6ac33b2 100644 --- a/filters/noextents/noextents.c +++ b/filters/noextents/noextents.c @@ -44,7 +44,6 @@ noextents_can_extents (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "noextents", .longname = "nbdkit noextents filter", - .version = PACKAGE_VERSION, .can_extents = noextents_can_extents, }; diff --git a/filters/noparallel/noparallel.c b/filters/noparallel/noparallel.c index 057485fa..f9006c06 100644 --- a/filters/noparallel/noparallel.c +++ b/filters/noparallel/noparallel.c @@ -76,7 +76,6 @@ noparallel_thread_model (void) static struct nbdkit_filter filter = { .name = "noparallel", .longname = "nbdkit noparallel filter", - .version = PACKAGE_VERSION, .config = noparallel_config, .config_help = noparallel_config_help, .thread_model = noparallel_thread_model, diff --git a/filters/nozero/nozero.c b/filters/nozero/nozero.c index 964cce9f..9187ec33 100644 --- a/filters/nozero/nozero.c +++ b/filters/nozero/nozero.c @@ -139,7 +139,6 @@ nozero_zero (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "nozero", .longname = "nbdkit nozero filter", - .version = PACKAGE_VERSION, .config = nozero_config, .config_help = nozero_config_help, .prepare = nozero_prepare, diff --git a/filters/offset/offset.c b/filters/offset/offset.c index faae0328..7dc94b1d 100644 --- a/filters/offset/offset.c +++ b/filters/offset/offset.c @@ -175,7 +175,6 @@ offset_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "offset", .longname = "nbdkit offset filter", - .version = PACKAGE_VERSION, .config = offset_config, .config_help = offset_config_help, .get_size = offset_get_size, diff --git a/filters/partition/partition.c b/filters/partition/partition.c index b1b1945d..39132ce8 100644 --- a/filters/partition/partition.c +++ b/filters/partition/partition.c @@ -265,7 +265,6 @@ partition_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "partition", .longname = "nbdkit partition filter", - .version = PACKAGE_VERSION, .config = partition_config, .config_complete = partition_config_complete, .config_help = partition_config_help, diff --git a/filters/rate/rate.c b/filters/rate/rate.c index dca5e9fc..2b105f91 100644 --- a/filters/rate/rate.c +++ b/filters/rate/rate.c @@ -310,7 +310,6 @@ rate_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "rate", .longname = "nbdkit rate filter", - .version = PACKAGE_VERSION, .unload = rate_unload, .config = rate_config, .config_complete = rate_config_complete, diff --git a/filters/readahead/readahead.c b/filters/readahead/readahead.c index b6c18096..3ec73e45 100644 --- a/filters/readahead/readahead.c +++ b/filters/readahead/readahead.c @@ -247,7 +247,6 @@ readahead_zero (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "readahead", .longname = "nbdkit readahead filter", - .version = PACKAGE_VERSION, .unload = readahead_unload, .prepare = readahead_prepare, .get_size = readahead_get_size, diff --git a/filters/stats/stats.c b/filters/stats/stats.c index d8ff02bc..98282e26 100644 --- a/filters/stats/stats.c +++ b/filters/stats/stats.c @@ -290,7 +290,6 @@ stats_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "stats", .longname = "nbdkit stats filter", - .version = PACKAGE_VERSION, .unload = stats_unload, .config = stats_config, .config_complete = stats_config_complete, diff --git a/filters/truncate/truncate.c b/filters/truncate/truncate.c index 93d8f074..ae1f8624 100644 --- a/filters/truncate/truncate.c +++ b/filters/truncate/truncate.c @@ -385,7 +385,6 @@ truncate_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "truncate", .longname = "nbdkit truncate filter", - .version = PACKAGE_VERSION, .config = truncate_config, .config_help = truncate_config_help, .open = truncate_open, diff --git a/filters/xz/xz.c b/filters/xz/xz.c index 51ac919f..78351859 100644 --- a/filters/xz/xz.c +++ b/filters/xz/xz.c @@ -254,7 +254,6 @@ static int xz_thread_model (void) static struct nbdkit_filter filter = { .name = "xz", .longname = "nbdkit XZ filter", - .version = PACKAGE_VERSION, .config = xz_config, .config_help = xz_config_help, .thread_model = xz_thread_model, diff --git a/tests/test-layers-filter.c b/tests/test-layers-filter.c index bd063bdf..465454c0 100644 --- a/tests/test-layers-filter.c +++ b/tests/test-layers-filter.c @@ -261,7 +261,6 @@ test_layers_filter_cache (struct nbdkit_next_ops *next_ops, void *nxdata, static struct nbdkit_filter filter = { .name = "testlayers" layer, - .version = PACKAGE_VERSION, .load = test_layers_filter_load, .unload = test_layers_filter_unload, .config = test_layers_filter_config, diff --git a/tests/test-cxx-filter.cpp b/tests/test-cxx-filter.cpp index ba5ff29e..57299a15 100644 --- a/tests/test-cxx-filter.cpp +++ b/tests/test-cxx-filter.cpp @@ -49,7 +49,6 @@ namespace { nbdkit_filter create_filter() { nbdkit_filter filter = nbdkit_filter (); filter.name = "cxxfilter"; - filter.version = PACKAGE_VERSION; return filter; } } -- 2.21.0

2 1

[PATCH libnbd] Add bindings for Rust language
by Richard W.M. Jones 31 Aug '19

31 Aug '19

Still not working, but I took the latest patch and: - rebased it against libnbd 1.0 - fixed it so it handles new args and cbargs The generator now runs without warnings. This patch doesn't handle optargs at all. In C these are converted to non-optional parameter. Rust doesn't (AFAIK) have optional or labelled arguments unfortunately. Rich.

1 1

[nbdkit PATCH 0/2] RFC: tighter filter versions
by Eric Blake 30 Aug '19

30 Aug '19

This is not intended for v1.14. In fact, we may decide that the second patch is too gross, although the first one still seems like a useful improvement in isolation. I will also point out that all our filters are in-tree, and set the user-controlled field .version to the current release string. We could replace the second patch with a simpler one that just checks ._api_version as an int (as before), but adds the additional requirement that .version match PACKAGE_VERSION from config.h. Then we don't have to hack around ABI changes of the first field member, but are slightly more protected if we forget to bump ._api_version in the future. Eric Blake (2): include: Expose nbdkit version information to public filters: Stronger version match requirements configure.ac | 15 ++++++++++++- server/filters.c | 32 +++++++++++++++++++++----- include/nbdkit-common.h | 2 ++ include/nbdkit-filter.h | 10 ++++----- .gitignore | 1 + include/Makefile.am | 1 + include/nbdkit-version.h.in | 45 +++++++++++++++++++++++++++++++++++++ 7 files changed, 93 insertions(+), 13 deletions(-) create mode 100644 include/nbdkit-version.h.in -- 2.21.0

2 4

[nbdkit PATCH 0/9] can_FOO caching, more filter validation
by Eric Blake 30 Aug '19

30 Aug '19

It's easy to use the sh script to demonstrate that nbdkit is inefficiently calling into .get_size, .can_fua, and friends more than necessary. We've also commented on the list in the past that it would be nice to ensure that when filters call into next_ops, they are not violating constraints (as we've have to fix several bugs in the past where we did not have such checking to protect ourselves). This series fixes both (well, we may still have buggy filters, but if so, now we'll get earlier assertion failures instead of random plugin behavior). Another benefit - this gives us a single point for adding any code for performing NBD_INFO_BLOCKSIZE support, where we'll have to hoist what the blocksize filter is currently doing into something that nbdkit does from any one backend calling into another with different sizing constraints. However, that is not done in this series. My pending work on fast zeroes was easy enough to rebase on top of this (in fact, I wrote this because I noticed that .can_zero was being called more than necessary in implementing .can_fast_zero). Eric Blake (9): server: Fewer dereferences in filter server: Consolidate common backend tasks into new backend.c server: Create new backend_* functions for central handling server: Rework storage of per-backend handle server: Cache per-connection size server: Cache per-connection can_FOO flags filters: Change semantics of .can_zero server: Move fallbacks from protocol.c to backend.c server: Move command validation from protocol.c to backend.c docs/nbdkit-filter.pod | 62 +++- docs/nbdkit-plugin.pod | 6 +- server/internal.h | 108 +++++- server/Makefile.am | 3 +- server/backend.c | 515 +++++++++++++++++++++++++++ server/connections.c | 51 +-- server/filters.c | 379 ++++++-------------- server/main.c | 2 +- server/plugins.c | 192 +++------- server/protocol-handshake-newstyle.c | 20 +- server/protocol-handshake-oldstyle.c | 3 +- server/protocol-handshake.c | 97 +++-- server/protocol.c | 201 +++-------- include/nbdkit-filter.h | 4 + filters/blocksize/blocksize.c | 2 - filters/cache/cache.c | 1 - filters/cow/cow.c | 16 +- filters/nozero/nozero.c | 9 +- tests/test-layers.c | 50 +-- 19 files changed, 963 insertions(+), 758 deletions(-) create mode 100644 server/backend.c -- 2.21.0

2 17

[PATCH] p2v: Ignore gtk2 deprecated declarations.
by Richard W.M. Jones 30 Aug '19

30 Aug '19

This patch is against libguestfs's old copy of p2v, but the same patch ought to also apply against virt-p2v. Rich.

2 2