ANNOUNCE: nbdkit 1.30 and libnbd 1.12 released
by Richard W.M. Jones
nbdkit is a Network Block Device (NBD) server with stable plugin ABI
and permissive license. libnbd is an NBD client library.
I'm pleased to announce the latest stable releases of both projects:
nbdkit 1.30.0 and libnbd 1.12.0. You can download both from the
download directories here:
https://download.libguestfs.org/nbdkit/
https://download.libguestfs.org/libnbd/
Release notes are online here and attached below:
https://libguestfs.org/nbdkit-release-notes-1.30.1.html
https://libguestfs.org/libnbd-release-notes-1.12.1.html
Rich.
------------------------------
These are the release notes for nbdkit stable release 1.30. This
describes the major changes since 1.28.
nbdkit 1.30.0 was released on 24 February 2022.
Security
There were no security issues found. All past security issues and
information about how to report new ones can be found in
nbdkit-security(1).
Plugins
nbdkit-streaming-plugin has been removed. It was deprecated in 1.26
and scheduled for removal in this release. If you were using this
plugin, use nbdcopy(1) instead.
nbdkit-vddk-plugin(1) has several changes:
• This plugin can now create (as well as reading and writing) local
VMDK files. See the new "create=true" option and the various
"create-*" options.
• Read and write is now implemented using the VDDK Async functions,
which improves performance. The full nbdkit parallel thread model
is supported.
• VDDK ≥ 6.5 (released Nov 2016) is now the minimum required version.
• Stats collected when using -D vddk.stats=1 have been extended to
include number of calls and bytes transferred.
• --dump-plugin output now includes the VDDK major version and
information about each VDDK API that is implemented by the library.
• A new example scripts/vddk-open.sh is provided to help with
automating connections to remote ESXi virtual machines.
nbdkit-curl-plugin(1) adds support for choosing TLS/SSL versions and
ciphers and TLS 1.3 cipher versions (Przemyslaw Czarnowski).
nbdkit-file-plugin(1) now implements "cache=none" for writes so that
the Linux page cache is not destroyed when writing huge files.
nbdkit-cc-plugin(1) now implements the ".cleanup" callback. Also we
document how to use this plugin to implement OCaml plugin scripts.
nbdkit-info-plugin(1) --dump-plugin option now prints
"info_address=yes" when the platform supports "mode=address".
Filters
New nbdkit-retry-request-filter(1), which is similar to
nbdkit-retry-filter(1) but only retries a single failing request.
New nbdkit-protect-filter(1) lets you add write-protection over regions
of a plugin.
New nbdkit-blocksize-policy-filter(1) lets you adjust or set the block
size constraints and policy of underlying plugins. See "API" below.
nbdkit-cow-filter(1) now permits the COW block size to be set as small
as 4096 bytes.
Debug messages in nbdkit-error-filter(1) are now easier to read because
they no longer all have an "error:" prefix (Eric Blake).
Language bindings
For plugins written in OCaml the minimum version of OCaml is now 4.03
(instead of 4.02.2 previously). Various source-level incompatible
changes were made to OCaml plugins in this release. Please consult the
new documentation and example if writing plugins in OCaml.
OCaml plugins now support "magic_config_key".
Several fixes to the handling of integers in Python plugins (Nir
Soffer).
New Python example which simulates request errors (Nir Soffer).
Server
The server no longer requests the "AI_ADDRCONFIG" hint when opening a
server socket. This improves handling of IPv6. In a related change,
the -i (--ipaddr) option now works as intended, and new -4 and -6
options have been added to restrict the listening socket to IPv4 or
IPv6 addresses (Laszlo Ersek).
API
There is a new ".block_size" callback for plugins and filters. nbdkit
uses this to export block size constraints to clients, specifically the
minimum, preferred and maximum block size that clients should use.
As well as regular C plugins, nbdkit-cc-plugin(3),
nbdkit-eval-plugin(1), nbdkit-ocaml-plugin(3), nbdkit-python-plugin(3)
and nbdkit-sh-plugin(3) support block size constraints.
nbdkit-nbd-plugin(1) reexports block size constraints from the proxied
NBD server. nbdkit-vddk-plugin(1) now advertises a minimum 512 byte
block size.
nbdkit-blocksize-filter(1), nbdkit-cache-filter(1),
nbdkit-cow-filter(1) and nbdkit-swab-filter(1) adjust block size
constraints from plugins (Eric Blake).
nbdkit-blocksize-filter(1) can also use the block size hints from the
plugin instead of needing manual adjustment on the command line.
nbdkit-log-filter(1) logs block size constraints (Eric Blake).
nbdkit-blocksize-policy-filter(1) can be used to add block size
constraints to plugins which don't support it, or adjust constraints,
or set the error policy.
Bug fixes
nbdkit-memory-plugin(1) (and others), using
"allocator=malloc,mlock=true" was broken. This was especially evident
on non-Linux or on Linux with 64K pages (eg. POWER). It should now
work correctly.
Tests
"./configure --disable-libguestfs-tests" can now be used to disable
tests which need libguestfs, without disabling
nbdkit-guestfs-plugin(1).
We now use mke2fs(8) instead of guestfish(1) to create the ext4 test
image. On armv7 this allows the test suite to run much more quickly.
The time taken to run the test suite has been reduced significantly.
Build
Multiple fixes to the Windows (MinGW) build. The test suite should now
pass fully if you have the version of Wine with the unofficial
"AF_UNIX" patch.
The top level bash source directory has been renamed to bash-
completion. This prevents problems when setting $PATH to the source
directory and trying to run bash.
Internals
The performance of the internal vector library has been improved
greatly and overflow is now handled correctly (Nir Soffer, Laszlo Ersek
and Eric Blake).
New "nbdkit-string.h" header file which defines a common string type
(based on vector). Existing places which defined a string based on
vector have been updated to use it.
"MIN" and "MAX" macros can be nested (thanks Eric Blake).
SEE ALSO
nbdkit(1).
AUTHORS
Authors of nbdkit 1.30:
Alan Somers
Eric Blake
Hilko Bengen
Laszlo Ersek
Nir Soffer
Przemyslaw Czarnowski
Richard W.M. Jones
------------------------------
These are the release notes for libnbd stable release 1.12. This
describes the major changes since 1.10.
libnbd 1.12.0 was released on 24 February 2022.
Security
CVE-2022-0485
Silent data corruption when using nbdcopy(1).
See the full announcement here:
https://listman.redhat.com/archives/libguestfs/2022-February/msg00104.html
If you find a security issue, please read SECURITY in the source
(online here: https://gitlab.com/nbdkit/libnbd/blob/master/SECURITY).
To find out about previous security issues in libnbd, see
libnbd-security(3).
New APIs
get_pread_initialize
set_pread_initialize
Control whether libnbd clears the pread buffer to avoid leaking
memory contents if the client does not properly handle errors.
These were added as part of the fix for CVE-2022-0485 (Eric Blake).
get_request_block_size
set_request_block_size
Control whether libnbd requests block size constraints from the
server during negotiation (Eric Blake).
Enhancements to existing APIs
Error messages about incorrect URIs in nbd_connect_uri(3) have been
improved to make common mistakes clearer for the user.
Tools
New syntax: "nbdinfo [ CMD ... ]" allowing you to query the properties
of an NBD server directly.
nbdcopy(1) new --queue-size option to control the maximum size of the
internal buffer.
nbdcopy(1) now names the source and destination handles to make it
easier to understand debug output.
Tests
Adopt libvirt-ci's ci/gitlab.yml (Martin Kletzander).
Fix the OCaml extents example (thanks Laszlo Ersek).
Golang benchmarks were added to the test suite (Nir Soffer).
The large tests/errors.c test was split into about a dozen separate
tests. Also this adds a new test for checking server-side block size
constraint error policy which was missing before.
Other improvements and bug fixes
New OCaml "NBD.with_handle" helper which ensures that "NBD.close" is
always called even if the inner code throws an exception.
The OCaml bindings now consistently use the correct types for buffer
sizes everywhere (Laszlo Ersek).
Several improvements and fixes to the golang bindings and tests. The
golang bindings have been relicensed to LGPLv2+ like the rest of the
library and are now published as a golang module at
https://libguestfs.org/libnbd (Nir Soffer).
The Python bindings no longer crash if you pass "None" in place of a
buffer parameter. In addition some memory leaks were fixed (Eric
Blake).
Various memory leaks have been fixed when using the optional
strict_mode settings (Eric Blake).
The performance of the internal vector library has been improved
greatly and overflow is now handled correctly (Nir Soffer, Laszlo Ersek
and Eric Blake).
Add "simple_copy" and "aio_copy" Golang examples (Nir Soffer).
Error handling was reviewed and fixed across many of the example
programs and documentation (Eric Blake, Nir Soffer).
Simplify and optimize handling of the extents callback in Golang (Nir
Soffer).
Golang AioBuffer was enhanced, making it safer to use, faster and
adding documentation (Nir Soffer).
Documentation
Document the limits on lengths and sizes of parameters to various APIs
(Eric Blake).
Build
You can now build programs that use the OCaml bindings of libnbd
against the libnbd build directory instead of requiring libnbd to be
installed (Laszlo Ersek).
Compatibility with OCaml 4.13.
Line locations in the state machine should be more accurate when
stepping through with a debugger like gdb.
.editorconfig file can be used to help code formatting, see
https://editorconfig.org/ (Nir Soffer, Eric Blake).
"MIN" and "MAX" macros can be nested (thanks Eric Blake).
SEE ALSO
libnbd(3).
AUTHORS
Eric Blake
Laszlo Ersek
Martin Kletzander
Nir Soffer
Richard W.M. Jones
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
2 years, 8 months
[libnbd PATCH v2] api: Add set_request_block_size
by Eric Blake
Our testsuite coverage of nbd_get_block_size() is pretty sparse (the
recent commit 6f5fec2ea uses them in errors-server-unaligned.c for
debug purposes, and even that requires recent patches in nbdkit). But
in the process of adding an interop test with qemu-nbd, I also noticed
that qemu-nbd (at least version 6.2) fails NBD_OPT_INFO for older
clients that don't request block size, and fudges the value to 1 for
NBD_OPT_GO for back-compat reasons. We still want to request by
default, but now we need a knob, similar to the existing
set_full_info(), for overriding our defaults for testing purposes.
---
In v2:
- drop RFC
- fix bugs in implementation (the RFC version passed a wrong size over
the wire, causing 'nbdkit nbd' to deadlock)
- actually implement interop-qemu-block-size.sh
- add test coverage of language bindings
lib/internal.h | 5 +-
generator/API.ml | 52 ++++++++++++--
generator/states-newstyle-opt-go.c | 27 ++++++--
lib/handle.c | 14 ++++
python/t/110-defaults.py | 1 +
python/t/120-set-non-defaults.py | 2 +
ocaml/tests/test_110_defaults.ml | 2 +
ocaml/tests/test_120_set_non_defaults.ml | 3 +
interop/Makefile.am | 4 +-
interop/interop-qemu-block-size.sh | 80 ++++++++++++++++++++++
golang/libnbd_110_defaults_test.go | 8 +++
golang/libnbd_120_set_non_defaults_test.go | 14 +++-
12 files changed, 196 insertions(+), 16 deletions(-)
create mode 100755 interop/interop-qemu-block-size.sh
diff --git a/lib/internal.h b/lib/internal.h
index 525499a9..3868a7f1 100644
--- a/lib/internal.h
+++ b/lib/internal.h
@@ -120,8 +120,9 @@ struct nbd_handle {
uint8_t opt_current; /* 0 or one of NBD_OPT_* */
struct command_cb opt_cb;
- /* Full info mode. */
- bool full_info;
+ /* Tweak what OPT_INFO/GO requests. */
+ bool request_block_size; /* default true, for INFO_BLOCK_SIZE */
+ bool full_info; /* default false, for INFO_NAME, INFO_DESCRIPTION */
/* Sanitization for pread. */
bool pread_initialize;
diff --git a/generator/API.ml b/generator/API.ml
index e63a10ee..3e948aa2 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -395,6 +395,40 @@ "get_export_name", {
Link "get_canonical_export_name"];
};
+ "set_request_block_size", {
+ default_call with
+ args = [Bool "request"]; ret = RErr;
+ permitted_states = [ Created; Negotiating ];
+ shortdesc = "control whether NBD_OPT_GO requests block size";
+ longdesc = "\
+By default, when connecting to an export, libnbd requests that the
+server report any block size restrictions. The NBD protocol states
+that a server may supply block sizes regardless of whether the client
+requests them, and libnbd will report those block sizes (see
+L<nbd_get_block_size(3)>); conversely, if a client does not request
+block sizes, the server may reject the connection instead of dealing
+with a client sending unaligned requests. This function makes it
+possible to test server behavior by emulating older clients.
+
+Note that even when block size is requested, the server is not
+obligated to provide any. Furthermore, if block sizes are provided
+(whether or not the client requested them), libnbd enforces alignment
+to those sizes unless L<nbd_set_strict_mode(3)> is used to bypass
+client-side safety checks.";
+ see_also = [Link "get_request_block_size"; Link "set_full_info";
+ Link "get_block_size"; Link "set_strict_mode"];
+ };
+
+ "get_request_block_size", {
+ default_call with
+ args = []; ret = RBool;
+ permitted_states = [];
+ shortdesc = "see if NBD_OPT_GO requests block size";
+ longdesc = "\
+Return the state of the block size request flag on this handle.";
+ see_also = [Link "set_request_block_size"];
+ };
+
"set_full_info", {
default_call with
args = [Bool "request"]; ret = RErr;
@@ -413,10 +447,11 @@ "set_full_info", {
Note that even when full info is requested, the server is not
obligated to reply with all information that libnbd requested.
Similarly, libnbd will ignore any optional server information that
-libnbd has not yet been taught to recognize.";
- example = Some "examples/server-flags.c";
+libnbd has not yet been taught to recognize. Furthermore, the
+hint to request block sizes is independently controlled via
+L<nbd_set_request_block_size(3)>.";
see_also = [Link "get_full_info"; Link "get_canonical_export_name";
- Link "get_export_description"];
+ Link "get_export_description"; Link "set_request_block_size"];
};
"get_full_info", {
@@ -1839,9 +1874,13 @@ "get_block_size", {
=back
Future NBD extensions may result in additional C<size_type> values.
+Note that by default, libnbd requests all available block sizes,
+but that a server may differ in what sizes it chooses to report
+if L<nbd_set_request_block_size(3)> alters whether the client
+requests sizes.
"
^ non_blocking_test_call_description;
- see_also = [Link "get_protocol";
+ see_also = [Link "get_protocol"; Link "set_request_block_size";
Link "get_size"; Link "opt_info"]
};
@@ -1976,7 +2015,8 @@ "pread_structured", {
^ strict_call_description;
see_also = [Link "can_df"; Link "pread";
Link "aio_pread_structured"; Link "get_block_size";
- Link "set_strict_mode"; Link "set_pread_initialize"];
+ Link "set_strict_mode"; Link "set_pread_initialize";
+ Link "set_request_block_size"];
};
"pwrite", {
@@ -3201,6 +3241,8 @@ let first_version =
(* Added in 1.11.x development cycle, will be stable and supported in 1.12. *)
"set_pread_initialize", (1, 12);
"get_pread_initialize", (1, 12);
+ "set_request_block_size", (1, 12);
+ "get_request_block_size", (1, 12);
(* These calls are proposed for a future version of libnbd, but
* have not been added to any released version so far.
diff --git a/generator/states-newstyle-opt-go.c b/generator/states-newstyle-opt-go.c
index eed769b3..b7354aed 100644
--- a/generator/states-newstyle-opt-go.c
+++ b/generator/states-newstyle-opt-go.c
@@ -1,5 +1,5 @@
/* nbd client library in userspace: state machine
- * Copyright (C) 2013-2020 Red Hat Inc.
+ * Copyright (C) 2013-2022 Red Hat Inc.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
@@ -20,7 +20,12 @@
STATE_MACHINE {
NEWSTYLE.OPT_GO.START:
- uint16_t nrinfos = h->full_info ? 3 : 1;
+ uint16_t nrinfos = 0;
+
+ if (h->request_block_size)
+ nrinfos++;
+ if (h->full_info)
+ nrinfos += 2;
assert (h->gflags & LIBNBD_HANDSHAKE_FLAG_FIXED_NEWSTYLE);
if (h->opt_current == NBD_OPT_INFO)
@@ -67,7 +72,12 @@ STATE_MACHINE {
return 0;
NEWSTYLE.OPT_GO.SEND_EXPORT:
- uint16_t nrinfos = h->full_info ? 3 : 1;
+ uint16_t nrinfos = 0;
+
+ if (h->request_block_size)
+ nrinfos++;
+ if (h->full_info)
+ nrinfos += 2;
switch (send_from_wbuf (h)) {
case -1: SET_NEXT_STATE (%.DEAD); return 0;
@@ -80,14 +90,17 @@ STATE_MACHINE {
return 0;
NEWSTYLE.OPT_GO.SEND_NRINFOS:
- uint16_t nrinfos = h->full_info ? 3 : 1;
+ uint16_t nrinfos = 0;
switch (send_from_wbuf (h)) {
case -1: SET_NEXT_STATE (%.DEAD); return 0;
case 0:
- h->sbuf.info[0] = htobe16 (NBD_INFO_BLOCK_SIZE);
- h->sbuf.info[1] = htobe16 (NBD_INFO_NAME);
- h->sbuf.info[2] = htobe16 (NBD_INFO_DESCRIPTION);
+ if (h->request_block_size)
+ h->sbuf.info[nrinfos++] = htobe16 (NBD_INFO_BLOCK_SIZE);
+ if (h->full_info) {
+ h->sbuf.info[nrinfos++] = htobe16 (NBD_INFO_NAME);
+ h->sbuf.info[nrinfos++] = htobe16 (NBD_INFO_DESCRIPTION);
+ }
h->wbuf = &h->sbuf;
h->wlen = sizeof h->sbuf.info[0] * nrinfos;
SET_NEXT_STATE (%SEND_INFO);
diff --git a/lib/handle.c b/lib/handle.c
index 4f00c059..8713e184 100644
--- a/lib/handle.c
+++ b/lib/handle.c
@@ -64,6 +64,7 @@ nbd_create (void)
h->unique = 1;
h->tls_verify_peer = true;
h->request_sr = true;
+ h->request_block_size = true;
h->pread_initialize = true;
h->uri_allow_transports = LIBNBD_ALLOW_TRANSPORT_MASK;
@@ -242,6 +243,19 @@ nbd_unlocked_get_export_name (struct nbd_handle *h)
return copy;
}
+int
+nbd_unlocked_set_request_block_size (struct nbd_handle *h, bool request)
+{
+ h->request_block_size = request;
+ return 0;
+}
+
+int
+nbd_unlocked_get_request_block_size (struct nbd_handle *h)
+{
+ return h->request_block_size;
+}
+
int
nbd_unlocked_set_full_info (struct nbd_handle *h, bool request)
{
diff --git a/python/t/110-defaults.py b/python/t/110-defaults.py
index a4262dae..749c94f4 100644
--- a/python/t/110-defaults.py
+++ b/python/t/110-defaults.py
@@ -22,6 +22,7 @@ assert h.get_export_name() == ""
assert h.get_full_info() is False
assert h.get_tls() == nbd.TLS_DISABLE
assert h.get_request_structured_replies() is True
+assert h.get_request_block_size() is True
assert h.get_pread_initialize() is True
assert h.get_handshake_flags() == nbd.HANDSHAKE_FLAG_MASK
assert h.get_opt_mode() is False
diff --git a/python/t/120-set-non-defaults.py b/python/t/120-set-non-defaults.py
index e71c6ad0..61a4160c 100644
--- a/python/t/120-set-non-defaults.py
+++ b/python/t/120-set-non-defaults.py
@@ -33,6 +33,8 @@ if h.supports_tls():
assert h.get_tls() == nbd.TLS_ALLOW
h.set_request_structured_replies(False)
assert h.get_request_structured_replies() is False
+h.set_request_block_size(False)
+assert h.get_request_block_size() is False
h.set_pread_initialize(False)
assert h.get_pread_initialize() is False
try:
diff --git a/ocaml/tests/test_110_defaults.ml b/ocaml/tests/test_110_defaults.ml
index 04aa744a..0fdd42e4 100644
--- a/ocaml/tests/test_110_defaults.ml
+++ b/ocaml/tests/test_110_defaults.ml
@@ -28,6 +28,8 @@ let
assert (tls = NBD.TLS.DISABLE);
let sr = NBD.get_request_structured_replies nbd in
assert (sr = true);
+ let bs = NBD.get_request_block_size nbd in
+ assert (bs = true);
let init = NBD.get_pread_initialize nbd in
assert (init = true);
let flags = NBD.get_handshake_flags nbd in
diff --git a/ocaml/tests/test_120_set_non_defaults.ml b/ocaml/tests/test_120_set_non_defaults.ml
index f9498076..28e6b9fc 100644
--- a/ocaml/tests/test_120_set_non_defaults.ml
+++ b/ocaml/tests/test_120_set_non_defaults.ml
@@ -42,6 +42,9 @@ let
NBD.set_request_structured_replies nbd false;
let sr = NBD.get_request_structured_replies nbd in
assert (sr = false);
+ NBD.set_request_block_size nbd false;
+ let bs = NBD.get_request_block_size nbd in
+ assert (bs = false);
NBD.set_pread_initialize nbd false;
let init = NBD.get_pread_initialize nbd in
assert (init = false);
diff --git a/interop/Makefile.am b/interop/Makefile.am
index 56571660..27b1064a 100644
--- a/interop/Makefile.am
+++ b/interop/Makefile.am
@@ -1,5 +1,5 @@
# nbd client library in userspace
-# Copyright (C) 2013-2021 Red Hat Inc.
+# Copyright (C) 2013-2022 Red Hat Inc.
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -20,6 +20,7 @@ include $(top_srcdir)/subdir-rules.mk
EXTRA_DIST = \
dirty-bitmap.sh \
interop-qemu-storage-daemon.sh \
+ interop-qemu-block-size.sh \
list-exports-nbd-config \
list-exports-test-dir/disk1 \
list-exports-test-dir/disk2 \
@@ -141,6 +142,7 @@ TESTS += \
socket-activation-qemu-nbd \
dirty-bitmap.sh \
structured-read.sh \
+ interop-qemu-block-size.sh \
$(NULL)
interop_qemu_nbd_SOURCES = \
diff --git a/interop/interop-qemu-block-size.sh b/interop/interop-qemu-block-size.sh
new file mode 100755
index 00000000..4ce287fd
--- /dev/null
+++ b/interop/interop-qemu-block-size.sh
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+# nbd client library in userspace
+# Copyright (C) 2019-2022 Red Hat Inc.
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+# Test interop with qemu-nbd block sizes.
+
+source ../tests/functions.sh
+set -e
+set -x
+
+requires qemu-nbd --version
+requires nbdsh --version
+requires qemu-img --version
+requires truncate --version
+
+f="qemu-block-size.raw"
+sock=$(mktemp -u /tmp/interop-qemu.XXXXXX)
+rm -f $f $sock
+cleanup_fn rm -f $f $sock
+
+truncate --size=10M $f
+export sock
+fail=0
+
+# run_test FMT REQUEST EXP_INFO EXP_GO
+run_test() {
+ # No -t or -e, so qemu-nbd should exit once nbdsh disconnects
+ qemu-nbd -k $sock $1 $f &
+ pid=$!
+ $VG nbdsh -c - <<EOF || fail=1
+import os
+
+sock = os.environ["sock"]
+
+h.set_opt_mode(True)
+assert h.get_request_block_size()
+h.set_request_block_size($2)
+assert h.get_request_block_size() is $2
+h.connect_unix(sock)
+
+try:
+ h.opt_info()
+ assert h.get_block_size(nbd.SIZE_MINIMUM) == $3
+except nbd.Error:
+ assert $3 == 0
+
+h.opt_go()
+assert h.get_block_size(nbd.SIZE_MINIMUM) == $4
+
+h.shutdown()
+EOF
+ wait $pid || fail=1
+}
+
+# Without '-f raw', qemu-nbd forces sector granularity to prevent writing
+# to sector 0 from changing the disk type. However, if the client does
+# not request block sizes, it reports a size then errors out for NBD_OPT_INFO,
+# while fudging size for NBD_OPT_GO.
+run_test '' True 512 512
+run_test '' False 0 1
+
+# With '-f raw', qemu-nbd always exposes byte-level granularity for files.
+run_test '-f raw' True 1 1
+run_test '-f raw' False 1 1
+
+exit $fail
diff --git a/golang/libnbd_110_defaults_test.go b/golang/libnbd_110_defaults_test.go
index ca7c1c41..f56c9656 100644
--- a/golang/libnbd_110_defaults_test.go
+++ b/golang/libnbd_110_defaults_test.go
@@ -59,6 +59,14 @@ func Test110Defaults(t *testing.T) {
t.Fatalf("unexpected structured replies state")
}
+ bs, err := h.GetRequestBlockSize()
+ if err != nil {
+ t.Fatalf("could not get block size state: %s", err)
+ }
+ if bs != true {
+ t.Fatalf("unexpected block size state")
+ }
+
init, err := h.GetPreadInitialize()
if err != nil {
t.Fatalf("could not get pread initialize state: %s", err)
diff --git a/golang/libnbd_120_set_non_defaults_test.go b/golang/libnbd_120_set_non_defaults_test.go
index 029f0db3..a4c411d0 100644
--- a/golang/libnbd_120_set_non_defaults_test.go
+++ b/golang/libnbd_120_set_non_defaults_test.go
@@ -1,5 +1,5 @@
/* libnbd golang tests
- * Copyright (C) 2013-2021 Red Hat Inc.
+ * Copyright (C) 2013-2022 Red Hat Inc.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
@@ -93,6 +93,18 @@ func Test120SetNonDefaults(t *testing.T) {
t.Fatalf("unexpected structured replies state")
}
+ err = h.SetRequestBlockSize(false)
+ if err != nil {
+ t.Fatalf("could not set block size state: %s", err)
+ }
+ bs, err := h.GetRequestBlockSize()
+ if err != nil {
+ t.Fatalf("could not get block size state: %s", err)
+ }
+ if bs != false {
+ t.Fatalf("unexpected block size state")
+ }
+
err = h.SetPreadInitialize(false)
if err != nil {
t.Fatalf("could not set pread initialize state: %s", err)
--
2.35.1
2 years, 8 months
[nbdkit PATCH v2 0/3] Saner --filter=blocksize defaults
by Eric Blake
In v2:
- add debug statements at end of .config_complete, .prepare, .block_size
- formatting cleanups
- split old patch 1 into two patches: first, a mechanical rename into
per-handle state with globals changed to config_*, second the
deferral of setting defaults until .prepare
- in patch 2, rename test-blocksize-wrap.sh to
test-blocksize-default.sh, and actually test two different exports
from one server
Eric Blake (3):
blocksize: Refactor to per-handle constraints
blocksize: Defer per-handle initialization to .prepare
blocksize: Default internal block size based on plugin
filters/blocksize/nbdkit-blocksize-filter.pod | 14 +-
tests/Makefile.am | 14 +-
filters/blocksize/blocksize.c | 201 ++++++++++++------
tests/test-blocksize-default.sh | 88 ++++++++
tests/test-blocksize.sh | 3 -
5 files changed, 246 insertions(+), 74 deletions(-)
create mode 100755 tests/test-blocksize-default.sh
--
2.35.1
2 years, 8 months
[nbdkit PATCH 0/2] Saner --filter=blocksize defaults
by Eric Blake
Now that we can learn block size limits of the underlying plugin, we
might as well obey them by default in our blocksize filter.
Eric Blake (2):
blocksize: Refactor to per-handle constraints
blocksize: Default internal block size based on plugin
filters/blocksize/nbdkit-blocksize-filter.pod | 14 +-
tests/Makefile.am | 14 +-
filters/blocksize/blocksize.c | 177 +++++++++++++-----
tests/test-blocksize-wrap.sh | 58 ++++++
tests/test-blocksize.sh | 3 -
5 files changed, 205 insertions(+), 61 deletions(-)
create mode 100755 tests/test-blocksize-wrap.sh
--
2.35.1
2 years, 8 months
[nbdkit PATCH] swab: Implement .block_size callback
by Eric Blake
We were previously enforcing minimum block size with EINVAL for
too-small requests. Advertise this to the client.
---
filters/swab/nbdkit-swab-filter.pod | 6 ++++++
filters/swab/swab.c | 24 +++++++++++++++++++++++-
2 files changed, 29 insertions(+), 1 deletion(-)
diff --git a/filters/swab/nbdkit-swab-filter.pod b/filters/swab/nbdkit-swab-filter.pod
index f8500150..030a0852 100644
--- a/filters/swab/nbdkit-swab-filter.pod
+++ b/filters/swab/nbdkit-swab-filter.pod
@@ -34,6 +34,11 @@ the last few bytes, combine this filter with
L<nbdkit-truncate-filter(1)>; fortunately, sector-based disk images
are already suitably sized.
+Note that this filter fails operations that are not aligned to the
+swab-bits boundaries; if you need byte-level access, apply the
+L<nbdkit-blocksize-filter(1)> before this one, to get
+read-modify-write access to individual bytes.
+
=head1 PARAMETERS
=over 4
@@ -90,6 +95,7 @@ L<nbdkit(1)>,
L<nbdkit-file-plugin(1)>,
L<nbdkit-pattern-plugin(1)>,
L<nbdkit-filter(3)>,
+L<nbdkit-blocksize-filter(1)>.
L<nbdkit-truncate-filter(1)>.
=head1 AUTHORS
diff --git a/filters/swab/swab.c b/filters/swab/swab.c
index 68776eee..6e8dc981 100644
--- a/filters/swab/swab.c
+++ b/filters/swab/swab.c
@@ -44,6 +44,7 @@
#include "byte-swapping.h"
#include "isaligned.h"
#include "cleanup.h"
+#include "minmax.h"
#include "rounding.h"
/* Can only be 8 (filter disabled), 16, 32 or 64. */
@@ -85,8 +86,28 @@ swab_get_size (nbdkit_next *next,
return ROUND_DOWN (size, bits/8);
}
+/* Block size constraints. */
+static int
+swab_block_size (nbdkit_next *next, void *handle,
+ uint32_t *minimum, uint32_t *preferred, uint32_t *maximum)
+{
+ if (next->block_size (next, minimum, preferred, maximum) == -1)
+ return -1;
+
+ if (*minimum == 0) { /* No constraints set by the plugin. */
+ *minimum = bits/8;
+ *preferred = 512;
+ *maximum = 0xffffffff;
+ }
+ else {
+ *minimum = MAX (*minimum, bits/8);
+ }
+
+ return 0;
+}
+
/* The request must be aligned.
- * XXX We could lift this restriction with more work.
+ * If you want finer alignment, use the blocksize filter.
*/
static bool
is_aligned (uint32_t count, uint64_t offset, int *err)
@@ -220,6 +241,7 @@ static struct nbdkit_filter filter = {
.config = swab_config,
.config_help = swab_config_help,
.get_size = swab_get_size,
+ .block_size = swab_block_size,
.pread = swab_pread,
.pwrite = swab_pwrite,
.trim = swab_trim,
--
2.35.1
2 years, 8 months
how can I run a subset of the tests?
by Laszlo Ersek
Hi,
"make check" in libguestfs takes very long (especially when it's run
after every patch in a series).
How can I run only those tests that are, for example, in "tests/luks/"?
Thanks,
Laszlo
2 years, 8 months
[PATCH nbdkit] cache, cow: Export block size constraints
by Richard W.M. Jones
Because these filters perform a read-modify-write cycle for requests
which are smaller than the block size of the filter, we can adjust or
set the preferred block size to the block size of the filter or the
preferred block size of the underlying plugin, whichever is larger.
We're careful not to set a preferred block size which is larger than
the maximum block size.
After this change:
$ ./nbdkit --filter=cow floppy /usr/share/doc/nbdkit-devel \
cow-block-size=128k \
--run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS
export="":
export-size: 1458176 (1424K)
content: DOS/MBR boot sector; partition 1 : ID=0xc, start-CHS (0x3ff,254,63), end-CHS (0x3ff,254,63), startsector 2048, 800 sectors
uri: nbd://localhost:10809/
contexts:
base:allocation
is_rotational: false
is_read_only: false
can_cache: true
can_df: true
can_fast_zero: false
can_flush: true
can_fua: true
can_multi_conn: true
can_trim: true
can_zero: false
block_size_minimum: 1
block_size_preferred: 131072 <--- note
block_size_maximum: 4294967295
---
filters/cache/cache.c | 21 +++++++++++++++++++++
filters/cow/cow.c | 21 +++++++++++++++++++++
2 files changed, 42 insertions(+)
diff --git a/filters/cache/cache.c b/filters/cache/cache.c
index c912c5fb..f0ee42f1 100644
--- a/filters/cache/cache.c
+++ b/filters/cache/cache.c
@@ -260,6 +260,26 @@ cache_get_size (nbdkit_next *next,
return size;
}
+/* Block size constraints. */
+static int
+cache_block_size (nbdkit_next *next, void *handle,
+ uint32_t *minimum, uint32_t *preferred, uint32_t *maximum)
+{
+ if (next->block_size (next, minimum, preferred, maximum) == -1)
+ return -1;
+
+ if (*minimum == 0) { /* No constraints set by the plugin. */
+ *minimum = 1;
+ *preferred = blksize;
+ *maximum = 0xffffffff;
+ }
+ else if (*maximum >= blksize) {
+ *preferred = MAX (*preferred, blksize);
+ }
+
+ return 0;
+}
+
/* Force an early call to cache_get_size because we have to set the
* backing file size and bitmap size before any other read or write
* calls.
@@ -716,6 +736,7 @@ static struct nbdkit_filter filter = {
.get_ready = cache_get_ready,
.prepare = cache_prepare,
.get_size = cache_get_size,
+ .block_size = cache_block_size,
.can_cache = cache_can_cache,
.can_fast_zero = cache_can_fast_zero,
.can_flush = cache_can_flush,
diff --git a/filters/cow/cow.c b/filters/cow/cow.c
index e111c60f..cdc5b44f 100644
--- a/filters/cow/cow.c
+++ b/filters/cow/cow.c
@@ -182,6 +182,26 @@ cow_get_size (nbdkit_next *next,
return size;
}
+/* Block size constraints. */
+static int
+cow_block_size (nbdkit_next *next, void *handle,
+ uint32_t *minimum, uint32_t *preferred, uint32_t *maximum)
+{
+ if (next->block_size (next, minimum, preferred, maximum) == -1)
+ return -1;
+
+ if (*minimum == 0) { /* No constraints set by the plugin. */
+ *minimum = 1;
+ *preferred = blksize;
+ *maximum = 0xffffffff;
+ }
+ else if (*maximum >= blksize) {
+ *preferred = MAX (*preferred, blksize);
+ }
+
+ return 0;
+}
+
/* Force an early call to cow_get_size because we have to set the
* backing file size and bitmap size before any other read or write
* calls.
@@ -762,6 +782,7 @@ static struct nbdkit_filter filter = {
.get_ready = cow_get_ready,
.prepare = cow_prepare,
.get_size = cow_get_size,
+ .block_size = cow_block_size,
.can_write = cow_can_write,
.can_flush = cow_can_flush,
.can_trim = cow_can_trim,
--
2.35.1
2 years, 8 months
[nbdkit PATCH] log: Implement .block_size callback
by Eric Blake
Enhance the amount of information logged during connection.
---
filters/log/nbdkit-log-filter.pod | 2 +-
filters/log/log.c | 12 ++++++++----
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/filters/log/nbdkit-log-filter.pod b/filters/log/nbdkit-log-filter.pod
index dd04f1ab..e0a0c209 100644
--- a/filters/log/nbdkit-log-filter.pod
+++ b/filters/log/nbdkit-log-filter.pod
@@ -81,7 +81,7 @@ before performing a single successful read is:
2020-08-06 02:07:23.080415 ListExports id=1 readonly=0 tls=0 ...
2020-08-06 02:07:23.080502 ...ListExports id=1 exports=("") return=0
- 2020-08-06 02:07:23.080712 connection=1 Connect export="" tls=0 size=0x400 write=1 flush=1 rotational=0 trim=1 zero=2 fua=2 extents=1 cache=2 fast_zero=1
+ 2020-08-06 02:07:23.080712 connection=1 Connect export="" tls=0 size=0x400 minsize=0x1 prefsize=0x200 maxsize=0xffffffff write=1 flush=1 rotational=0 trim=1 zero=2 fua=2 extents=1 cache=2 fast_zero=1
2020-08-06 02:07:23.080907 connection=1 Read id=1 offset=0x0 count=0x200 ...
2020-08-06 02:07:23.080927 connection=1 ...Read id=1 return=0
2020-08-06 02:07:23.081255 connection=1 Disconnect transactions=1
diff --git a/filters/log/log.c b/filters/log/log.c
index acc9d77e..4255f85f 100644
--- a/filters/log/log.c
+++ b/filters/log/log.c
@@ -1,5 +1,5 @@
/* nbdkit
- * Copyright (C) 2018-2020 Red Hat Inc.
+ * Copyright (C) 2018-2022 Red Hat Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
@@ -251,6 +251,7 @@ log_prepare (nbdkit_next *next, void *handle,
struct handle *h = handle;
const char *exportname = h->exportname;
int64_t size = next->get_size (next);
+ uint32_t minsize, prefsize, maxsize;
int w = next->can_write (next);
int f = next->can_flush (next);
int r = next->is_rotational (next);
@@ -260,9 +261,10 @@ log_prepare (nbdkit_next *next, void *handle,
int e = next->can_extents (next);
int c = next->can_cache (next);
int Z = next->can_fast_zero (next);
+ int s = next->block_size (next, &minsize, &prefsize, &maxsize);
if (size < 0 || w < 0 || f < 0 || r < 0 || t < 0 || z < 0 || F < 0 ||
- e < 0 || c < 0 || Z < 0)
+ e < 0 || c < 0 || Z < 0 || s < 0)
return -1;
fp = open_memstream (&str, &len);
@@ -270,10 +272,12 @@ log_prepare (nbdkit_next *next, void *handle,
fprintf (fp, "export=");
shell_quote (exportname, fp);
fprintf (fp,
- " tls=%d size=0x%" PRIx64 " write=%d "
+ " tls=%d size=0x%" PRIx64 " minsize=0x%" PRIx32 " prefsize=0x%"
+ PRIx32 " maxsize=0x%" PRIx32 " write=%d "
"flush=%d rotational=%d trim=%d zero=%d fua=%d extents=%d "
"cache=%d fast_zero=%d",
- h->tls, size, w, f, r, t, z, F, e, c, Z);
+ h->tls, size, minsize, prefsize, maxsize,
+ w, f, r, t, z, F, e, c, Z);
fclose (fp);
print (h, "Connect", "%s", str);
}
--
2.35.1
2 years, 8 months
