[PATCH nbdkit] blocksize: Export block size constraints
by Richard W.M. Jones
This filter is a little unusual because it allows clients to send a
wider range of request sizes than the underlying plugin allows.
Therefore we advertise the widest possible minimum and maximum block
size to clients.
We still need to pick a suitable preferred block size assuming the
plugin itself doesn't advertise one.
---
filters/blocksize/blocksize.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/filters/blocksize/blocksize.c b/filters/blocksize/blocksize.c
index a6fa00cb..46e759b0 100644
--- a/filters/blocksize/blocksize.c
+++ b/filters/blocksize/blocksize.c
@@ -156,6 +156,29 @@ blocksize_get_size (nbdkit_next *next,
return ROUND_DOWN (size, minblock);
}
+/* Block size constraints.
+ *
+ * Note that the purpose of this filter is to allow clients to send a
+ * wider range of request sizes than the underlying plugin permits,
+ * and therefore this callback advertises the full availability of the
+ * filter, likely widening the constraints from the plugin.
+ */
+static int
+blocksize_block_size (nbdkit_next *next, void *handle,
+ uint32_t *minimum, uint32_t *preferred, uint32_t *maximum)
+{
+ if (next->block_size (next, minimum, preferred, maximum) == -1)
+ return -1;
+
+ if (*preferred == 0)
+ *preferred = MAX (4096, minblock);
+
+ *minimum = 1;
+ *maximum = 0xffffffff;
+
+ return 0;
+}
+
static int
blocksize_pread (nbdkit_next *next,
void *handle, void *b, uint32_t count, uint64_t offs,
@@ -432,6 +455,7 @@ static struct nbdkit_filter filter = {
.config_complete = blocksize_config_complete,
.config_help = blocksize_config_help,
.get_size = blocksize_get_size,
+ .block_size = blocksize_block_size,
.pread = blocksize_pread,
.pwrite = blocksize_pwrite,
.trim = blocksize_trim,
--
2.35.1
3 years, 1 month
[libnbd PATCH] RFC: api: Add set_request_block_size
by Eric Blake
WIP: I need to finish writing the actual interop-qemu-block-size.sh to
demonstrate scenarios where qemu-nbd advertises a block size > 1 to
clients that request it, but sticks to 1 otherwise (see
https://gitlab.com/qemu-project/qemu/-/blob/master/nbd/server.c#L660).
Our testsuite coverage of nbd_get_block_size() is pretty sparse (a
single line in tests/errors.c, which was skipping until patches to
nbdkit finally made it possible to utilize). But in the process of
adding an interop test with qemu-nbd, I also noticed that qemu-nbd (at
least version 6.2) exposes different block sizes to older clients that
don't request block size than it does to newer clients which promise
to obey block sizes. We still want to request by default, but now we
need a knob, similar to the existing set_full_info(), for overriding
that default for testing purposes.
---
lib/internal.h | 5 +--
generator/API.ml | 57 +++++++++++++++++++++++++++---
generator/states-newstyle-opt-go.c | 20 +++++++----
lib/handle.c | 13 +++++++
interop/Makefile.am | 4 ++-
5 files changed, 85 insertions(+), 14 deletions(-)
diff --git a/lib/internal.h b/lib/internal.h
index 525499a9..3868a7f1 100644
--- a/lib/internal.h
+++ b/lib/internal.h
@@ -120,8 +120,9 @@ struct nbd_handle {
uint8_t opt_current; /* 0 or one of NBD_OPT_* */
struct command_cb opt_cb;
- /* Full info mode. */
- bool full_info;
+ /* Tweak what OPT_INFO/GO requests. */
+ bool request_block_size; /* default true, for INFO_BLOCK_SIZE */
+ bool full_info; /* default false, for INFO_NAME, INFO_DESCRIPTION */
/* Sanitization for pread. */
bool pread_initialize;
diff --git a/generator/API.ml b/generator/API.ml
index e63a10ee..414c6b61 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -395,6 +395,45 @@ "get_export_name", {
Link "get_canonical_export_name"];
};
+ "set_request_block_size", {
+ default_call with
+ args = [Bool "request"]; ret = RErr;
+ permitted_states = [ Created; Negotiating ];
+ shortdesc = "control whether NBD_OPT_GO requests block size";
+ longdesc = "\
+By default, when connecting to an export, libnbd requests that the
+server report any block size restrictions. The NBD protocol states
+that a server may supply block sizes regardless of whether the client
+requests them, and libnbd will report those block sizes (see
+L<nbd_get_block_size(3)>). However, it also states that unless a
+client requested block sizes, the server must also be prepared for the
+client to send requests that do not match the server's constraints;
+thus, requesting block sizes puts a burden on the client to be more
+careful in the requests it makes (libnbd enforces block sizes by
+default, although L<nbd_set_strict_mode(3)> can relax things to
+bypass the NBD protocol requirements). Therefore, there are some
+NBD servers which will report a different block size to clients
+that did not request any than to the clients that promise to abide
+by block sizes. This function makes it possible to disable the
+client block size request, to observe server behavior when there
+is no promise of abiding by block size constraints.
+
+Note that even when block size is requested, the server is not
+obligated to provide any.";
+ see_also = [Link "get_request_block_size"; Link "set_full_info";
+ Link "get_block_size"; Link "set_strict_mode"];
+ };
+
+ "get_request_block_size", {
+ default_call with
+ args = []; ret = RBool;
+ permitted_states = [];
+ shortdesc = "see if NBD_OPT_GO requests block size";
+ longdesc = "\
+Return the state of the block size request flag on this handle.";
+ see_also = [Link "set_request_block_size"];
+ };
+
"set_full_info", {
default_call with
args = [Bool "request"]; ret = RErr;
@@ -413,10 +452,11 @@ "set_full_info", {
Note that even when full info is requested, the server is not
obligated to reply with all information that libnbd requested.
Similarly, libnbd will ignore any optional server information that
-libnbd has not yet been taught to recognize.";
- example = Some "examples/server-flags.c";
+libnbd has not yet been taught to recognize. Furthermore, the
+hint to request block sizes is independently controlled via
+L<nbd_set_request_block_size(3)>.";
see_also = [Link "get_full_info"; Link "get_canonical_export_name";
- Link "get_export_description"];
+ Link "get_export_description"; Link "set_request_block_size"];
};
"get_full_info", {
@@ -1839,9 +1879,13 @@ "get_block_size", {
=back
Future NBD extensions may result in additional C<size_type> values.
+Note that by default, libnbd requests all available block sizes,
+but that a server may differ in what sizes it chooses to report
+if L<nbd_set_request_block_size(3)> alters whether the client
+requests sizes.
"
^ non_blocking_test_call_description;
- see_also = [Link "get_protocol";
+ see_also = [Link "get_protocol"; Link "set_request_block_size";
Link "get_size"; Link "opt_info"]
};
@@ -1976,7 +2020,8 @@ "pread_structured", {
^ strict_call_description;
see_also = [Link "can_df"; Link "pread";
Link "aio_pread_structured"; Link "get_block_size";
- Link "set_strict_mode"; Link "set_pread_initialize"];
+ Link "set_strict_mode"; Link "set_pread_initialize";
+ Link "set_request_block_size"];
};
"pwrite", {
@@ -3201,6 +3246,8 @@ let first_version =
(* Added in 1.11.x development cycle, will be stable and supported in 1.12. *)
"set_pread_initialize", (1, 12);
"get_pread_initialize", (1, 12);
+ "set_request_block_size", (1, 12);
+ "get_request_block_size", (1, 12);
(* These calls are proposed for a future version of libnbd, but
* have not been added to any released version so far.
diff --git a/generator/states-newstyle-opt-go.c b/generator/states-newstyle-opt-go.c
index eed769b3..b60c9eaa 100644
--- a/generator/states-newstyle-opt-go.c
+++ b/generator/states-newstyle-opt-go.c
@@ -1,5 +1,5 @@
/* nbd client library in userspace: state machine
- * Copyright (C) 2013-2020 Red Hat Inc.
+ * Copyright (C) 2013-2022 Red Hat Inc.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
@@ -67,7 +67,12 @@ STATE_MACHINE {
return 0;
NEWSTYLE.OPT_GO.SEND_EXPORT:
- uint16_t nrinfos = h->full_info ? 3 : 1;
+ uint16_t nrinfos = 0;
+
+ if (h->request_block_size)
+ nrinfos++;
+ if (h->full_info)
+ nrinfos += 2;
switch (send_from_wbuf (h)) {
case -1: SET_NEXT_STATE (%.DEAD); return 0;
@@ -80,14 +85,17 @@ STATE_MACHINE {
return 0;
NEWSTYLE.OPT_GO.SEND_NRINFOS:
- uint16_t nrinfos = h->full_info ? 3 : 1;
+ uint16_t nrinfos = 0;
switch (send_from_wbuf (h)) {
case -1: SET_NEXT_STATE (%.DEAD); return 0;
case 0:
- h->sbuf.info[0] = htobe16 (NBD_INFO_BLOCK_SIZE);
- h->sbuf.info[1] = htobe16 (NBD_INFO_NAME);
- h->sbuf.info[2] = htobe16 (NBD_INFO_DESCRIPTION);
+ if (h->request_block_size)
+ h->sbuf.info[nrinfos++] = htobe16 (NBD_INFO_BLOCK_SIZE);
+ if (h->full_info) {
+ h->sbuf.info[nrinfos++] = htobe16 (NBD_INFO_NAME);
+ h->sbuf.info[nrinfos++] = htobe16 (NBD_INFO_DESCRIPTION);
+ }
h->wbuf = &h->sbuf;
h->wlen = sizeof h->sbuf.info[0] * nrinfos;
SET_NEXT_STATE (%SEND_INFO);
diff --git a/lib/handle.c b/lib/handle.c
index 4f00c059..ea2ac3d8 100644
--- a/lib/handle.c
+++ b/lib/handle.c
@@ -242,6 +242,19 @@ nbd_unlocked_get_export_name (struct nbd_handle *h)
return copy;
}
+int
+nbd_unlocked_set_request_block_size (struct nbd_handle *h, bool request)
+{
+ h->request_block_size = request;
+ return 0;
+}
+
+int
+nbd_unlocked_get_request_block_size (struct nbd_handle *h)
+{
+ return h->request_block_size;
+}
+
int
nbd_unlocked_set_full_info (struct nbd_handle *h, bool request)
{
diff --git a/interop/Makefile.am b/interop/Makefile.am
index 56571660..cb7b3234 100644
--- a/interop/Makefile.am
+++ b/interop/Makefile.am
@@ -1,5 +1,5 @@
# nbd client library in userspace
-# Copyright (C) 2013-2021 Red Hat Inc.
+# Copyright (C) 2013-2022 Red Hat Inc.
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -20,6 +20,7 @@ include $(top_srcdir)/subdir-rules.mk
EXTRA_DIST = \
dirty-bitmap.sh \
interop-qemu-storage-daemon.sh \
+ interop-qemu-block-size.sh \
list-exports-nbd-config \
list-exports-test-dir/disk1 \
list-exports-test-dir/disk2 \
@@ -141,6 +142,7 @@ TESTS += \
socket-activation-qemu-nbd \
dirty-bitmap.sh \
structured-read.sh \
+ interop-qemu-block-size.sh \
$(NULL)
interop_qemu_nbd_SOURCES = \
--
2.35.1
3 years, 1 month
[PATCH nbdkit v2 0/7] server: Add new plugin/filter .block_size
by Richard W.M. Jones
v1 was posted here:
https://listman.redhat.com/archives/libguestfs/2022-February/thread.html#...
The changes since v1:
Extra checks for plugin:
* minimum between 1 and 64K
* preferred must be a power of 2
* prefered must be 512 .. 32M
* maximum must be 0xffff_ffff or a multiple of minimum
Same extra checks added to the filter
Add the tls-fallback filter patch. This has to be added early on to
avoid a bisection problem with the tests.
Documentation
* fixed minor typos
* document what "preferred" means wrt read-modify-write
* document returning all zeroes to mean no information
Rename new filter to "nbdkit-blocksize-policy-filter"
Document how to use nbdkit-blocksize-filter + blocksize-policy together
Implement error policy
* add new test to cover it
However I chose _not_ to implement other error policies. We can
extend them in future if required.
Rich.
3 years, 1 month
Getting ready for libnbd 1.12 & nbdkit 1.30 releases
by Richard W.M. Jones
I don't think I announced it, but back in January I thought I would be
doing a new stable release of libnbd (1.12) and nbdkit (1.30). That
didn't happen obviously because once I'd written up the release notes
they seemed a bit thin:
https://libguestfs.org/libnbd-release-notes-1.12.1.html
https://libguestfs.org/nbdkit-release-notes-1.30.1.html
We've done quite a lot on these projects this month, plus it's been
five months since the previous stable releases, so we should aim to do
stable releases by the end of this month or the beginning of March,
ie. just 1-2 weeks away.
Do we have anything that's waiting to get in?
Note that any really big changes should wait until _after_ the release
so they get plenty of time to marinate in the development branch.
I think Nir Soffer's --queue-size feature should go before the
cut-off, subject to Eric Blake approving it.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3 years, 1 month
Re: [Libguestfs] virt-customize
by Richard W.M. Jones
On Fri, Feb 18, 2022 at 07:01:22PM +0100, Lukáš Doktor wrote:
> Hello Rich,
>
> by any chance do you know whether it's possible to escape ':' in "virt-customize" command? I'm trying to upload a file:
>
> --upload '/etc/yum.repos.d/_copr\:copr.devel.redhat.com\:ndokos\:pbench.repo:/etc/yum.repos.d/'
>
> tried various combinations of doubling the ':' or escaping them and so on, but it always considers the first ':' as splitter.
It's not possible because the function always splits on the first ':'...
https://github.com/libguestfs/libguestfs-common/blob/41126802097f0a864cab...
... but there's several workarounds.
* Make a local copy of the file; you can still give the full name when
you upload:
$ cp /etc/yum.repos.d/_copr:copr.devel.redhat.com:ndokos:pbench.repo tmp.repo
$ virt-customize .. --upload tmp.repo:/etc/yum.repos.d/full:name:including:colons.repo
* If the file is small, --write + --move.
* Use --copy-in (if you can copy the whole directory).
* Use guestfish or the API.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3 years, 1 month
[v2v PATCH] docs: highlight that "-ip" with vmx+ssh still requires user interaction
by Laszlo Ersek
The functions "remote_file_exists" and "scp_from_remote_to_temporary"
[input/parse_domain_from_vmx.ml] rely on "ssh" and "scp" shell commands,
which intentionally only take a password from a tty. This means that for
vmx+ssh, "-ip" is incomplete.
Completing the feature would require a lot of work; we'd have to
reimplement "remote_file_exists" and "scp_from_remote_to_temporary" in C,
using libssh (an sftp session). For "remote_file_exists", we'd have to
replace "test -f" with sftp_lstat(). For "scp_from_remote_to_temporary",
we'd need to write an SFTP download loop. vmx+ssh is too niche for this,
so let's just document the limitation.
Extends: 784be60842d088596d7af938f90c689083677dca
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1854275
Signed-off-by: Laszlo Ersek <lersek(a)redhat.com>
---
docs/virt-v2v-input-vmware.pod | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/docs/virt-v2v-input-vmware.pod b/docs/virt-v2v-input-vmware.pod
index f13861339034..599a45cbfb63 100644
--- a/docs/virt-v2v-input-vmware.pod
+++ b/docs/virt-v2v-input-vmware.pod
@@ -139,6 +139,11 @@ virt-v2v server to the ESXi hypervisor. For example:
$ ssh root(a)esxi.example.com
[ logs straight into the shell, no password is requested ]
+Note that support for non-interactive authentication via the I<-ip>
+option is incomplete. Some operations remain that still require the
+user to enter the password manually. Therefore ssh-agent is recommended
+over the I<-ip> option.
+
=head3 VMX: Construct the SSH URI
When using the SSH input transport you must specify a remote
base-commit: f77770eed0e3f1b5e45b63e644b2b0279bf8d420
--
2.19.1.3.g30247aa5d201
3 years, 1 month
