[libnbd PATCH v3 00/22] NBD 64-bit extensions (libnbd portion)
v2 was here: https://listman.redhat.com/archives/libguestfs/2022-November/030292.html For v3, there are now approved specs to code to: https://listman.redhat.com/archives/libguestfs/2023-April/031251.html visible on an upstream branch at https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-header/doc/prot... (upstream leaves extensions in a branch state until there are multiple implementations that comply with that extension) as well as interoperability with counterpart qemu patches: https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg03607.html also available at https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/exthdr-v3 I'm still working on adding extended header support into nbdkit, which will involve adding a v3 protocol for plugins; that work is probably much further out. Changes since v2: 001/22:[0002] [FC] 'block_status: Refactor array storage' 002/22:[0016] [FC] 'internal: Refactor layout of replies in sbuf' 003/22:[0002] [FC] 'protocol: Add definitions for extended headers' 004/22:[0006] [FC] 'states: Prepare to send 64-bit requests' 005/22:[0013] [FC] 'states: Prepare to receive 64-bit replies' 006/22:[0007] [FC] 'states: Break deadlock if server goofs on extended replies' 007/22:[0021] [FC] 'generator: Add struct nbd_extent in prep for 64-bit extents' 008/22:[----] [-C] 'block_status: Track 64-bit extents internally' 009/22:[0012] [FC] 'block_status: Accept 64-bit extents during block status' 010/22:[0063] [FC] 'api: Add [aio_]nbd_block_status_64' 011/22:[0006] [FC] 'api: Add several functions for controlling extended headers' 012/22:[----] [--] 'copy: Update nbdcopy to use 64-bit block status' 013/22:[0004] [FC] 'dump: Update nbddump to use 64-bit block status' 014/22:[----] [--] 'info: Expose extended-headers support through nbdinfo' 015/22:[0006] [FC] 'info: Update nbdinfo --map to use 64-bit block status' 016/22:[----] [-C] 'examples: Update copy-libev to use 64-bit block status' 017/22:[0002] [FC] 'ocaml: Add example for 64-bit extents' 018/22:[0004] [FC] 'generator: Actually request extended headers' 019/22:[0008] [FC] 'api: Add nbd_[aio_]opt_extended_headers()' 020/22:[0004] [FC] 'interop: Add test of 64-bit block status' 021/22:[0011] [FC] 'api: Add nbd_can_block_status_payload()' 022/22:[0004] [FC] 'api: Add nbd_[aio_]block_status_filter()' The changes are mostly fallout from rebasing on top of style cleanups that Laszlo has been working on, but also deal with some changes with the spec compared to how it was proposed in v2 (for example, upstream decided that for NBD_CMD_BLOCK_STATUS, the server should always use 64-bit replies, rather than making the client support both 32- and 64-bit replies). Eric Blake (22): block_status: Refactor array storage internal: Refactor layout of replies in sbuf protocol: Add definitions for extended headers states: Prepare to send 64-bit requests states: Prepare to receive 64-bit replies states: Break deadlock if server goofs on extended replies generator: Add struct nbd_extent in prep for 64-bit extents block_status: Track 64-bit extents internally block_status: Accept 64-bit extents during block status api: Add [aio_]nbd_block_status_64 api: Add several functions for controlling extended headers copy: Update nbdcopy to use 64-bit block status dump: Update nbddump to use 64-bit block status info: Expose extended-headers support through nbdinfo info: Update nbdinfo --map to use 64-bit block status examples: Update copy-libev to use 64-bit block status ocaml: Add example for 64-bit extents generator: Actually request extended headers api: Add nbd_[aio_]opt_extended_headers() interop: Add test of 64-bit block status api: Add nbd_can_block_status_payload() api: Add nbd_[aio_]block_status_filter() docs/libnbd.pod | 18 +- info/nbdinfo.pod | 21 +- sh/nbdsh.pod | 2 +- lib/internal.h | 41 +- lib/nbd-protocol.h | 112 +++- generator/API.mli | 1 + generator/API.ml | 534 +++++++++++++++--- generator/C.ml | 24 +- generator/GoLang.ml | 24 + generator/Makefile.am | 1 + generator/OCaml.ml | 23 +- generator/Python.ml | 20 +- generator/state_machine.ml | 50 +- generator/states-issue-command.c | 33 +- .../states-newstyle-opt-extended-headers.c | 110 ++++ generator/states-newstyle-opt-starttls.c | 7 +- .../states-newstyle-opt-structured-reply.c | 3 +- generator/states-newstyle.c | 3 + generator/states-reply-simple.c | 4 +- generator/states-reply-structured.c | 253 ++++++--- generator/states-reply.c | 69 ++- lib/aio.c | 7 +- lib/flags.c | 12 + lib/handle.c | 25 +- lib/opt.c | 44 ++ lib/rw.c | 250 +++++++- python/t/110-defaults.py | 1 + python/t/120-set-non-defaults.py | 2 + python/t/465-block-status-64.py | 56 ++ ocaml/examples/Makefile.am | 1 + ocaml/examples/extents64.ml | 42 ++ ocaml/helpers.c | 20 + ocaml/nbd-c.h | 1 + ocaml/tests/Makefile.am | 1 + ocaml/tests/test_110_defaults.ml | 2 + ocaml/tests/test_120_set_non_defaults.ml | 3 + ocaml/tests/test_465_block_status_64.ml | 58 ++ tests/Makefile.am | 4 + tests/meta-base-allocation.c | 104 +++- tests/pwrite-extended.c | 112 ++++ examples/copy-libev.c | 21 +- examples/server-flags.c | 7 +- interop/Makefile.am | 18 + interop/block-status-payload.c | 241 ++++++++ interop/block-status-payload.sh | 80 +++ interop/large-status.c | 186 ++++++ interop/large-status.sh | 49 ++ interop/opt-extended-headers.c | 153 +++++ interop/opt-extended-headers.sh | 29 + .gitignore | 4 + copy/nbd-ops.c | 22 +- dump/dump.c | 27 +- fuzzing/libnbd-fuzz-wrapper.c | 20 +- golang/Makefile.am | 1 + golang/handle.go | 6 + golang/libnbd_110_defaults_test.go | 8 + golang/libnbd_120_set_non_defaults_test.go | 12 + golang/libnbd_465_block_status_64_test.go | 119 ++++ info/can.c | 14 + info/info-can.sh | 30 + info/info-packets.sh | 17 +- info/main.c | 7 +- info/map.c | 65 ++- info/show.c | 9 +- 64 files changed, 2892 insertions(+), 351 deletions(-) create mode 100644 generator/states-newstyle-opt-extended-headers.c create mode 100644 python/t/465-block-status-64.py create mode 100644 ocaml/examples/extents64.ml create mode 100644 ocaml/tests/test_465_block_status_64.ml create mode 100644 tests/pwrite-extended.c create mode 100644 interop/block-status-payload.c create mode 100755 interop/block-status-payload.sh create mode 100644 interop/large-status.c create mode 100755 interop/large-status.sh create mode 100644 interop/opt-extended-headers.c create mode 100755 interop/opt-extended-headers.sh create mode 100644 golang/libnbd_465_block_status_64_test.go base-commit: 17d0bc6a71bbc67f3d7707d129bec4acf7e7a382 -- 2.40.1
For 32-bit block status, we were able to cheat and use an array with an odd number of elements, with array[0] holding the context id, and passing &array[1] to the user's callback. But once we have 64-bit extents, we can no longer abuse array element 0 like that, for two reasons: 64-bit extents contain uint64_t which might not be alignment-compatible with an array of uint32_t on all architectures, and the new NBD_REPLY_TYPE_BLOCK_STATUS_EXT adds an additional count field before the array. Split out a new state STRUCTURED_REPLY.BS_HEADER to receive the context id (and eventually the new count field for 64-bit replies) separately from the extents array, and add another structured_reply type in the payload section for tracking it. No behavioral change, other than the rare possibility of landing in the new state. Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 1 + lib/nbd-protocol.h | 17 +++++---- generator/state_machine.ml | 9 ++++- generator/states-reply-structured.c | 56 ++++++++++++++++++++--------- 4 files changed, 60 insertions(+), 23 deletions(-) diff --git a/lib/internal.h b/lib/internal.h index b155681d..25eeea34 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -242,6 +242,7 @@ struct nbd_handle { union { struct nbd_structured_reply_offset_data offset_data; struct nbd_structured_reply_offset_hole offset_hole; + struct nbd_structured_reply_block_status_hdr bs_hdr; struct { struct nbd_structured_reply_error error; char msg[NBD_MAX_STRING]; /* Common to all error types */ diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h index 0217891e..f4640a98 100644 --- a/lib/nbd-protocol.h +++ b/lib/nbd-protocol.h @@ -182,12 +182,6 @@ struct nbd_fixed_new_option_reply_meta_context { /* followed by a string */ } NBD_ATTRIBUTE_PACKED; -/* NBD_REPLY_TYPE_BLOCK_STATUS block descriptor. */ -struct nbd_block_descriptor { - uint32_t length; /* length of block */ - uint32_t status_flags; /* block type (hole etc) */ -} NBD_ATTRIBUTE_PACKED; - /* Request (client -> server). */ struct nbd_request { uint32_t magic; /* NBD_REQUEST_MAGIC. */ @@ -224,6 +218,17 @@ struct nbd_structured_reply_offset_hole { uint32_t length; /* Length of hole. */ } NBD_ATTRIBUTE_PACKED; +/* NBD_REPLY_TYPE_BLOCK_STATUS block descriptor. */ +struct nbd_block_descriptor { + uint32_t length; /* length of block */ + uint32_t status_flags; /* block type (hole etc) */ +} NBD_ATTRIBUTE_PACKED; + +struct nbd_structured_reply_block_status_hdr { + uint32_t context_id; /* metadata context ID */ + /* followed by array of nbd_block_descriptor extents */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_error { uint32_t error; /* NBD_E* error number */ uint16_t len; /* Length of human readable error. */ diff --git a/generator/state_machine.ml b/generator/state_machine.ml index 126159b9..1f0d00b0 100644 --- a/generator/state_machine.ml +++ b/generator/state_machine.ml @@ -871,10 +871,17 @@ and external_events = []; }; + State { + default_state with + name = "RECV_BS_HEADER"; + comment = "Receive header of structured reply block-status payload"; + external_events = []; + }; + State { default_state with name = "RECV_BS_ENTRIES"; - comment = "Receive a structured reply block-status payload"; + comment = "Receive entries array of structured reply block-status payload"; external_events = []; }; diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 5aca7262..96182222 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -126,19 +126,10 @@ REPLY.STRUCTURED_REPLY.CHECK: length < 12 || ((length-4) & 7) != 0) goto resync; assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); - /* We read the context ID followed by all the entries into a - * single array and deal with it at the end. - */ - free (h->bs_entries); - h->bs_entries = malloc (length); - if (h->bs_entries == NULL) { - SET_NEXT_STATE (%.DEAD); - set_error (errno, "malloc"); - break; - } - h->rbuf = h->bs_entries; - h->rlen = length; - SET_NEXT_STATE (%RECV_BS_ENTRIES); + /* Start by reading the context ID. */ + h->rbuf = &h->sbuf.sr.payload.bs_hdr; + h->rlen = sizeof h->sbuf.sr.payload.bs_hdr; + SET_NEXT_STATE (%RECV_BS_HEADER); break; default: @@ -424,9 +415,41 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_HOLE: } return 0; + REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: + struct command *cmd = h->reply_cmd; + uint32_t length; + + switch (recv_into_rbuf (h)) { + case -1: SET_NEXT_STATE (%.DEAD); return 0; + case 1: + save_reply_state (h); + SET_NEXT_STATE (%.READY); + return 0; + case 0: + length = be32toh (h->sbuf.sr.structured_reply.length); + + assert (cmd); /* guaranteed by CHECK */ + assert (cmd->type == NBD_CMD_BLOCK_STATUS); + assert (length >= 12); + length -= sizeof h->sbuf.sr.payload.bs_hdr; + + free (h->bs_entries); + h->bs_entries = malloc (length); + if (h->bs_entries == NULL) { + SET_NEXT_STATE (%.DEAD); + set_error (errno, "malloc"); + return 0; + } + h->rbuf = h->bs_entries; + h->rlen = length; + SET_NEXT_STATE (%RECV_BS_ENTRIES); + } + return 0; + REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: struct command *cmd = h->reply_cmd; uint32_t length; + uint32_t count; size_t i; uint32_t context_id; @@ -445,15 +468,16 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (h->bs_entries); assert (length >= 12); assert (h->meta_valid); + count = (length - sizeof h->sbuf.sr.payload.bs_hdr) / sizeof *h->bs_entries; /* Need to byte-swap the entries returned, but apart from that we * don't validate them. */ - for (i = 0; i < length/4; ++i) + for (i = 0; i < count; ++i) h->bs_entries[i] = be32toh (h->bs_entries[i]); /* Look up the context ID. */ - context_id = h->bs_entries[0]; + context_id = be32toh (h->sbuf.sr.payload.bs_hdr.context_id); for (i = 0; i < h->meta_contexts.len; ++i) if (context_id == h->meta_contexts.ptr[i].context_id) break; @@ -464,7 +488,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: if (CALL_CALLBACK (cmd->cb.fn.extent, h->meta_contexts.ptr[i].name, cmd->offset, - &h->bs_entries[1], (length-4) / 4, + h->bs_entries, count, &error) == -1) if (cmd->error == 0) cmd->error = error ? error : EPROTO; -- 2.40.1
On 5/25/23 15:00, Eric Blake wrote:
For 32-bit block status, we were able to cheat and use an array with an odd number of elements, with array[0] holding the context id, and passing &array[1] to the user's callback. But once we have 64-bit extents, we can no longer abuse array element 0 like that, for two reasons: 64-bit extents contain uint64_t which might not be alignment-compatible with an array of uint32_t on all architectures, and the new NBD_REPLY_TYPE_BLOCK_STATUS_EXT adds an additional count field before the array.
Split out a new state STRUCTURED_REPLY.BS_HEADER to receive the context id (and eventually the new count field for 64-bit replies) separately from the extents array, and add another structured_reply type in the payload section for tracking it. No behavioral change, other than the rare possibility of landing in the new state.
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 1 + lib/nbd-protocol.h | 17 +++++---- generator/state_machine.ml | 9 ++++- generator/states-reply-structured.c | 56 ++++++++++++++++++++--------- 4 files changed, 60 insertions(+), 23 deletions(-)
diff --git a/lib/internal.h b/lib/internal.h index b155681d..25eeea34 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -242,6 +242,7 @@ struct nbd_handle { union { struct nbd_structured_reply_offset_data offset_data; struct nbd_structured_reply_offset_hole offset_hole; + struct nbd_structured_reply_block_status_hdr bs_hdr; struct { struct nbd_structured_reply_error error; char msg[NBD_MAX_STRING]; /* Common to all error types */ diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h index 0217891e..f4640a98 100644 --- a/lib/nbd-protocol.h +++ b/lib/nbd-protocol.h @@ -182,12 +182,6 @@ struct nbd_fixed_new_option_reply_meta_context { /* followed by a string */ } NBD_ATTRIBUTE_PACKED;
-/* NBD_REPLY_TYPE_BLOCK_STATUS block descriptor. */ -struct nbd_block_descriptor { - uint32_t length; /* length of block */ - uint32_t status_flags; /* block type (hole etc) */ -} NBD_ATTRIBUTE_PACKED; - /* Request (client -> server). */ struct nbd_request { uint32_t magic; /* NBD_REQUEST_MAGIC. */ @@ -224,6 +218,17 @@ struct nbd_structured_reply_offset_hole { uint32_t length; /* Length of hole. */ } NBD_ATTRIBUTE_PACKED;
+/* NBD_REPLY_TYPE_BLOCK_STATUS block descriptor. */ +struct nbd_block_descriptor { + uint32_t length; /* length of block */ + uint32_t status_flags; /* block type (hole etc) */ +} NBD_ATTRIBUTE_PACKED; + +struct nbd_structured_reply_block_status_hdr { + uint32_t context_id; /* metadata context ID */ + /* followed by array of nbd_block_descriptor extents */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_error { uint32_t error; /* NBD_E* error number */ uint16_t len; /* Length of human readable error. */ diff --git a/generator/state_machine.ml b/generator/state_machine.ml index 126159b9..1f0d00b0 100644 --- a/generator/state_machine.ml +++ b/generator/state_machine.ml @@ -871,10 +871,17 @@ and external_events = []; };
+ State { + default_state with + name = "RECV_BS_HEADER"; + comment = "Receive header of structured reply block-status payload"; + external_events = []; + }; + State { default_state with name = "RECV_BS_ENTRIES"; - comment = "Receive a structured reply block-status payload"; + comment = "Receive entries array of structured reply block-status payload"; external_events = []; };
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 5aca7262..96182222 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -126,19 +126,10 @@ REPLY.STRUCTURED_REPLY.CHECK: length < 12 || ((length-4) & 7) != 0)
This is important (the context doesn't show it in full): we're under NBD_REPLY_TYPE_BLOCK_STATUS here (nested under REPLY.STRUCTURED_REPLY.CHECK), and we enforce that length = be32toh (h->sbuf.sr.structured_reply.length); contains the context_id (4 bytes), plus a positive integral number of block descriptor structures (8 bytes each).
goto resync; assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); - /* We read the context ID followed by all the entries into a - * single array and deal with it at the end. - */ - free (h->bs_entries); - h->bs_entries = malloc (length); - if (h->bs_entries == NULL) { - SET_NEXT_STATE (%.DEAD); - set_error (errno, "malloc"); - break; - } - h->rbuf = h->bs_entries; - h->rlen = length; - SET_NEXT_STATE (%RECV_BS_ENTRIES); + /* Start by reading the context ID. */ + h->rbuf = &h->sbuf.sr.payload.bs_hdr; + h->rlen = sizeof h->sbuf.sr.payload.bs_hdr; + SET_NEXT_STATE (%RECV_BS_HEADER); break;
default:
Makes sense.
@@ -424,9 +415,41 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_HOLE: } return 0;
+ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: + struct command *cmd = h->reply_cmd; + uint32_t length; + + switch (recv_into_rbuf (h)) { + case -1: SET_NEXT_STATE (%.DEAD); return 0; + case 1: + save_reply_state (h); + SET_NEXT_STATE (%.READY); + return 0; + case 0: + length = be32toh (h->sbuf.sr.structured_reply.length); + + assert (cmd); /* guaranteed by CHECK */ + assert (cmd->type == NBD_CMD_BLOCK_STATUS); + assert (length >= 12);
All of this matches what we have under REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES *pre-patch*, so OK. (Also matches what recv_into_rbuf() does.) We've not copied the CALLBACK_IS_NOT_NULL() assertion or the h->meta_valid assertion over from RECV_BS_ENTRIES -- are we saying that we're going to assert those later anyway, in RECV_BS_ENTRIES? Sounds OK.
+ length -= sizeof h->sbuf.sr.payload.bs_hdr;
Seems OK, relying on the assertion just above, which in turn relies on the explicit (length < 12) catch under REPLY.STRUCTURED_REPLY.CHECK.
+ + free (h->bs_entries); + h->bs_entries = malloc (length); + if (h->bs_entries == NULL) { + SET_NEXT_STATE (%.DEAD); + set_error (errno, "malloc"); + return 0; + } + h->rbuf = h->bs_entries; + h->rlen = length; + SET_NEXT_STATE (%RECV_BS_ENTRIES); + } + return 0; +
This preparation for RECV_BS_ENTRIES is being moved from REPLY.STRUCTURED_REPLY.CHECK; seems OK. (Importantly, after storing the "length" we've just decreased by 4 bytes into "h->rlen", "length" ceases to exist, so we have to re-fetch it in the next state handler, below. Not shown in the context, but it's being done, under "case 0".)
REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: struct command *cmd = h->reply_cmd; uint32_t length; + uint32_t count; size_t i; uint32_t context_id;
@@ -445,15 +468,16 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (h->bs_entries); assert (length >= 12); assert (h->meta_valid); + count = (length - sizeof h->sbuf.sr.payload.bs_hdr) / sizeof *h->bs_entries;
I have a slight problem with the pre-patch code here. We keep the existent assertions (good), but I think the pre-patch RECV_BS_ENTRIES code misses an assertion. Namely, after the size check (i.e., 12+ bytes), the pre-patch code should have said assert ((length-4) & 7) == 0); emphasizing the explicit check under REPLY.STRUCTURED_REPLY.CHECK. The pre-patch code relies on this (a) silently by expecting (length/4) to be an integer (in the mathematical sense), and (b) very silently by expecting (length/4) to be an *odd* integer >= 3.
/* Need to byte-swap the entries returned, but apart from that we * don't validate them. */ - for (i = 0; i < length/4; ++i) + for (i = 0; i < count; ++i) h->bs_entries[i] = be32toh (h->bs_entries[i]);
/* Look up the context ID. */ - context_id = h->bs_entries[0]; + context_id = be32toh (h->sbuf.sr.payload.bs_hdr.context_id); for (i = 0; i < h->meta_contexts.len; ++i) if (context_id == h->meta_contexts.ptr[i].context_id) break; @@ -464,7 +488,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES:
if (CALL_CALLBACK (cmd->cb.fn.extent, h->meta_contexts.ptr[i].name, cmd->offset, - &h->bs_entries[1], (length-4) / 4, + h->bs_entries, count, &error) == -1) if (cmd->error == 0) cmd->error = error ? error : EPROTO;
Here's what I suggest as an update for this patch (to be squashed): diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index da1e46929cd0..6cd4a49baa26 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -43,6 +43,20 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length, return true; } +static bool +bs_reply_length_ok (uint32_t length) +{ + if (length < (sizeof (struct nbd_structured_reply_block_status_hdr) + + sizeof (struct nbd_block_descriptor))) + return false; + + length -= sizeof (struct nbd_structured_reply_block_status_hdr); + if (length % sizeof (struct nbd_block_descriptor) != 0) + return false; + + return true; +} + STATE_MACHINE { REPLY.STRUCTURED_REPLY.START: /* We've only read the simple_reply. The structured_reply is longer, @@ -123,7 +137,7 @@ REPLY.STRUCTURED_REPLY.CHECK: case NBD_REPLY_TYPE_BLOCK_STATUS: if (cmd->type != NBD_CMD_BLOCK_STATUS || - length < 12 || ((length-4) & 7) != 0) + !bs_reply_length_ok (length)) goto resync; assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); /* Start by reading the context ID. */ @@ -430,7 +444,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); - assert (length >= 12); + assert (bs_reply_length_ok (length)); length -= sizeof h->sbuf.sr.payload.bs_hdr; free (h->bs_entries); @@ -466,8 +480,11 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); assert (h->bs_entries); - assert (length >= 12); + assert (bs_reply_length_ok (length)); assert (h->meta_valid); + STATIC_ASSERT ((sizeof (struct nbd_block_descriptor) % + sizeof *h->bs_entries) == 0, + _block_desc_is_multiple_of_bs_entry); count = (length - sizeof h->sbuf.sr.payload.bs_hdr) / sizeof *h->bs_entries; /* Need to byte-swap the entries returned, but apart from that we Laszlo
On Thu, May 25, 2023 at 06:30:37PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
For 32-bit block status, we were able to cheat and use an array with an odd number of elements, with array[0] holding the context id, and passing &array[1] to the user's callback. But once we have 64-bit extents, we can no longer abuse array element 0 like that, for two reasons: 64-bit extents contain uint64_t which might not be alignment-compatible with an array of uint32_t on all architectures, and the new NBD_REPLY_TYPE_BLOCK_STATUS_EXT adds an additional count field before the array.
+++ b/generator/states-reply-structured.c @@ -126,19 +126,10 @@ REPLY.STRUCTURED_REPLY.CHECK: length < 12 || ((length-4) & 7) != 0)
This is important (the context doesn't show it in full): we're under NBD_REPLY_TYPE_BLOCK_STATUS here (nested under REPLY.STRUCTURED_REPLY.CHECK), and we enforce that
length = be32toh (h->sbuf.sr.structured_reply.length);
contains the context_id (4 bytes), plus a positive integral number of block descriptor structures (8 bytes each).
And when 64-bit replies are added, there's a counterpart that contains a header (4+4 bytes) and then a positive integral number of block descriptors (16 bytes each).
@@ -445,15 +468,16 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (h->bs_entries); assert (length >= 12); assert (h->meta_valid); + count = (length - sizeof h->sbuf.sr.payload.bs_hdr) / sizeof *h->bs_entries;
I have a slight problem with the pre-patch code here. We keep the existent assertions (good), but I think the pre-patch RECV_BS_ENTRIES code misses an assertion. Namely, after the size check (i.e., 12+ bytes), the pre-patch code should have said
assert ((length-4) & 7) == 0);
emphasizing the explicit check under REPLY.STRUCTURED_REPLY.CHECK.
The pre-patch code relies on this (a) silently by expecting (length/4) to be an integer (in the mathematical sense), and (b) very silently by expecting (length/4) to be an *odd* integer >= 3.
Likewise, once 64-bit replies are added, we will depend on (h->len/8) being an *odd* integer >= 3, but additionally that the count in the header match the (h->len-8)/16 computation.
Here's what I suggest as an update for this patch (to be squashed):
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index da1e46929cd0..6cd4a49baa26 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -43,6 +43,20 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length, return true; }
+static bool +bs_reply_length_ok (uint32_t length) +{ + if (length < (sizeof (struct nbd_structured_reply_block_status_hdr) + + sizeof (struct nbd_block_descriptor))) + return false; + + length -= sizeof (struct nbd_structured_reply_block_status_hdr); + if (length % sizeof (struct nbd_block_descriptor) != 0) + return false; + + return true; +}
This is only valid for the 32-bit reply; but could easily be generalized to cover the 64-bit reply as well. I definitely like wrapping the computation behind a helper function, so that we don't have to open-code repeat it in multiple spots. I could even go with doing this sort of cleanup as its own prerequisite patch instead of squashing it in with this one, because....
+ STATE_MACHINE { REPLY.STRUCTURED_REPLY.START: /* We've only read the simple_reply. The structured_reply is longer, @@ -123,7 +137,7 @@ REPLY.STRUCTURED_REPLY.CHECK:
case NBD_REPLY_TYPE_BLOCK_STATUS: if (cmd->type != NBD_CMD_BLOCK_STATUS || - length < 12 || ((length-4) & 7) != 0) + !bs_reply_length_ok (length)) goto resync;
...that does indeed look easier to read.
@@ -466,8 +480,11 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); assert (h->bs_entries); - assert (length >= 12); + assert (bs_reply_length_ok (length)); assert (h->meta_valid); + STATIC_ASSERT ((sizeof (struct nbd_block_descriptor) % + sizeof *h->bs_entries) == 0, + _block_desc_is_multiple_of_bs_entry); count = (length - sizeof h->sbuf.sr.payload.bs_hdr) / sizeof *h->bs_entries;
That static assertion also makes sense. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
In order to more easily add a third reply type with an even larger header, but where the payload will look the same for both structured and extended replies, it is nicer if simple and structured replies are nested inside the same layer of sbuf.reply.hdr. While at it, note that while .or and .sr are structs declared within the overall sbuf union, we never read into both halves of those structs at the same time, so it does not matter if their two halves are consecutive. Dropping the packed notation on those structs means the compiler can align .payload more naturally, which may slightly improve performance on some platforms, even if it makes the overall union a few bytes larger due to padding. Visually, this patch changes the layout from: offset simple structured +------------------------------------------------------------+ | union sbuf | | +---------------------+------------------------------+ | | | struct simple_reply | union sr | | | | +-----------------+ | +--------------------------+ | | | | | | | | struct structured_reply | | | | | | | | | +----------------------+ | | | | 0 | | uint32_t magic | | | | uint32_t magic | | | | | 4 | | uint32_t error | | | | uint16_t flags | | | | | 6 | | | | | | uint16_t type | | | | | 8 | | uint64_t handle | | | | uint64_t handle | | | | | | +-----------------+ | | | | | | | | 16 | [padding] | | | uint32_t length | | | | | | | | +----------------------+ | | | | | | | union payload | | | | | | | +-----------+----------+ | | | | 20 | | | | ... | ... | | | | | | | | +-----------+----------+ | | | | | | +--------------------------+ | | | +---------------------+------------------------------+ | +------------------------------------------------------------+ to: offset simple structured +-------------------------------------------------------------+ | union sbuf | | +-----------------------------------------------------+ | | | struct reply | | | | +-------------------------------------------------+ | | | | | union hdr | | | | | | +--------------------+------------------------+ | | | | | | | struct simple | struct structured | | | | | | | | +----------------+ | +--------------------+ | | | | | 0 | | | | uint32_t magic | | | uint32_t magic | | | | | | 4 | | | | uint32_t error | | | uint16_t flags | | | | | | 6 | | | | | | | uint16_t type | | | | | | 8 | | | | uint64_t handle| | | uint64_t handle | | | | | | | | | +----------------+ | | | | | | | | 16 | | | [padding] | | uint32_t length | | | | | | | | | | +--------------------+ | | | | | 20 | | | | [padding] | | | | | | | +--------------------+------------------------+ | | | | | | union payload | | | | | | +--------------------+------------------------+ | | | | 24 | | | ... | ... | | | | | | | +--------------------+------------------------+ | | | | | +-------------------------------------------------+ | | | +-----------------------------------------------------+ | +-------------------------------------------------------------+ Technically, whether the payload union offset moves to byte 24 (with 20-23 now padding) or stays at 20 depends on compiler ABI; but many systems prefer that any struct with a uint64_t provide 8-byte alignment to its containing union. The commit is largely mechanical, and there should be no semantic change. Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 12 ++-- generator/states-reply-simple.c | 4 +- generator/states-reply-structured.c | 103 ++++++++++++++-------------- generator/states-reply.c | 14 ++-- 4 files changed, 68 insertions(+), 65 deletions(-) diff --git a/lib/internal.h b/lib/internal.h index 25eeea34..c71980ef 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -231,14 +231,16 @@ struct nbd_handle { struct { struct nbd_fixed_new_option_reply_meta_context context; char str[NBD_MAX_STRING]; - } __attribute__ ((packed)) context; + } __attribute__ ((packed)) context; char err_msg[NBD_MAX_STRING]; } payload; - } __attribute__ ((packed)) or; + } or; struct nbd_export_name_option_reply export_name_reply; - struct nbd_simple_reply simple_reply; struct { - struct nbd_structured_reply structured_reply; + union { + struct nbd_simple_reply simple; + struct nbd_structured_reply structured; + } hdr; union { struct nbd_structured_reply_offset_data offset_data; struct nbd_structured_reply_offset_hole offset_hole; @@ -249,7 +251,7 @@ struct nbd_handle { uint64_t offset; /* Only used for NBD_REPLY_TYPE_ERROR_OFFSET */ } __attribute__ ((packed)) error; } payload; - } __attribute__ ((packed)) sr; + } reply; uint16_t gflags; uint32_t cflags; uint32_t len; diff --git a/generator/states-reply-simple.c b/generator/states-reply-simple.c index 8fd9f62a..e6f1ee23 100644 --- a/generator/states-reply-simple.c +++ b/generator/states-reply-simple.c @@ -23,7 +23,7 @@ REPLY.SIMPLE_REPLY.START: struct command *cmd = h->reply_cmd; uint32_t error; - error = be32toh (h->sbuf.simple_reply.error); + error = be32toh (h->sbuf.reply.hdr.simple.error); if (cmd == NULL) { /* Unexpected reply. If error was set or we have structured @@ -39,7 +39,7 @@ REPLY.SIMPLE_REPLY.START: if (error || h->structured_replies) SET_NEXT_STATE (%^FINISH_COMMAND); else { - uint64_t cookie = be64toh (h->sbuf.simple_reply.handle); + uint64_t cookie = be64toh (h->sbuf.reply.hdr.simple.handle); SET_NEXT_STATE (%.DEAD); set_error (EPROTO, "no matching cookie %" PRIu64 " found for server reply, " diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 96182222..6f96945a 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -49,9 +49,9 @@ REPLY.STRUCTURED_REPLY.START: * so read the remaining part. */ h->rbuf = &h->sbuf; - h->rbuf = (char *)h->rbuf + sizeof h->sbuf.simple_reply; - h->rlen = sizeof h->sbuf.sr.structured_reply; - h->rlen -= sizeof h->sbuf.simple_reply; + h->rbuf = (char *)h->rbuf + sizeof h->sbuf.reply.hdr.simple; + h->rlen = sizeof h->sbuf.reply.hdr.structured; + h->rlen -= sizeof h->sbuf.reply.hdr.simple; SET_NEXT_STATE (%RECV_REMAINING); return 0; @@ -71,9 +71,9 @@ REPLY.STRUCTURED_REPLY.CHECK: uint16_t flags, type; uint32_t length; - flags = be16toh (h->sbuf.sr.structured_reply.flags); - type = be16toh (h->sbuf.sr.structured_reply.type); - length = be32toh (h->sbuf.sr.structured_reply.length); + flags = be16toh (h->sbuf.reply.hdr.structured.flags); + type = be16toh (h->sbuf.reply.hdr.structured.type); + length = be32toh (h->sbuf.reply.hdr.structured.length); /* Reject a server that replies with too much information, but don't * reject a single structured reply to NBD_CMD_READ on the largest @@ -82,7 +82,7 @@ REPLY.STRUCTURED_REPLY.CHECK: * oversized reply is going to take long enough to resync that it is * not worth keeping the connection alive. */ - if (length > MAX_REQUEST_SIZE + sizeof h->sbuf.sr.payload.offset_data) { + if (length > MAX_REQUEST_SIZE + sizeof h->sbuf.reply.payload.offset_data) { set_error (0, "invalid server reply length %" PRIu32, length); SET_NEXT_STATE (%.DEAD); return 0; @@ -105,19 +105,19 @@ REPLY.STRUCTURED_REPLY.CHECK: * them as an extension, so we use < instead of <=. */ if (cmd->type != NBD_CMD_READ || - length < sizeof h->sbuf.sr.payload.offset_data) + length < sizeof h->sbuf.reply.payload.offset_data) goto resync; - h->rbuf = &h->sbuf.sr.payload.offset_data; - h->rlen = sizeof h->sbuf.sr.payload.offset_data; + h->rbuf = &h->sbuf.reply.payload.offset_data; + h->rlen = sizeof h->sbuf.reply.payload.offset_data; SET_NEXT_STATE (%RECV_OFFSET_DATA); break; case NBD_REPLY_TYPE_OFFSET_HOLE: if (cmd->type != NBD_CMD_READ || - length != sizeof h->sbuf.sr.payload.offset_hole) + length != sizeof h->sbuf.reply.payload.offset_hole) goto resync; - h->rbuf = &h->sbuf.sr.payload.offset_hole; - h->rlen = sizeof h->sbuf.sr.payload.offset_hole; + h->rbuf = &h->sbuf.reply.payload.offset_hole; + h->rlen = sizeof h->sbuf.reply.payload.offset_hole; SET_NEXT_STATE (%RECV_OFFSET_HOLE); break; @@ -127,8 +127,8 @@ REPLY.STRUCTURED_REPLY.CHECK: goto resync; assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); /* Start by reading the context ID. */ - h->rbuf = &h->sbuf.sr.payload.bs_hdr; - h->rlen = sizeof h->sbuf.sr.payload.bs_hdr; + h->rbuf = &h->sbuf.reply.payload.bs_hdr; + h->rlen = sizeof h->sbuf.reply.payload.bs_hdr; SET_NEXT_STATE (%RECV_BS_HEADER); break; @@ -139,10 +139,10 @@ REPLY.STRUCTURED_REPLY.CHECK: * compliant, will favor the wire error over EPROTO during more * length checks in RECV_ERROR_MESSAGE and RECV_ERROR_TAIL. */ - if (length < sizeof h->sbuf.sr.payload.error.error.error) + if (length < sizeof h->sbuf.reply.payload.error.error.error) goto resync; - h->rbuf = &h->sbuf.sr.payload.error.error; - h->rlen = MIN (length, sizeof h->sbuf.sr.payload.error.error); + h->rbuf = &h->sbuf.reply.payload.error.error; + h->rlen = MIN (length, sizeof h->sbuf.reply.payload.error.error); SET_NEXT_STATE (%RECV_ERROR); } else @@ -168,19 +168,19 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.sr.structured_reply.length); - assert (length >= sizeof h->sbuf.sr.payload.error.error.error); + length = be32toh (h->sbuf.reply.hdr.structured.length); + assert (length >= sizeof h->sbuf.reply.payload.error.error.error); assert (cmd); - if (length < sizeof h->sbuf.sr.payload.error.error) + if (length < sizeof h->sbuf.reply.payload.error.error) goto resync; - msglen = be16toh (h->sbuf.sr.payload.error.error.len); - if (msglen > length - sizeof h->sbuf.sr.payload.error.error || - msglen > sizeof h->sbuf.sr.payload.error.msg) + msglen = be16toh (h->sbuf.reply.payload.error.error.len); + if (msglen > length - sizeof h->sbuf.reply.payload.error.error || + msglen > sizeof h->sbuf.reply.payload.error.msg) goto resync; - h->rbuf = h->sbuf.sr.payload.error.msg; + h->rbuf = h->sbuf.reply.payload.error.msg; h->rlen = msglen; SET_NEXT_STATE (%RECV_ERROR_MESSAGE); } @@ -188,11 +188,11 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR: resync: /* Favor the error packet's errno over RESYNC's EPROTO. */ - error = be32toh (h->sbuf.sr.payload.error.error.error); + error = be32toh (h->sbuf.reply.payload.error.error.error); if (cmd->error == 0) cmd->error = nbd_internal_errno_of_nbd_error (error); h->rbuf = NULL; - h->rlen = length - MIN (length, sizeof h->sbuf.sr.payload.error.error); + h->rlen = length - MIN (length, sizeof h->sbuf.reply.payload.error.error); SET_NEXT_STATE (%RESYNC); return 0; @@ -207,15 +207,15 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR_MESSAGE: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.sr.structured_reply.length); - msglen = be16toh (h->sbuf.sr.payload.error.error.len); - type = be16toh (h->sbuf.sr.structured_reply.type); + length = be32toh (h->sbuf.reply.hdr.structured.length); + msglen = be16toh (h->sbuf.reply.payload.error.error.len); + type = be16toh (h->sbuf.reply.hdr.structured.type); - length -= sizeof h->sbuf.sr.payload.error.error + msglen; + length -= sizeof h->sbuf.reply.payload.error.error + msglen; if (msglen) debug (h, "structured error server message: %.*s", (int)msglen, - h->sbuf.sr.payload.error.msg); + h->sbuf.reply.payload.error.msg); /* Special case two specific errors; silently ignore tail for all others */ h->rbuf = NULL; @@ -227,11 +227,11 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR_MESSAGE: "the server may have a bug"); break; case NBD_REPLY_TYPE_ERROR_OFFSET: - if (length != sizeof h->sbuf.sr.payload.error.offset) + if (length != sizeof h->sbuf.reply.payload.error.offset) debug (h, "unable to safely extract error offset, " "the server may have a bug"); else - h->rbuf = &h->sbuf.sr.payload.error.offset; + h->rbuf = &h->sbuf.reply.payload.error.offset; break; } SET_NEXT_STATE (%RECV_ERROR_TAIL); @@ -250,8 +250,8 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR_TAIL: SET_NEXT_STATE (%.READY); return 0; case 0: - error = be32toh (h->sbuf.sr.payload.error.error.error); - type = be16toh (h->sbuf.sr.structured_reply.type); + error = be32toh (h->sbuf.reply.payload.error.error.error); + type = be16toh (h->sbuf.reply.hdr.structured.type); assert (cmd); /* guaranteed by CHECK */ @@ -266,7 +266,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR_TAIL: * user callback if present. Ignore the offset if it was bogus. */ if (type == NBD_REPLY_TYPE_ERROR_OFFSET && h->rbuf) { - uint64_t offset = be64toh (h->sbuf.sr.payload.error.offset); + uint64_t offset = be64toh (h->sbuf.reply.payload.error.offset); if (structured_reply_in_bounds (offset, 0, cmd) && cmd->type == NBD_CMD_READ && CALLBACK_IS_NOT_NULL (cmd->cb.fn.chunk)) { @@ -307,8 +307,8 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.sr.structured_reply.length); - offset = be64toh (h->sbuf.sr.payload.offset_data.offset); + length = be32toh (h->sbuf.reply.hdr.structured.length); + offset = be64toh (h->sbuf.reply.payload.offset_data.offset); assert (cmd); /* guaranteed by CHECK */ @@ -346,8 +346,8 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.sr.structured_reply.length); - offset = be64toh (h->sbuf.sr.payload.offset_data.offset); + length = be32toh (h->sbuf.reply.hdr.structured.length); + offset = be64toh (h->sbuf.reply.payload.offset_data.offset); assert (cmd); /* guaranteed by CHECK */ if (CALLBACK_IS_NOT_NULL (cmd->cb.fn.chunk)) { @@ -377,8 +377,8 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_HOLE: SET_NEXT_STATE (%.READY); return 0; case 0: - offset = be64toh (h->sbuf.sr.payload.offset_hole.offset); - length = be32toh (h->sbuf.sr.payload.offset_hole.length); + offset = be64toh (h->sbuf.reply.payload.offset_hole.offset); + length = be32toh (h->sbuf.reply.payload.offset_hole.length); assert (cmd); /* guaranteed by CHECK */ @@ -426,12 +426,12 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.sr.structured_reply.length); + length = be32toh (h->sbuf.reply.hdr.structured.length); assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (length >= 12); - length -= sizeof h->sbuf.sr.payload.bs_hdr; + length -= sizeof h->sbuf.reply.payload.bs_hdr; free (h->bs_entries); h->bs_entries = malloc (length); @@ -460,7 +460,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.sr.structured_reply.length); + length = be32toh (h->sbuf.reply.hdr.structured.length); assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); @@ -468,7 +468,8 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (h->bs_entries); assert (length >= 12); assert (h->meta_valid); - count = (length - sizeof h->sbuf.sr.payload.bs_hdr) / sizeof *h->bs_entries; + count = (length - sizeof h->sbuf.reply.payload.bs_hdr) / + sizeof *h->bs_entries; /* Need to byte-swap the entries returned, but apart from that we * don't validate them. @@ -477,7 +478,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: h->bs_entries[i] = be32toh (h->bs_entries[i]); /* Look up the context ID. */ - context_id = be32toh (h->sbuf.sr.payload.bs_hdr.context_id); + context_id = be32toh (h->sbuf.reply.payload.bs_hdr.context_id); for (i = 0; i < h->meta_contexts.len; ++i) if (context_id == h->meta_contexts.ptr[i].context_id) break; @@ -524,8 +525,8 @@ REPLY.STRUCTURED_REPLY.RESYNC: SET_NEXT_STATE (%^FINISH_COMMAND); return 0; } - type = be16toh (h->sbuf.sr.structured_reply.type); - length = be32toh (h->sbuf.sr.structured_reply.length); + type = be16toh (h->sbuf.reply.hdr.structured.type); + length = be32toh (h->sbuf.reply.hdr.structured.length); debug (h, "unexpected reply type %u or payload length %" PRIu32 " for cookie %" PRIu64 " and command %" PRIu32 ", this is probably a server bug", @@ -539,7 +540,7 @@ REPLY.STRUCTURED_REPLY.RESYNC: REPLY.STRUCTURED_REPLY.FINISH: uint16_t flags; - flags = be16toh (h->sbuf.sr.structured_reply.flags); + flags = be16toh (h->sbuf.reply.hdr.structured.flags); if (flags & NBD_REPLY_FLAG_DONE) { SET_NEXT_STATE (%^FINISH_COMMAND); } diff --git a/generator/states-reply.c b/generator/states-reply.c index f7888154..31e4bd2f 100644 --- a/generator/states-reply.c +++ b/generator/states-reply.c @@ -69,8 +69,8 @@ REPLY.START: assert (h->reply_cmd == NULL); assert (h->rlen == 0); - h->rbuf = &h->sbuf; - h->rlen = sizeof h->sbuf.simple_reply; + h->rbuf = &h->sbuf.reply.hdr; + h->rlen = sizeof h->sbuf.reply.hdr.simple; r = h->sock->ops->recv (h, h->sock, h->rbuf, h->rlen); if (r == -1) { @@ -115,7 +115,7 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: uint32_t magic; uint64_t cookie; - magic = be32toh (h->sbuf.simple_reply.magic); + magic = be32toh (h->sbuf.reply.hdr.simple.magic); if (magic == NBD_SIMPLE_REPLY_MAGIC) { SET_NEXT_STATE (%SIMPLE_REPLY.START); } @@ -127,8 +127,8 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: SET_NEXT_STATE (%.DEAD); set_error (0, "invalid reply magic 0x%" PRIx32, magic); #if 0 /* uncomment to see desynchronized data */ - nbd_internal_hexdump (&h->sbuf.simple_reply, - sizeof (h->sbuf.simple_reply), + nbd_internal_hexdump (&h->sbuf.reply.hdr.simple, + sizeof (h->sbuf.reply.hdr.simple), stderr); #endif return 0; @@ -138,7 +138,7 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: * handle (our cookie) is stored at the same offset. */ h->chunks_received++; - cookie = be64toh (h->sbuf.simple_reply.handle); + cookie = be64toh (h->sbuf.reply.hdr.simple.handle); /* Find the command amongst the commands in flight. If the server sends * a reply for an unknown cookie, FINISH will diagnose that later. */ @@ -157,7 +157,7 @@ REPLY.FINISH_COMMAND: /* NB: This works for both simple and structured replies because the * handle (our cookie) is stored at the same offset. */ - cookie = be64toh (h->sbuf.simple_reply.handle); + cookie = be64toh (h->sbuf.reply.hdr.simple.handle); /* Find the command amongst the commands in flight. */ for (cmd = h->cmds_in_flight, prev_cmd = NULL; cmd != NULL; -- 2.40.1
On 5/25/23 15:00, Eric Blake wrote:
In order to more easily add a third reply type with an even larger header, but where the payload will look the same for both structured and extended replies, it is nicer if simple and structured replies are nested inside the same layer of sbuf.reply.hdr.
This makes sense, and the following, strictly related code change in the patch body corresponds to the explanation (except for the "__attribute__ ((packed))" removal, but more on that later): - struct nbd_simple_reply simple_reply; struct { - struct nbd_structured_reply structured_reply; + union { + struct nbd_simple_reply simple; + struct nbd_structured_reply structured; + } hdr; union { struct nbd_structured_reply_offset_data offset_data; struct nbd_structured_reply_offset_hole offset_hole; struct nbd_structured_reply_block_status_hdr bs_hdr; struct { struct nbd_structured_reply_error error; char msg[NBD_MAX_STRING]; /* Common to all error types */ uint64_t offset; /* Only used for NBD_REPLY_TYPE_ERROR_OFFSET */ } __attribute__ ((packed)) error; } payload; - } __attribute__ ((packed)) sr; + } reply; One interesting consequence is that the payload union now becomes possible after the simple header, which IIUC makes no sense per NBD spec. But, it doesn't hurt either. At the same time:
While at it, note that while .or and .sr are structs declared within the overall sbuf union, we never read into both halves of those structs at the same time, so it does not matter if their two halves are consecutive.
I've been staring at this part for an extremely long time, and I just can't make any sense of it. My (strongly held) opinion is that this part of the patch should be split off to a separate patch. Here's a possible interpretation: - The "sbuf.or" ("option reply") member of the "sbuf" union is a structure. It has two members: "sbuf.or.option_reply" (another structure) and "sbuf.or.payload" (a union). We never read into "sbuf.or.option_reply" and "sbuf.or.payload" at the same time (i.e., we never cross the structure field boundary between them, with a single recv operation), therefore we can remove the "packed" attribute from "sbuf.or". This permits the compiler to insert padding between "sbuf.or.option_reply" and "sbuf.or.payload", and/or after "sbuf.or". - Similarly, the "sbuf.sr" ("structured reply") member of the "sbuf" union is a structure. It has two members: "sbuf.sr.structured_reply" (a structure) and "sbuf.sr.payload" (a union). We never read into "sbuf.sr.structured_reply" and "sbuf.sr.payload" at the same time (i.e., we never cross the structure field boundary between them, with a single recv operation), therefore we can remove the "packed" attribute from "sbuf.sr". This permits the compiler to insert padding between "sbuf.sr.structured_reply" and "sbuf.sr.payload", and/or after "sbuf.sr". In turn, my problem with this interpretation is that, if we never cross the struct member boundary within "sbuf.or" (i.e., between "option_reply" and "payload"), and similarly we never cross the struct member boundary within "sbuf.sr" (i.e., between "structured_reply" and "payload"), then *why* are "sbuf.or" and "sbuf.sr" structures in the first place (and not unions)? Now, I can imagine that the answer sounds like this: indeed we don't cross those field boundaries with a single recv, however we still need the data from both fields *at the same time*, later on. And this certainly sounds like a valid explanation, and dropping the "packed" attributes is fine / justified as well, under this interpreation, but it absolutely must be a separate patch, IMO. I think I've spent at least half an hour explaining it to myself.
Dropping the packed notation on those structs means the compiler can align .payload more naturally, which may slightly improve performance on some platforms, even if it makes the overall union a few bytes larger due to padding.
Visually, this patch changes the layout from:
offset simple structured +------------------------------------------------------------+ | union sbuf | | +---------------------+------------------------------+ | | | struct simple_reply | union sr | | | | +-----------------+ | +--------------------------+ | | | | | | | | struct structured_reply | | | | | | | | | +----------------------+ | | | | 0 | | uint32_t magic | | | | uint32_t magic | | | | | 4 | | uint32_t error | | | | uint16_t flags | | | | | 6 | | | | | | uint16_t type | | | | | 8 | | uint64_t handle | | | | uint64_t handle | | | | | | +-----------------+ | | | | | | | | 16 | [padding] | | | uint32_t length | | | | | | | | +----------------------+ | | | | | | | union payload | | | | | | | +-----------+----------+ | | | | 20 | | | | ... | ... | | | | | | | | +-----------+----------+ | | | | | | +--------------------------+ | | | +---------------------+------------------------------+ | +------------------------------------------------------------+
to:
offset simple structured +-------------------------------------------------------------+ | union sbuf | | +-----------------------------------------------------+ | | | struct reply | | | | +-------------------------------------------------+ | | | | | union hdr | | | | | | +--------------------+------------------------+ | | | | | | | struct simple | struct structured | | | | | | | | +----------------+ | +--------------------+ | | | | | 0 | | | | uint32_t magic | | | uint32_t magic | | | | | | 4 | | | | uint32_t error | | | uint16_t flags | | | | | | 6 | | | | | | | uint16_t type | | | | | | 8 | | | | uint64_t handle| | | uint64_t handle | | | | | | | | | +----------------+ | | | | | | | | 16 | | | [padding] | | uint32_t length | | | | | | | | | | +--------------------+ | | | | | 20 | | | | [padding] | | | | | | | +--------------------+------------------------+ | | | | | | union payload | | | | | | +--------------------+------------------------+ | | | | 24 | | | ... | ... | | | | | | | +--------------------+------------------------+ | | | | | +-------------------------------------------------+ | | | +-----------------------------------------------------+ | +-------------------------------------------------------------+
Technically, whether the payload union offset moves to byte 24 (with 20-23 now padding) or stays at 20 depends on compiler ABI; but many systems prefer that any struct with a uint64_t provide 8-byte alignment to its containing union.
The commit is largely mechanical, and there should be no semantic change.
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 12 ++-- generator/states-reply-simple.c | 4 +- generator/states-reply-structured.c | 103 ++++++++++++++-------------- generator/states-reply.c | 14 ++-- 4 files changed, 68 insertions(+), 65 deletions(-)
diff --git a/lib/internal.h b/lib/internal.h index 25eeea34..c71980ef 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -231,14 +231,16 @@ struct nbd_handle { struct { struct nbd_fixed_new_option_reply_meta_context context; char str[NBD_MAX_STRING]; - } __attribute__ ((packed)) context; + } __attribute__ ((packed)) context;
Whitespace fixup, tolerable as a part of this patch.
char err_msg[NBD_MAX_STRING]; } payload; - } __attribute__ ((packed)) or; + } or;
So I disagree with doing this here. I'd like it to be its own patch.
struct nbd_export_name_option_reply export_name_reply; - struct nbd_simple_reply simple_reply; struct { - struct nbd_structured_reply structured_reply; + union { + struct nbd_simple_reply simple; + struct nbd_structured_reply structured; + } hdr; union { struct nbd_structured_reply_offset_data offset_data; struct nbd_structured_reply_offset_hole offset_hole;
Yes, fine (although formatting it with a much larger context would have helped -- anyway I'm reviewing the series with -W in my local clone).
@@ -249,7 +251,7 @@ struct nbd_handle { uint64_t offset; /* Only used for NBD_REPLY_TYPE_ERROR_OFFSET */ } __attribute__ ((packed)) error; } payload; - } __attribute__ ((packed)) sr; + } reply; uint16_t gflags; uint32_t cflags; uint32_t len;
Here too, the __attribute__ ((packed)) removal shoould be in a separate patch. Renaming "sr" to "reply" is "in scope", of course.
diff --git a/generator/states-reply-simple.c b/generator/states-reply-simple.c index 8fd9f62a..e6f1ee23 100644 --- a/generator/states-reply-simple.c +++ b/generator/states-reply-simple.c @@ -23,7 +23,7 @@ REPLY.SIMPLE_REPLY.START: struct command *cmd = h->reply_cmd; uint32_t error;
- error = be32toh (h->sbuf.simple_reply.error); + error = be32toh (h->sbuf.reply.hdr.simple.error);
if (cmd == NULL) { /* Unexpected reply. If error was set or we have structured
Yes.
@@ -39,7 +39,7 @@ REPLY.SIMPLE_REPLY.START: if (error || h->structured_replies) SET_NEXT_STATE (%^FINISH_COMMAND); else { - uint64_t cookie = be64toh (h->sbuf.simple_reply.handle); + uint64_t cookie = be64toh (h->sbuf.reply.hdr.simple.handle); SET_NEXT_STATE (%.DEAD); set_error (EPROTO, "no matching cookie %" PRIu64 " found for server reply, "
Yes; same thing, just different name.
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 96182222..6f96945a 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -49,9 +49,9 @@ REPLY.STRUCTURED_REPLY.START: * so read the remaining part. */ h->rbuf = &h->sbuf; - h->rbuf = (char *)h->rbuf + sizeof h->sbuf.simple_reply; - h->rlen = sizeof h->sbuf.sr.structured_reply; - h->rlen -= sizeof h->sbuf.simple_reply; + h->rbuf = (char *)h->rbuf + sizeof h->sbuf.reply.hdr.simple; + h->rlen = sizeof h->sbuf.reply.hdr.structured; + h->rlen -= sizeof h->sbuf.reply.hdr.simple; SET_NEXT_STATE (%RECV_REMAINING); return 0;
Here I disagree with the mechanical approach. I even take issue with the pre-patch code. Pre-patch, we fill in "h->sbuf.simple_reply" (in "generator/states-reply.c"), i.e., one member of the top-level "sbuf" union, but then continue filling a member of a *different* member (i.e., "sr.structured_reply") of the "sbuf" union here. This looks undefined by the C standard, but even if it's not undefined, it's supremely confusing. Now, if structure "nbd_simple_reply" were the first member of "nbd_structured_reply", maybe we could salvage this somehow -- but that's not the case. ... Right, in "generator/states-reply.c", we have: /* We read all replies initially as if they are simple replies, but * check the magic in CHECK_SIMPLE_OR_STRUCTURED_REPLY below. * This works because the structured_reply header is larger. */ This is precisely my problem. The common initial subsequence is only the "uint32_t magic" field. Therefore the code in "generator/states-reply.c" should only read the magic field at first, and then branch to simple vs. structured reply. Post-patch, the sbuf.reply.hdr union actually reflects this well! The structure members of that union all begin with the common subsequence "uint32_t magic", so using the special guarantee from the C standard, it's fine to fill in "magic" via one union member, and read it back via another. Pre-patch, the guarantee doesn't apply, because "sbuf.simple_reply" and "sbuf.sr.structured_reply" are not members of the same union. However, even with this patch applied (i.e., with "sbuf.reply.hdr" existing), the code performing the reading continues making me uncomfortable. We effectively have this sequence: - take address of "h->sbuf.reply.hdr" - read "sizeof h->sbuf.reply.hdr.simple" bytes into it - (at this point we can safely access "h->sbuf.reply.hdr.simple", *and* "h->sbuf.reply.hdr.structured.magic" as well -- but not the rest of "h->sbuf.reply.hdr.structured"!) - take the address of "h->sbuf.reply.hdr.structured.length" -- but not by name; instead, only by virtue of the previous read having filled in "magic", "flags", "type", and "handle", as a *happenstance*, via populating "h->sbuf.reply.hdr.simple" - read as many bytes as necessary in order to complete "h->sbuf.reply.hdr.structured", from that point onward. This smells to me. Optimally, the simple reply and the structured reply should look like this: struct nbd_reply_header { uint32_t magic; union { struct { uint32_t error; uint64_t handle; } simple; struct { uint16_t flags; uint16_t type; uint64_t handle; uint32_t length; } structured; } magic_specific; }; and we should have separate automaton states for reading "magic_specific.simple" and "magic_specific.structured". In REPLY.START, we should only read "magic". We should have a sepate state called REPLY.SIMPLE_REPLY.START, for reading "magic_specific.simple". In REPLY.STRUCTURED_REPLY.START, we should point h->rbuf at "magic_specific.structured", and read "sizeof magic_specific.structured" bytes. I'll stop here because there's already much to discuss. Thanks, Laszlo
On 5/26/23 17:53, Laszlo Ersek wrote:
Optimally, the simple reply and the structured reply should look like this:
struct nbd_reply_header { uint32_t magic; union { struct { uint32_t error; uint64_t handle; } simple; struct { uint16_t flags; uint16_t type; uint64_t handle; uint32_t length; } structured; } magic_specific; };
and we should have separate automaton states for reading "magic_specific.simple" and "magic_specific.structured".
In REPLY.START, we should only read "magic".
We should have a sepate state called REPLY.SIMPLE_REPLY.START, for reading "magic_specific.simple".
In REPLY.STRUCTURED_REPLY.START, we should point h->rbuf at "magic_specific.structured", and read "sizeof magic_specific.structured" bytes.
This (pre-patch) part: /* NB: This works for both simple and structured replies because the * handle (our cookie) is stored at the same offset. */ [...] cookie = be64toh (h->sbuf.simple_reply.handle); is disconcerting as well. I think it's well-defined C, but a hack nonetheless. IMO, unions are justified for two purposes: - deliberately reinterpreting one object representation as another - saving space, when at most one of N objects is expected to exist at any given time. Both of those uses follow from intentional elements of a design. But the fact that "handle" is at the same offset in both "simple" and "structured" is totally arbitrary. IMO this is a hack. Laszlo
On Fri, May 26, 2023 at 06:09:00PM +0200, Laszlo Ersek wrote:
On 5/26/23 17:53, Laszlo Ersek wrote:
Optimally, the simple reply and the structured reply should look like this:
struct nbd_reply_header { uint32_t magic; union { struct { uint32_t error; uint64_t handle; } simple; struct { uint16_t flags; uint16_t type; uint64_t handle; uint32_t length; } structured; } magic_specific; };
and we should have separate automaton states for reading "magic_specific.simple" and "magic_specific.structured".
In REPLY.START, we should only read "magic".
We should have a sepate state called REPLY.SIMPLE_REPLY.START, for reading "magic_specific.simple".
In REPLY.STRUCTURED_REPLY.START, we should point h->rbuf at "magic_specific.structured", and read "sizeof magic_specific.structured" bytes.
This (pre-patch) part:
/* NB: This works for both simple and structured replies because the * handle (our cookie) is stored at the same offset. */ [...] cookie = be64toh (h->sbuf.simple_reply.handle);
is disconcerting as well. I think it's well-defined C, but a hack nonetheless.
IMO, unions are justified for two purposes:
- deliberately reinterpreting one object representation as another
- saving space, when at most one of N objects is expected to exist at any given time.
Both of those uses follow from intentional elements of a design. But the fact that "handle" is at the same offset in both "simple" and "structured" is totally arbitrary. IMO this is a hack.
It is not completely arbitrary: when structured replies were added to the NBD spec, the choice of having handle at the same offset was intentional. Similarly, extended replies have it at the same offset as well. But a STATIC_ASSERT proving that would go a long way to proving our intent, more than just a comment in the code. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 5/26/23 23:26, Eric Blake wrote:
On Fri, May 26, 2023 at 06:09:00PM +0200, Laszlo Ersek wrote:
On 5/26/23 17:53, Laszlo Ersek wrote:
Optimally, the simple reply and the structured reply should look like this:
struct nbd_reply_header { uint32_t magic; union { struct { uint32_t error; uint64_t handle; } simple; struct { uint16_t flags; uint16_t type; uint64_t handle; uint32_t length; } structured; } magic_specific; };
and we should have separate automaton states for reading "magic_specific.simple" and "magic_specific.structured".
In REPLY.START, we should only read "magic".
We should have a sepate state called REPLY.SIMPLE_REPLY.START, for reading "magic_specific.simple".
In REPLY.STRUCTURED_REPLY.START, we should point h->rbuf at "magic_specific.structured", and read "sizeof magic_specific.structured" bytes.
This (pre-patch) part:
/* NB: This works for both simple and structured replies because the * handle (our cookie) is stored at the same offset. */ [...] cookie = be64toh (h->sbuf.simple_reply.handle);
is disconcerting as well. I think it's well-defined C, but a hack nonetheless.
IMO, unions are justified for two purposes:
- deliberately reinterpreting one object representation as another
- saving space, when at most one of N objects is expected to exist at any given time.
Both of those uses follow from intentional elements of a design. But the fact that "handle" is at the same offset in both "simple" and "structured" is totally arbitrary. IMO this is a hack.
It is not completely arbitrary: when structured replies were added to the NBD spec, the choice of having handle at the same offset was intentional. Similarly, extended replies have it at the same offset as well. But a STATIC_ASSERT proving that would go a long way to proving our intent, more than just a comment in the code.
Right, a STATIC_ASSERT would be great. Thanks! Laszlo
On Fri, May 26, 2023 at 05:53:38PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
In order to more easily add a third reply type with an even larger header, but where the payload will look the same for both structured and extended replies, it is nicer if simple and structured replies are nested inside the same layer of sbuf.reply.hdr.
This makes sense, and the following, strictly related code change in the patch body corresponds to the explanation (except for the "__attribute__ ((packed))" removal, but more on that later):
- struct nbd_simple_reply simple_reply; struct { - struct nbd_structured_reply structured_reply; + union { + struct nbd_simple_reply simple; + struct nbd_structured_reply structured; + } hdr; union { struct nbd_structured_reply_offset_data offset_data; struct nbd_structured_reply_offset_hole offset_hole; struct nbd_structured_reply_block_status_hdr bs_hdr; struct { struct nbd_structured_reply_error error; char msg[NBD_MAX_STRING]; /* Common to all error types */ uint64_t offset; /* Only used for NBD_REPLY_TYPE_ERROR_OFFSET */ } __attribute__ ((packed)) error; } payload; - } __attribute__ ((packed)) sr; + } reply;
One interesting consequence is that the payload union now becomes possible after the simple header, which IIUC makes no sense per NBD spec. But, it doesn't hurt either.
Indeed. 'union sbuf' is mostly used as a dumping grounds for "give me enough scratch storage that I can have a place to read in all places of interest across multiple recv() calls; I don't need the resulting storage to match wire format, but any time multiple pieces of information are needed at once, they must reside in non-overlapping portions of sbuf, via a sub-struct. But where we know we will not be accessing disjoint regions of sbuf, we can use further nested unions to compress the overall memory taken by sbuf". While we will never have a simple header used with a reply payload (because the only reply with payload for a simple header is NBD_CMD_READ, and we read that directly into the user's buffer instead of into sbuf), it doesn't hurt that union sbuf has reserved that much space (because we do need that much space for handilng a structured reply, and soon to handle an extended reply).
At the same time:
While at it, note that while .or and .sr are structs declared within the overall sbuf union, we never read into both halves of those structs at the same time, so it does not matter if their two halves are consecutive.
I've been staring at this part for an extremely long time, and I just can't make any sense of it. My (strongly held) opinion is that this part of the patch should be split off to a separate patch.
Can do. Looks like I'll be spinning a v4, but I'll let you get further along in the review process to minimize list churn before I start posting updates.
Here's a possible interpretation:
- The "sbuf.or" ("option reply") member of the "sbuf" union is a structure. It has two members: "sbuf.or.option_reply" (another structure) and "sbuf.or.payload" (a union).
We never read into "sbuf.or.option_reply" and "sbuf.or.payload" at the same time (i.e., we never cross the structure field boundary between them, with a single recv operation), therefore we can remove the "packed" attribute from "sbuf.or". This permits the compiler to insert padding between "sbuf.or.option_reply" and "sbuf.or.payload", and/or after "sbuf.or".
Correct, and since option replies are distinct from server replies, I agree with your argument of splitting removal of the attribute((packed)) into a separate patch.
- Similarly, the "sbuf.sr" ("structured reply") member of the "sbuf" union is a structure. It has two members: "sbuf.sr.structured_reply" (a structure) and "sbuf.sr.payload" (a union).
We never read into "sbuf.sr.structured_reply" and "sbuf.sr.payload" at the same time (i.e., we never cross the structure field boundary between them, with a single recv operation), therefore we can remove the "packed" attribute from "sbuf.sr". This permits the compiler to insert padding between "sbuf.sr.structured_reply" and "sbuf.sr.payload", and/or after "sbuf.sr".
Also correct. While it happens to more directly touch the rest of the layout update in this patch, it is still a better fit logically to remove the attribute to both sub-structs at the same time, independently from reflowing the layout of sbuf.simple_reply and sbuf.sr.
In turn, my problem with this interpretation is that, if we never cross the struct member boundary within "sbuf.or" (i.e., between "option_reply" and "payload"), and similarly we never cross the struct member boundary within "sbuf.sr" (i.e., between "structured_reply" and "payload"), then *why* are "sbuf.or" and "sbuf.sr" structures in the first place (and not unions)?
'union sbuf' is already a union. If we didn't need or.option_reply and or.payload at the same time, then it would need make sense to just have: union sbuf { ... struct nbd_fixed_new_option_reply option_reply; struct nbd_fixed_ne struct nbd_fixed_new_option_reply_server server; char str[NBD_MAX_STRING * 2 + 1]; /* name, description, NUL */ } __attribute__ ((packed)) or_payload_server; struct nbd_fixed_new_option_reply_info_export or_payload_export; ... }; But because we DO need access to both or.option_reply and or.payload at the same time (for example, see generator/states-newstyle.c:handle_reply_error() called after both pieces of sbuf.or have been populated), we instead packaged the entire option reply into a nested struct, while still recognizing that the distinct payload types can overlap via a nested union: union sbuf { ... struct { struct nbd_fixed_new_option_reply option_reply; union { struct nbd_fixed_ne struct nbd_fixed_new_option_reply_server server; char str[NBD_MAX_STRING * 2 + 1]; /* name, description, NUL */ } __attribute__ ((packed)) server; struct nbd_fixed_new_option_reply_info_export export; ... } payload; } or; ... }; And referring back to your earlier insight, or.payload.server is a nested struct that DOES need __attribute__((packed)), because that is a case where we DO use a single recv() to populate two distinct C types into guaranteed-adjacent memory, and don't want the compiler throwing in any padding that might mess with us.
Now, I can imagine that the answer sounds like this: indeed we don't cross those field boundaries with a single recv, however we still need the data from both fields *at the same time*, later on.
Bingo. On the wire, the two portions of the reply message happen to be adjacent, and we need both portions of the reply in memory to decipher the reply. But there is no reason why our in-memory representation has to be packed to match the wire layout. And allowing the compiler to insert padding can sometimes improve performance.
And this certainly sounds like a valid explanation, and dropping the "packed" attributes is fine / justified as well, under this interpreation, but it absolutely must be a separate patch, IMO. I think I've spent at least half an hour explaining it to myself.
And convinced me of it likewise. A half hour well spent.
Dropping the packed notation on those structs means the compiler can align .payload more naturally, which may slightly improve performance on some platforms, even if it makes the overall union a few bytes larger due to padding.
Visually, this patch changes the layout from:
offset simple structured +------------------------------------------------------------+ | union sbuf | | +---------------------+------------------------------+ | | | struct simple_reply | union sr | |
This line is wrong. Pre-patch, it is 'struct sr', not 'union sr'.
| | +-----------------+ | +--------------------------+ | | | | | | | | struct structured_reply | | | | | | | | | +----------------------+ | | | | 0 | | uint32_t magic | | | | uint32_t magic | | | | | 4 | | uint32_t error | | | | uint16_t flags | | | | | 6 | | | | | | uint16_t type | | | | | 8 | | uint64_t handle | | | | uint64_t handle | | | | | | +-----------------+ | | | | | | | | 16 | [padding] | | | uint32_t length | | | | | | | | +----------------------+ | | | | | | | union payload | | | | | | | +-----------+----------+ | | | | 20 | | | | ... | ... | | | | | | | | +-----------+----------+ | | | | | | +--------------------------+ | | | +---------------------+------------------------------+ | +------------------------------------------------------------+
to:
offset simple structured +-------------------------------------------------------------+ | union sbuf | | +-----------------------------------------------------+ | | | struct reply | | | | +-------------------------------------------------+ | | | | | union hdr | | | | | | +--------------------+------------------------+ | | | | | | | struct simple | struct structured | | | | | | | | +----------------+ | +--------------------+ | | | | | 0 | | | | uint32_t magic | | | uint32_t magic | | | | | | 4 | | | | uint32_t error | | | uint16_t flags | | | | | | 6 | | | | | | | uint16_t type | | | | | | 8 | | | | uint64_t handle| | | uint64_t handle | | | | | | | | | +----------------+ | | | | | | | | 16 | | | [padding] | | uint32_t length | | | | | | | | | | +--------------------+ | | | | | 20 | | | | [padding] | | | | | | | +--------------------+------------------------+ | | | | | | union payload | | | | | | +--------------------+------------------------+ | | | | 24 | | | ... | ... | | | | | | | +--------------------+------------------------+ | | | | | +-------------------------------------------------+ | | | +-----------------------------------------------------+ | +-------------------------------------------------------------+
Technically, whether the payload union offset moves to byte 24 (with 20-23 now padding) or stays at 20 depends on compiler ABI; but many systems prefer that any struct with a uint64_t provide 8-byte alignment to its containing union.
The commit is largely mechanical, and there should be no semantic change.
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 12 ++-- generator/states-reply-simple.c | 4 +- generator/states-reply-structured.c | 103 ++++++++++++++-------------- generator/states-reply.c | 14 ++-- 4 files changed, 68 insertions(+), 65 deletions(-)
diff --git a/lib/internal.h b/lib/internal.h index 25eeea34..c71980ef 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -231,14 +231,16 @@ struct nbd_handle { struct { struct nbd_fixed_new_option_reply_meta_context context; char str[NBD_MAX_STRING]; - } __attribute__ ((packed)) context; + } __attribute__ ((packed)) context;
Whitespace fixup, tolerable as a part of this patch.
char err_msg[NBD_MAX_STRING]; } payload; - } __attribute__ ((packed)) or; + } or;
So I disagree with doing this here. I'd like it to be its own patch.
struct nbd_export_name_option_reply export_name_reply; - struct nbd_simple_reply simple_reply; struct { - struct nbd_structured_reply structured_reply; + union { + struct nbd_simple_reply simple; + struct nbd_structured_reply structured; + } hdr; union { struct nbd_structured_reply_offset_data offset_data; struct nbd_structured_reply_offset_hole offset_hole;
Yes, fine (although formatting it with a much larger context would have helped -- anyway I'm reviewing the series with -W in my local clone).
I should indeed try harder to follow your useful example of generating specific patches with more than the default 3 lines of context, when it would make review easier. Alas, 'git format-patch' doesn't seem to have an easy way to pick a different context size on a per-patch basis, so this basically implies writing (or finding and reusing existing) wrapper tooling to automate that.
@@ -249,7 +251,7 @@ struct nbd_handle { uint64_t offset; /* Only used for NBD_REPLY_TYPE_ERROR_OFFSET */ } __attribute__ ((packed)) error; } payload; - } __attribute__ ((packed)) sr; + } reply; uint16_t gflags; uint32_t cflags; uint32_t len;
Here too, the __attribute__ ((packed)) removal shoould be in a separate patch. Renaming "sr" to "reply" is "in scope", of course.
diff --git a/generator/states-reply-simple.c b/generator/states-reply-simple.c index 8fd9f62a..e6f1ee23 100644 --- a/generator/states-reply-simple.c +++ b/generator/states-reply-simple.c @@ -23,7 +23,7 @@ REPLY.SIMPLE_REPLY.START: struct command *cmd = h->reply_cmd; uint32_t error;
- error = be32toh (h->sbuf.simple_reply.error); + error = be32toh (h->sbuf.reply.hdr.simple.error);
if (cmd == NULL) { /* Unexpected reply. If error was set or we have structured
Yes.
@@ -39,7 +39,7 @@ REPLY.SIMPLE_REPLY.START: if (error || h->structured_replies) SET_NEXT_STATE (%^FINISH_COMMAND); else { - uint64_t cookie = be64toh (h->sbuf.simple_reply.handle); + uint64_t cookie = be64toh (h->sbuf.reply.hdr.simple.handle); SET_NEXT_STATE (%.DEAD); set_error (EPROTO, "no matching cookie %" PRIu64 " found for server reply, "
Yes; same thing, just different name.
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 96182222..6f96945a 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -49,9 +49,9 @@ REPLY.STRUCTURED_REPLY.START: * so read the remaining part. */ h->rbuf = &h->sbuf; - h->rbuf = (char *)h->rbuf + sizeof h->sbuf.simple_reply; - h->rlen = sizeof h->sbuf.sr.structured_reply; - h->rlen -= sizeof h->sbuf.simple_reply; + h->rbuf = (char *)h->rbuf + sizeof h->sbuf.reply.hdr.simple; + h->rlen = sizeof h->sbuf.reply.hdr.structured; + h->rlen -= sizeof h->sbuf.reply.hdr.simple; SET_NEXT_STATE (%RECV_REMAINING); return 0;
Here I disagree with the mechanical approach.
I even take issue with the pre-patch code. Pre-patch, we fill in "h->sbuf.simple_reply" (in "generator/states-reply.c"), i.e., one member of the top-level "sbuf" union, but then continue filling a member of a *different* member (i.e., "sr.structured_reply") of the "sbuf" union here. This looks undefined by the C standard, but even if it's not undefined, it's supremely confusing.
It happens to work, but I agree with you that we are probably stretching aliasing rules about memory effective types that it is shaky ground to begin with. Even adding a STATIC_ASSERT that offsetof(struct simple_reply, handle) == offsetof(struct structured_reply, handle) would be helpful to show that we depend on that.
Now, if structure "nbd_simple_reply" were the first member of "nbd_structured_reply", maybe we could salvage this somehow -- but that's not the case.
... Right, in "generator/states-reply.c", we have:
/* We read all replies initially as if they are simple replies, but * check the magic in CHECK_SIMPLE_OR_STRUCTURED_REPLY below. * This works because the structured_reply header is larger. */
This is precisely my problem. The common initial subsequence is only the "uint32_t magic" field. Therefore the code in "generator/states-reply.c" should only read the magic field at first, and then branch to simple vs. structured reply.
While both structs share only a common prefix of uint32_t magic, they intentionally DO share uint64_t handle at the SAME offset of 8 bytes in. The commit message tries to show that visually both pre- and post-patch. (Oh, that reminds me; I need to do a patch to s/handle/cookie/ to match my recent patch to the NBD spec.)
Post-patch, the sbuf.reply.hdr union actually reflects this well! The structure members of that union all begin with the common subsequence "uint32_t magic", so using the special guarantee from the C standard, it's fine to fill in "magic" via one union member, and read it back via another. Pre-patch, the guarantee doesn't apply, because "sbuf.simple_reply" and "sbuf.sr.structured_reply" are not members of the same union.
However, even with this patch applied (i.e., with "sbuf.reply.hdr" existing), the code performing the reading continues making me uncomfortable. We effectively have this sequence:
- take address of "h->sbuf.reply.hdr"
- read "sizeof h->sbuf.reply.hdr.simple" bytes into it
- (at this point we can safely access "h->sbuf.reply.hdr.simple", *and* "h->sbuf.reply.hdr.structured.magic" as well -- but not the rest of "h->sbuf.reply.hdr.structured"!)
- take the address of "h->sbuf.reply.hdr.structured.length" -- but not by name; instead, only by virtue of the previous read having filled in "magic", "flags", "type", and "handle", as a *happenstance*, via populating "h->sbuf.reply.hdr.simple"
- read as many bytes as necessary in order to complete "h->sbuf.reply.hdr.structured", from that point onward.
This smells to me.
Optimally, the simple reply and the structured reply should look like this:
struct nbd_reply_header { uint32_t magic; union { struct { uint32_t error; uint64_t handle; } simple; struct { uint16_t flags; uint16_t type; uint64_t handle; uint32_t length; } structured; } magic_specific; };
Hmm. As long as this resides only in sbuf, it makes sense for libnbd. But if we also touch the structs in lib/nbd-protocol.h, then it is code that is shared with nbdkit and has an even wider ripple effect. My initial problem with THIS proposal is that it REQUIRES the use of multiple __attribute__((packed)) if we intend to read in multiple fields in one recv: for example the struct for nbd_reply_header.simple starts with uint32_t followed by uint64_t, so many ABIs want to align it as: offsets 0-3 error offsets 4-7 padding offsets 8-15 handle and when thrown in the larger nbd_reply_header, you now have a natural alignment that prefers: offsets 0-3 magic offsets 4-7 padding offsets 8-11 error offsets 12-15 padding offsets 16-23 handle Note that in lib/nbd-protocol.h, 'struct nbd_simple_reply' is already marked packed; but even if it weren't, the sizings of the member fields HAPPEN to be naturally aligned, for both simple and structured replies (I can't speak to the design of the simple reply since that pre-dates my involvement in Linux, but as I was the one that added structured replies to the NBD spec, it was intentional on my part to pick sizes that filled in all space assuming alignment matches sizing, and where magic and handle both overlap). But a STATIC_ASSERT wouldn't hurt, showing that we want: offsets 0-3 magic offsets 4-7 error OR offsets 4-5 flags, offsets 6-7 type offsets 8-15 handle (Hmm - another thing I should consider: why do we use NBD_ATTRIBUTE_PACKED in nbd-protocol.h but open-code __attribute__((packed)) in internal.h? Consistent spelling would be nice.) A secondary problem with this proposal is what happens later in the series when adding extended headers. When we added structured replies, we were forced to do (at least) two partial reads (which type of reply packet is on the wire, followed by reading the rest of the packet). But with extended headers, we know a priori that all expected responses will have a 32-byte header, and can do a single recv() for all 32 bytes. And for a server that does not support structured replies, always reading 16 bytes happens to work for simple replies (it is only when we have structured replies that the difference of 16 vs. 20 bytes of recv() is dependent on magic). Thus, having the state machine try to favor as few recv() as possible is still a worthwhile goal. But having said all that, maybe I could live with: struct nbd_reply_header { uint32_t magic; union { uint32_t simple_error; struct { uint16_t flags; uint16_t type; } structured; } middle; uint64_t handle; uint32_t length; /* only used for structured */ }; although once extended headers are in play, it would end up looking more like struct nbd_reply_header { uint32_t magic; union { uint32_t simple_error; struct { uint16_t flags; uint16_t type; } structured; /* including extended */ } middle; uint64_t handle; union { /* unused for simple */ uint32_t structured_length; struct { uint64_t offset; uint64_t length; } extended; } tail; }; At any rate, as written there, natural alignment still works for the struct prefix (no 64-bit entities in struct middle.structured, so the overall union middle has only 4-byte alignment and 4-byte size). Okay, natural alignment says the overall struct nbd_reply_header would be 24 bytes, even though structured replies only take 20 bytes (because of the 8-byte handle, the compiler inserts four bytes of padding after length), so we may still want __attribute__((packed)) or at a bare minimum STATIC_ASSERT to show that we insist on handle being at offset 8.
and we should have separate automaton states for reading "magic_specific.simple" and "magic_specific.structured".
In REPLY.START, we should only read "magic".
We should have a sepate state called REPLY.SIMPLE_REPLY.START, for reading "magic_specific.simple".
If we weren't guaranteed that 'handle' overlaps between the two types, I might agree. But my argument is that the wire protocol DOES intend for handle to be at the same offset, regardless of the interpretation of the middle bytes, and forcing multiple recv() is just more syscalls for something that we know we'll eventually need to interpret according to magic. Even if we just read into a char[] and then reinterpret into simple or structured, rather than reading a prefix into simple and then reading more bytes into a tail of structured, is at least fewer syscalls, and also less likely to run foul of C's effective type rules (I've recently had to refamiliarize myself with C17 section 6.5 paragraph 7 as part of working on how POSIX should specify intentional aliasing occuring in struct sockaddr and friends, https://austingroupbugs.net/view.php?id=1641).
In REPLY.STRUCTURED_REPLY.START, we should point h->rbuf at "magic_specific.structured", and read "sizeof magic_specific.structured" bytes.
I'll stop here because there's already much to discuss.
Thanks, Laszlo
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 5/26/23 23:06, Eric Blake wrote:
I should indeed try harder to follow your useful example of generating specific patches with more than the default 3 lines of context, when it would make review easier. Alas, 'git format-patch' doesn't seem to have an easy way to pick a different context size on a per-patch basis, so this basically implies writing (or finding and reusing existing) wrapper tooling to automate that.
If it's of any help, please see my script attached. It lets me put "context:-W" and "context:-U5" style hints in the Notes sections of the patches (with git-notes).
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 96182222..6f96945a 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -49,9 +49,9 @@ REPLY.STRUCTURED_REPLY.START: * so read the remaining part. */ h->rbuf = &h->sbuf; - h->rbuf = (char *)h->rbuf + sizeof h->sbuf.simple_reply; - h->rlen = sizeof h->sbuf.sr.structured_reply; - h->rlen -= sizeof h->sbuf.simple_reply; + h->rbuf = (char *)h->rbuf + sizeof h->sbuf.reply.hdr.simple; + h->rlen = sizeof h->sbuf.reply.hdr.structured; + h->rlen -= sizeof h->sbuf.reply.hdr.simple; SET_NEXT_STATE (%RECV_REMAINING); return 0;
Here I disagree with the mechanical approach.
I even take issue with the pre-patch code. Pre-patch, we fill in "h->sbuf.simple_reply" (in "generator/states-reply.c"), i.e., one member of the top-level "sbuf" union, but then continue filling a member of a *different* member (i.e., "sr.structured_reply") of the "sbuf" union here. This looks undefined by the C standard, but even if it's not undefined, it's supremely confusing.
It happens to work, but I agree with you that we are probably stretching aliasing rules about memory effective types that it is shaky ground to begin with. Even adding a STATIC_ASSERT that offsetof(struct simple_reply, handle) == offsetof(struct structured_reply, handle) would be helpful to show that we depend on that.
After I sent my comments last week, I kept pondering this topic. I now believe my approach to unions in this instance was too rigid. The code is not trivial to read for sure, but there are code comments that help with that. If you can introduce that STATIC_ASSERT, IMO that will suffice, for sticking with this patch. ( Given that we already depend on (non-standard) packing, and depend on the accesses to those packed fields not to fault, it's mostly irrelevant how exactly we calculate the byte offsets. It's really difficult to use unions effectively, if one doesn't go beyond what the standard guarantees. :/ ) Thanks, Laszlo
Add the magic numbers and new structs necessary to implement the NBD protocol extension of extended headers providing 64-bit lengths. This corresponds to upstream nbd commits 36abf47d and a9384e2f on the extension-ext-header branch[1] (commit e6f3b94a for NBD_FLAG_BLOCK_STATUS_PAYLOAD is saved for a later patch). [1] https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-header/doc/prot... Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/nbd-protocol.h | 66 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 14 deletions(-) diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h index f4640a98..b6fa9b8a 100644 --- a/lib/nbd-protocol.h +++ b/lib/nbd-protocol.h @@ -124,6 +124,7 @@ struct nbd_fixed_new_option_reply { #define NBD_OPT_STRUCTURED_REPLY 8 #define NBD_OPT_LIST_META_CONTEXT 9 #define NBD_OPT_SET_META_CONTEXT 10 +#define NBD_OPT_EXTENDED_HEADERS 11 #define NBD_REP_ERR(val) (0x80000000 | (val)) #define NBD_REP_IS_ERR(val) (!!((val) & 0x80000000)) @@ -141,6 +142,7 @@ struct nbd_fixed_new_option_reply { #define NBD_REP_ERR_SHUTDOWN NBD_REP_ERR (7) #define NBD_REP_ERR_BLOCK_SIZE_REQD NBD_REP_ERR (8) #define NBD_REP_ERR_TOO_BIG NBD_REP_ERR (9) +#define NBD_REP_ERR_EXT_HEADER_REQD NBD_REP_ERR (10) #define NBD_INFO_EXPORT 0 #define NBD_INFO_NAME 1 @@ -182,16 +184,26 @@ struct nbd_fixed_new_option_reply_meta_context { /* followed by a string */ } NBD_ATTRIBUTE_PACKED; -/* Request (client -> server). */ +/* Compact request (client -> server). */ struct nbd_request { uint32_t magic; /* NBD_REQUEST_MAGIC. */ - uint16_t flags; /* Request flags. */ - uint16_t type; /* Request type. */ + uint16_t flags; /* Request flags: NBD_CMD_FLAG_*. */ + uint16_t type; /* Request type: NBD_CMD_*. */ uint64_t handle; /* Opaque handle. */ uint64_t offset; /* Request offset. */ uint32_t count; /* Request length. */ } NBD_ATTRIBUTE_PACKED; +/* Extended request (client -> server). */ +struct nbd_request_ext { + uint32_t magic; /* NBD_EXTENDED_REQUEST_MAGIC. */ + uint16_t flags; /* Request flags: NBD_CMD_FLAG_*. */ + uint16_t type; /* Request type: NBD_CMD_*. */ + uint64_t handle; /* Opaque handle. */ + uint64_t offset; /* Request offset. */ + uint64_t count; /* Request effect or payload length. */ +} NBD_ATTRIBUTE_PACKED; + /* Simple reply (server -> client). */ struct nbd_simple_reply { uint32_t magic; /* NBD_SIMPLE_REPLY_MAGIC. */ @@ -208,6 +220,16 @@ struct nbd_structured_reply { uint32_t length; /* Length of payload which follows. */ } NBD_ATTRIBUTE_PACKED; +/* Extended reply (server -> client). */ +struct nbd_extended_reply { + uint32_t magic; /* NBD_EXTENDED_REPLY_MAGIC. */ + uint16_t flags; /* NBD_REPLY_FLAG_* */ + uint16_t type; /* NBD_REPLY_TYPE_* */ + uint64_t handle; /* Opaque handle. */ + uint64_t offset; /* Client's offset. */ + uint64_t length; /* Length of payload which follows. */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_offset_data { uint64_t offset; /* offset */ /* Followed by data. */ @@ -224,11 +246,23 @@ struct nbd_block_descriptor { uint32_t status_flags; /* block type (hole etc) */ } NBD_ATTRIBUTE_PACKED; +/* NBD_REPLY_TYPE_BLOCK_STATUS_EXT block descriptor. */ +struct nbd_block_descriptor_ext { + uint64_t length; /* length of block */ + uint64_t status_flags; /* block type (hole etc) */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_block_status_hdr { uint32_t context_id; /* metadata context ID */ /* followed by array of nbd_block_descriptor extents */ } NBD_ATTRIBUTE_PACKED; +struct nbd_structured_reply_block_status_ext_hdr { + uint32_t context_id; /* metadata context ID */ + uint32_t count; /* length of following array */ + /* followed by array of nbd_block_descriptor_ext extents */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_error { uint32_t error; /* NBD_E* error number */ uint16_t len; /* Length of human readable error. */ @@ -236,8 +270,10 @@ struct nbd_structured_reply_error { } NBD_ATTRIBUTE_PACKED; #define NBD_REQUEST_MAGIC 0x25609513 +#define NBD_EXTENDED_REQUEST_MAGIC 0x21e41c71 #define NBD_SIMPLE_REPLY_MAGIC 0x67446698 #define NBD_STRUCTURED_REPLY_MAGIC 0x668e33ef +#define NBD_EXTENDED_REPLY_MAGIC 0x6e8a278c /* Structured reply flags. */ #define NBD_REPLY_FLAG_DONE (1<<0) @@ -246,12 +282,13 @@ struct nbd_structured_reply_error { #define NBD_REPLY_TYPE_IS_ERR(val) (!!((val) & (1<<15))) /* Structured reply types. */ -#define NBD_REPLY_TYPE_NONE 0 -#define NBD_REPLY_TYPE_OFFSET_DATA 1 -#define NBD_REPLY_TYPE_OFFSET_HOLE 2 -#define NBD_REPLY_TYPE_BLOCK_STATUS 5 -#define NBD_REPLY_TYPE_ERROR NBD_REPLY_TYPE_ERR (1) -#define NBD_REPLY_TYPE_ERROR_OFFSET NBD_REPLY_TYPE_ERR (2) +#define NBD_REPLY_TYPE_NONE 0 +#define NBD_REPLY_TYPE_OFFSET_DATA 1 +#define NBD_REPLY_TYPE_OFFSET_HOLE 2 +#define NBD_REPLY_TYPE_BLOCK_STATUS 5 +#define NBD_REPLY_TYPE_BLOCK_STATUS_EXT 6 +#define NBD_REPLY_TYPE_ERROR NBD_REPLY_TYPE_ERR (1) +#define NBD_REPLY_TYPE_ERROR_OFFSET NBD_REPLY_TYPE_ERR (2) /* NBD commands. */ #define NBD_CMD_READ 0 @@ -263,11 +300,12 @@ struct nbd_structured_reply_error { #define NBD_CMD_WRITE_ZEROES 6 #define NBD_CMD_BLOCK_STATUS 7 -#define NBD_CMD_FLAG_FUA (1<<0) -#define NBD_CMD_FLAG_NO_HOLE (1<<1) -#define NBD_CMD_FLAG_DF (1<<2) -#define NBD_CMD_FLAG_REQ_ONE (1<<3) -#define NBD_CMD_FLAG_FAST_ZERO (1<<4) +#define NBD_CMD_FLAG_FUA (1<<0) +#define NBD_CMD_FLAG_NO_HOLE (1<<1) +#define NBD_CMD_FLAG_DF (1<<2) +#define NBD_CMD_FLAG_REQ_ONE (1<<3) +#define NBD_CMD_FLAG_FAST_ZERO (1<<4) +#define NBD_CMD_FLAG_PAYLOAD_LEN (1<<5) /* NBD error codes. */ #define NBD_SUCCESS 0 -- 2.40.1
On 5/25/23 15:00, Eric Blake wrote:
Add the magic numbers and new structs necessary to implement the NBD protocol extension of extended headers providing 64-bit lengths. This corresponds to upstream nbd commits 36abf47d and a9384e2f on the extension-ext-header branch[1] (commit e6f3b94a for NBD_FLAG_BLOCK_STATUS_PAYLOAD is saved for a later patch).
[1] https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-header/doc/prot...
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/nbd-protocol.h | 66 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 14 deletions(-)
diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h index f4640a98..b6fa9b8a 100644 --- a/lib/nbd-protocol.h +++ b/lib/nbd-protocol.h @@ -124,6 +124,7 @@ struct nbd_fixed_new_option_reply { #define NBD_OPT_STRUCTURED_REPLY 8 #define NBD_OPT_LIST_META_CONTEXT 9 #define NBD_OPT_SET_META_CONTEXT 10 +#define NBD_OPT_EXTENDED_HEADERS 11
#define NBD_REP_ERR(val) (0x80000000 | (val)) #define NBD_REP_IS_ERR(val) (!!((val) & 0x80000000)) @@ -141,6 +142,7 @@ struct nbd_fixed_new_option_reply { #define NBD_REP_ERR_SHUTDOWN NBD_REP_ERR (7) #define NBD_REP_ERR_BLOCK_SIZE_REQD NBD_REP_ERR (8) #define NBD_REP_ERR_TOO_BIG NBD_REP_ERR (9) +#define NBD_REP_ERR_EXT_HEADER_REQD NBD_REP_ERR (10)
#define NBD_INFO_EXPORT 0 #define NBD_INFO_NAME 1 @@ -182,16 +184,26 @@ struct nbd_fixed_new_option_reply_meta_context { /* followed by a string */ } NBD_ATTRIBUTE_PACKED;
-/* Request (client -> server). */ +/* Compact request (client -> server). */ struct nbd_request { uint32_t magic; /* NBD_REQUEST_MAGIC. */ - uint16_t flags; /* Request flags. */ - uint16_t type; /* Request type. */ + uint16_t flags; /* Request flags: NBD_CMD_FLAG_*. */ + uint16_t type; /* Request type: NBD_CMD_*. */ uint64_t handle; /* Opaque handle. */ uint64_t offset; /* Request offset. */ uint32_t count; /* Request length. */ } NBD_ATTRIBUTE_PACKED;
+/* Extended request (client -> server). */ +struct nbd_request_ext { + uint32_t magic; /* NBD_EXTENDED_REQUEST_MAGIC. */ + uint16_t flags; /* Request flags: NBD_CMD_FLAG_*. */ + uint16_t type; /* Request type: NBD_CMD_*. */ + uint64_t handle; /* Opaque handle. */ + uint64_t offset; /* Request offset. */ + uint64_t count; /* Request effect or payload length. */ +} NBD_ATTRIBUTE_PACKED; + /* Simple reply (server -> client). */ struct nbd_simple_reply { uint32_t magic; /* NBD_SIMPLE_REPLY_MAGIC. */ @@ -208,6 +220,16 @@ struct nbd_structured_reply { uint32_t length; /* Length of payload which follows. */ } NBD_ATTRIBUTE_PACKED;
+/* Extended reply (server -> client). */ +struct nbd_extended_reply { + uint32_t magic; /* NBD_EXTENDED_REPLY_MAGIC. */ + uint16_t flags; /* NBD_REPLY_FLAG_* */ + uint16_t type; /* NBD_REPLY_TYPE_* */ + uint64_t handle; /* Opaque handle. */ + uint64_t offset; /* Client's offset. */ + uint64_t length; /* Length of payload which follows. */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_offset_data { uint64_t offset; /* offset */ /* Followed by data. */ @@ -224,11 +246,23 @@ struct nbd_block_descriptor { uint32_t status_flags; /* block type (hole etc) */ } NBD_ATTRIBUTE_PACKED;
+/* NBD_REPLY_TYPE_BLOCK_STATUS_EXT block descriptor. */ +struct nbd_block_descriptor_ext { + uint64_t length; /* length of block */ + uint64_t status_flags; /* block type (hole etc) */
I wonder if making the status_flags fields uint64_t too was really necessary, or just a consequence of us wanting to treat a sequence of these records as a (double as long) sequence of uint64_t's. Anyway, this is a spec-level question, and I totally don't want to question the spec. :)
+} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_block_status_hdr { uint32_t context_id; /* metadata context ID */ /* followed by array of nbd_block_descriptor extents */ } NBD_ATTRIBUTE_PACKED;
+struct nbd_structured_reply_block_status_ext_hdr { + uint32_t context_id; /* metadata context ID */ + uint32_t count; /* length of following array */
(1) I think that "length of following array" is confusing here. Per spec, this is "descriptor count" -- that's clearer. "Length" could be mistaken for "number of bytes". Also, what was the justification for *not* making "count" uint64_t? (Just asking.) I do understand a server is permitted to respond with a block status reply that covers less than the requested range, so I understand a server, if it needs to, *can* truncate its response for the sake of fitting "count" into a uint32_t.
+ /* followed by array of nbd_block_descriptor_ext extents */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_error { uint32_t error; /* NBD_E* error number */ uint16_t len; /* Length of human readable error. */ @@ -236,8 +270,10 @@ struct nbd_structured_reply_error { } NBD_ATTRIBUTE_PACKED;
#define NBD_REQUEST_MAGIC 0x25609513 +#define NBD_EXTENDED_REQUEST_MAGIC 0x21e41c71 #define NBD_SIMPLE_REPLY_MAGIC 0x67446698 #define NBD_STRUCTURED_REPLY_MAGIC 0x668e33ef +#define NBD_EXTENDED_REPLY_MAGIC 0x6e8a278c
/* Structured reply flags. */ #define NBD_REPLY_FLAG_DONE (1<<0) @@ -246,12 +282,13 @@ struct nbd_structured_reply_error { #define NBD_REPLY_TYPE_IS_ERR(val) (!!((val) & (1<<15)))
/* Structured reply types. */ -#define NBD_REPLY_TYPE_NONE 0 -#define NBD_REPLY_TYPE_OFFSET_DATA 1 -#define NBD_REPLY_TYPE_OFFSET_HOLE 2 -#define NBD_REPLY_TYPE_BLOCK_STATUS 5 -#define NBD_REPLY_TYPE_ERROR NBD_REPLY_TYPE_ERR (1) -#define NBD_REPLY_TYPE_ERROR_OFFSET NBD_REPLY_TYPE_ERR (2) +#define NBD_REPLY_TYPE_NONE 0 +#define NBD_REPLY_TYPE_OFFSET_DATA 1 +#define NBD_REPLY_TYPE_OFFSET_HOLE 2 +#define NBD_REPLY_TYPE_BLOCK_STATUS 5 +#define NBD_REPLY_TYPE_BLOCK_STATUS_EXT 6 +#define NBD_REPLY_TYPE_ERROR NBD_REPLY_TYPE_ERR (1) +#define NBD_REPLY_TYPE_ERROR_OFFSET NBD_REPLY_TYPE_ERR (2)
/* NBD commands. */ #define NBD_CMD_READ 0 @@ -263,11 +300,12 @@ struct nbd_structured_reply_error { #define NBD_CMD_WRITE_ZEROES 6 #define NBD_CMD_BLOCK_STATUS 7
-#define NBD_CMD_FLAG_FUA (1<<0) -#define NBD_CMD_FLAG_NO_HOLE (1<<1) -#define NBD_CMD_FLAG_DF (1<<2) -#define NBD_CMD_FLAG_REQ_ONE (1<<3) -#define NBD_CMD_FLAG_FAST_ZERO (1<<4) +#define NBD_CMD_FLAG_FUA (1<<0) +#define NBD_CMD_FLAG_NO_HOLE (1<<1) +#define NBD_CMD_FLAG_DF (1<<2) +#define NBD_CMD_FLAG_REQ_ONE (1<<3) +#define NBD_CMD_FLAG_FAST_ZERO (1<<4) +#define NBD_CMD_FLAG_PAYLOAD_LEN (1<<5)
Ah, this seems like a nice addition: "... the client sets NBD_CMD_FLAG_PAYLOAD_LEN in order to pass a payload that informs the server to limit its replies to the metacontext id(s) in the client's request payload, rather than giving an answer on all possible metacontext ids".
/* NBD error codes. */ #define NBD_SUCCESS 0
With (1) clarified: Reviewed-by: Laszlo Ersek <lersek@redhat.com>
On Tue, May 30, 2023 at 01:50:59PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
Add the magic numbers and new structs necessary to implement the NBD protocol extension of extended headers providing 64-bit lengths. This corresponds to upstream nbd commits 36abf47d and a9384e2f on the extension-ext-header branch[1] (commit e6f3b94a for NBD_FLAG_BLOCK_STATUS_PAYLOAD is saved for a later patch).
[1] https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-header/doc/prot...
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/nbd-protocol.h | 66 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 14 deletions(-)
diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h index f4640a98..b6fa9b8a 100644 --- a/lib/nbd-protocol.h +++ b/lib/nbd-protocol.h @@ -124,6 +124,7 @@ struct nbd_fixed_new_option_reply { #define NBD_OPT_STRUCTURED_REPLY 8 #define NBD_OPT_LIST_META_CONTEXT 9 #define NBD_OPT_SET_META_CONTEXT 10 +#define NBD_OPT_EXTENDED_HEADERS 11
#define NBD_REP_ERR(val) (0x80000000 | (val)) #define NBD_REP_IS_ERR(val) (!!((val) & 0x80000000)) @@ -141,6 +142,7 @@ struct nbd_fixed_new_option_reply { #define NBD_REP_ERR_SHUTDOWN NBD_REP_ERR (7) #define NBD_REP_ERR_BLOCK_SIZE_REQD NBD_REP_ERR (8) #define NBD_REP_ERR_TOO_BIG NBD_REP_ERR (9) +#define NBD_REP_ERR_EXT_HEADER_REQD NBD_REP_ERR (10)
#define NBD_INFO_EXPORT 0 #define NBD_INFO_NAME 1 @@ -182,16 +184,26 @@ struct nbd_fixed_new_option_reply_meta_context { /* followed by a string */ } NBD_ATTRIBUTE_PACKED;
-/* Request (client -> server). */ +/* Compact request (client -> server). */ struct nbd_request { uint32_t magic; /* NBD_REQUEST_MAGIC. */ - uint16_t flags; /* Request flags. */ - uint16_t type; /* Request type. */ + uint16_t flags; /* Request flags: NBD_CMD_FLAG_*. */ + uint16_t type; /* Request type: NBD_CMD_*. */ uint64_t handle; /* Opaque handle. */ uint64_t offset; /* Request offset. */ uint32_t count; /* Request length. */ } NBD_ATTRIBUTE_PACKED;
+/* Extended request (client -> server). */ +struct nbd_request_ext { + uint32_t magic; /* NBD_EXTENDED_REQUEST_MAGIC. */ + uint16_t flags; /* Request flags: NBD_CMD_FLAG_*. */ + uint16_t type; /* Request type: NBD_CMD_*. */ + uint64_t handle; /* Opaque handle. */ + uint64_t offset; /* Request offset. */ + uint64_t count; /* Request effect or payload length. */ +} NBD_ATTRIBUTE_PACKED; + /* Simple reply (server -> client). */ struct nbd_simple_reply { uint32_t magic; /* NBD_SIMPLE_REPLY_MAGIC. */ @@ -208,6 +220,16 @@ struct nbd_structured_reply { uint32_t length; /* Length of payload which follows. */ } NBD_ATTRIBUTE_PACKED;
+/* Extended reply (server -> client). */ +struct nbd_extended_reply { + uint32_t magic; /* NBD_EXTENDED_REPLY_MAGIC. */ + uint16_t flags; /* NBD_REPLY_FLAG_* */ + uint16_t type; /* NBD_REPLY_TYPE_* */ + uint64_t handle; /* Opaque handle. */ + uint64_t offset; /* Client's offset. */ + uint64_t length; /* Length of payload which follows. */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_offset_data { uint64_t offset; /* offset */ /* Followed by data. */ @@ -224,11 +246,23 @@ struct nbd_block_descriptor { uint32_t status_flags; /* block type (hole etc) */ } NBD_ATTRIBUTE_PACKED;
+/* NBD_REPLY_TYPE_BLOCK_STATUS_EXT block descriptor. */ +struct nbd_block_descriptor_ext { + uint64_t length; /* length of block */ + uint64_t status_flags; /* block type (hole etc) */
I wonder if making the status_flags fields uint64_t too was really necessary, or just a consequence of us wanting to treat a sequence of these records as a (double as long) sequence of uint64_t's. Anyway, this is a spec-level question, and I totally don't want to question the spec. :)
The actual interpretation of the status flag is up to the spec of the metadata namespace that is being used for this particular request. Some metadata namespaces treat the "flags" field as an unsigned integer, to specify offsets or counters. I that context, the flags field should indeed be extended to 64 bits.
+} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_block_status_hdr { uint32_t context_id; /* metadata context ID */ /* followed by array of nbd_block_descriptor extents */ } NBD_ATTRIBUTE_PACKED;
+struct nbd_structured_reply_block_status_ext_hdr { + uint32_t context_id; /* metadata context ID */ + uint32_t count; /* length of following array */
(1) I think that "length of following array" is confusing here. Per spec, this is "descriptor count" -- that's clearer. "Length" could be mistaken for "number of bytes".
Also, what was the justification for *not* making "count" uint64_t? (Just asking.) I do understand a server is permitted to respond with a block status reply that covers less than the requested range, so I understand a server, if it needs to, *can* truncate its response for the sake of fitting "count" into a uint32_t.
This is, as you say, the number of block descriptor messages we are going to send to the client. Each such message is at 8 bytes long. That would mean that by the time you overflow a uint32_t with the number of extents that are to be sent, you'll be promising to send 2^32 * 8 (i.e., 2^36) bytes of data, or 32 GiB. That would be a ridiculously amount of data, and no user is going to wait for a client to finish parsing that amount of data without a hard reboot of their client. The only change that would be reasonable here is to reduce the size of this field 16 to bits, rather than increasing it (but then we lose alignment, so that's also not a good idea)
+ /* followed by array of nbd_block_descriptor_ext extents */ +} NBD_ATTRIBUTE_PACKED; + struct nbd_structured_reply_error { uint32_t error; /* NBD_E* error number */ uint16_t len; /* Length of human readable error. */ @@ -236,8 +270,10 @@ struct nbd_structured_reply_error { } NBD_ATTRIBUTE_PACKED;
#define NBD_REQUEST_MAGIC 0x25609513 +#define NBD_EXTENDED_REQUEST_MAGIC 0x21e41c71 #define NBD_SIMPLE_REPLY_MAGIC 0x67446698 #define NBD_STRUCTURED_REPLY_MAGIC 0x668e33ef +#define NBD_EXTENDED_REPLY_MAGIC 0x6e8a278c
/* Structured reply flags. */ #define NBD_REPLY_FLAG_DONE (1<<0) @@ -246,12 +282,13 @@ struct nbd_structured_reply_error { #define NBD_REPLY_TYPE_IS_ERR(val) (!!((val) & (1<<15)))
/* Structured reply types. */ -#define NBD_REPLY_TYPE_NONE 0 -#define NBD_REPLY_TYPE_OFFSET_DATA 1 -#define NBD_REPLY_TYPE_OFFSET_HOLE 2 -#define NBD_REPLY_TYPE_BLOCK_STATUS 5 -#define NBD_REPLY_TYPE_ERROR NBD_REPLY_TYPE_ERR (1) -#define NBD_REPLY_TYPE_ERROR_OFFSET NBD_REPLY_TYPE_ERR (2) +#define NBD_REPLY_TYPE_NONE 0 +#define NBD_REPLY_TYPE_OFFSET_DATA 1 +#define NBD_REPLY_TYPE_OFFSET_HOLE 2 +#define NBD_REPLY_TYPE_BLOCK_STATUS 5 +#define NBD_REPLY_TYPE_BLOCK_STATUS_EXT 6 +#define NBD_REPLY_TYPE_ERROR NBD_REPLY_TYPE_ERR (1) +#define NBD_REPLY_TYPE_ERROR_OFFSET NBD_REPLY_TYPE_ERR (2)
/* NBD commands. */ #define NBD_CMD_READ 0 @@ -263,11 +300,12 @@ struct nbd_structured_reply_error { #define NBD_CMD_WRITE_ZEROES 6 #define NBD_CMD_BLOCK_STATUS 7
-#define NBD_CMD_FLAG_FUA (1<<0) -#define NBD_CMD_FLAG_NO_HOLE (1<<1) -#define NBD_CMD_FLAG_DF (1<<2) -#define NBD_CMD_FLAG_REQ_ONE (1<<3) -#define NBD_CMD_FLAG_FAST_ZERO (1<<4) +#define NBD_CMD_FLAG_FUA (1<<0) +#define NBD_CMD_FLAG_NO_HOLE (1<<1) +#define NBD_CMD_FLAG_DF (1<<2) +#define NBD_CMD_FLAG_REQ_ONE (1<<3) +#define NBD_CMD_FLAG_FAST_ZERO (1<<4) +#define NBD_CMD_FLAG_PAYLOAD_LEN (1<<5)
Ah, this seems like a nice addition: "... the client sets NBD_CMD_FLAG_PAYLOAD_LEN in order to pass a payload that informs the server to limit its replies to the metacontext id(s) in the client's request payload, rather than giving an answer on all possible metacontext ids".
/* NBD error codes. */ #define NBD_SUCCESS 0
With (1) clarified:
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
-- w@uter.{be,co.za} wouter@{grep.be,fosdem.org,debian.org} I will have a Tin-Actinium-Potassium mixture, thanks.
On 5/30/23 16:56, Wouter Verhelst wrote:
On Tue, May 30, 2023 at 01:50:59PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
+struct nbd_structured_reply_block_status_ext_hdr { + uint32_t context_id; /* metadata context ID */ + uint32_t count; /* length of following array */
(1) I think that "length of following array" is confusing here. Per spec, this is "descriptor count" -- that's clearer. "Length" could be mistaken for "number of bytes".
Also, what was the justification for *not* making "count" uint64_t? (Just asking.) I do understand a server is permitted to respond with a block status reply that covers less than the requested range, so I understand a server, if it needs to, *can* truncate its response for the sake of fitting "count" into a uint32_t.
This is, as you say, the number of block descriptor messages we are going to send to the client. Each such message is at 8 bytes long.
16 bytes, isn't it? (uint64_t length, uint64_t status_flags)
That would mean that by the time you overflow a uint32_t with the number of extents that are to be sent, you'll be promising to send 2^32 * 8 (i.e., 2^36) bytes of data, or 32 GiB.
(Or 64 GiB if we agree that sizeof(nbd_block_descriptor_ext)=16.) But... yes, I'm aware this is exorbitant, practically speaking. My concern was that "practical considerations" must have been what led to the original 32-bit-only: struct nbd_block_descriptor { uint32_t length; /* length of block */ uint32_t status_flags; /* block type (hole etc) */ } NBD_ATTRIBUTE_PACKED; which is now proving too restrictive (being the basis for this entire work, IIUC). A 2^(32+9) B = 2 TB image is not unthinkable. If the image used 512B sectors, and sectors alternated between being a hole and being allocated, then 2^32 extended descriptors would be justified. May not be practical, but that's "policy"; the "mechanism" could still exist (if it doesn't come with undesirable costs).
That would be a ridiculously amount of data, and no user is going to wait for a client to finish parsing that amount of data without a hard reboot of their client.
Over a 10gbit/s connection, transferring 64GB should take on the order of a minute. ... *parsing* 4.3 billion entries is a different matter, of course ;) OK!
The only change that would be reasonable here is to reduce the size of this field 16 to bits, rather than increasing it (but then we lose alignment, so that's also not a good idea)
Putting aside alignment even, I don't understand why reducing "count" to uint16_t would be reasonable. With the current 32-bit-only block descriptor, we already need to write loops in libnbd clients, because we can't cover the entire remote image in one API call [*]. If I understood Eric right earlier, the 64-bit extensions were supposed to remedy that -- but as it stands, clients will still need loops ("chunking") around block status fetching; is that right? Let me emphasize again that I'm not challenging the spec; also I don't secretly wish for the patches to do more than required by the spec. I guess I'd like to understand the intent of the spec, plus the consequences for client applications. [*] References (in this order!): - https://github.com/libguestfs/virt-v2v/commit/27c056cdc6aa0 - https://gitlab.com/nbdkit/libnbd/-/commit/0e714a6e06e6 - https://github.com/libguestfs/virt-v2v/commit/a2afed32d8b1f To be less cryptic, the first commit added "chunked" block status fetching to virt-v2v. Because our OCaml language libnbd mappings weren't proper at the time, that loop could move backwards and get stuck. We fixed that in the second commit (regarding the bindings) and then adapted virt-v2v's loop to the fixed bindings in the thirds commit. I've been hoping that such complexities could be eliminated in the future by virtue of not having to "chunk" the block status fetching. ( BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API. I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t. ) Thanks! Laszlo
On Tue, May 30, 2023 at 05:48:25PM +0200, Laszlo Ersek wrote:
On 5/30/23 16:56, Wouter Verhelst wrote:
On Tue, May 30, 2023 at 01:50:59PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
+struct nbd_structured_reply_block_status_ext_hdr { + uint32_t context_id; /* metadata context ID */ + uint32_t count; /* length of following array */
(1) I think that "length of following array" is confusing here. Per spec, this is "descriptor count" -- that's clearer. "Length" could be mistaken for "number of bytes".
Also, what was the justification for *not* making "count" uint64_t? (Just asking.) I do understand a server is permitted to respond with a block status reply that covers less than the requested range, so I understand a server, if it needs to, *can* truncate its response for the sake of fitting "count" into a uint32_t.
This is, as you say, the number of block descriptor messages we are going to send to the client. Each such message is at 8 bytes long.
16 bytes, isn't it? (uint64_t length, uint64_t status_flags)
Existing block status was 8 bytes per extent (uint32_t * 2), but yes, the extended extent info is 16 bytes (uint64_t * 2).
That would mean that by the time you overflow a uint32_t with the number of extents that are to be sent, you'll be promising to send 2^32 * 8 (i.e., 2^36) bytes of data, or 32 GiB.
(Or 64 GiB if we agree that sizeof(nbd_block_descriptor_ext)=16.)
But... yes, I'm aware this is exorbitant, practically speaking.
We already coded into the NBD spec (commit 926a51df) that a server SHOULD truncate its response rather than sending more than 2M extent entries; I picked that number because 32M (the existing payload cap in qemu) / 16 bytes is 2M extents (for non-extended headers, it actually caps things at 16M instead of 32M). In that commit, I pointed out that qemu actually caps things at 128k entries (payload cap 1M without extended headers), and nbdkit caps things at 1M entries.
My concern was that "practical considerations" must have been what led to the original 32-bit-only:
struct nbd_block_descriptor { uint32_t length; /* length of block */ uint32_t status_flags; /* block type (hole etc) */ } NBD_ATTRIBUTE_PACKED;
which is now proving too restrictive (being the basis for this entire work, IIUC).
A 2^(32+9) B = 2 TB image is not unthinkable. If the image used 512B sectors, and sectors alternated between being a hole and being allocated, then 2^32 extended descriptors would be justified.
May not be practical, but that's "policy"; the "mechanism" could still exist (if it doesn't come with undesirable costs).
That would be a ridiculously amount of data, and no user is going to wait for a client to finish parsing that amount of data without a hard reboot of their client.
Over a 10gbit/s connection, transferring 64GB should take on the order of a minute.
... *parsing* 4.3 billion entries is a different matter, of course ;)
OK!
The only change that would be reasonable here is to reduce the size of this field 16 to bits, rather than increasing it (but then we lose alignment, so that's also not a good idea)
Putting aside alignment even, I don't understand why reducing "count" to uint16_t would be reasonable. With the current 32-bit-only block descriptor, we already need to write loops in libnbd clients, because we can't cover the entire remote image in one API call [*]. If I understood Eric right earlier, the 64-bit extensions were supposed to remedy that -- but as it stands, clients will still need loops ("chunking") around block status fetching; is that right?
While the larger extents reduce the need for looping, it does not entirely eliminate it. For example, just because the server can now tell you that an image is entirely data in just one reply does not mean that it will actually do so - qemu in particular limits block status of a qcow2 file to reporting just one cluster at a time for consistency reasons, where even if you use the maximum size of 2M clusters, you can never get more than (2M/16)*2M = 256G status reported in a single request. But without 64-bit lengths, you are guaranteed to need at least 2 (and possibly more) round trips to map out an entire 6G image, while with 64-bit lengths, you can often get an entire image map in one round trip. Reducing 'count' to uint16_t for 64-bit responses would be possible if we wanted to hard-code a server limit of never sending more than 64k extents in one go; but above, I argued that existing servers currently do exceed that cap even for 32-bit responses.
Let me emphasize again that I'm not challenging the spec; also I don't secretly wish for the patches to do more than required by the spec. I guess I'd like to understand the intent of the spec, plus the consequences for client applications.
[*] References (in this order!):
- https://github.com/libguestfs/virt-v2v/commit/27c056cdc6aa0 - https://gitlab.com/nbdkit/libnbd/-/commit/0e714a6e06e6 - https://github.com/libguestfs/virt-v2v/commit/a2afed32d8b1f
To be less cryptic, the first commit added "chunked" block status fetching to virt-v2v. Because our OCaml language libnbd mappings weren't proper at the time, that loop could move backwards and get stuck. We fixed that in the second commit (regarding the bindings) and then adapted virt-v2v's loop to the fixed bindings in the thirds commit. I've been hoping that such complexities could be eliminated in the future by virtue of not having to "chunk" the block status fetching.
We'll always have to deal with servers that send shorter replies than we asked for, and where consecutive replies may have the same status. The best the spec was able to do was recommend that the server return as much as it can, and without consecutive status, where feasible.
(
BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API.
At one point, I was playing with an idea to add a patch to the NBD spec to REQUIRE that an export be capped at 2^63-1 bytes (that is, the maximum of 'off_t'). It doesn't change any existing implementations, and actually frees us up to use signed 64-bit numbers where negative values are indeed error cases. I'll probably try to revisit my thoughts on that patch front, but don't think it holds up this series.
I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t.
I _did_ go with the (currently implicit) assumption that no server will ever expose larger than off_t can represent, and therefore a signed 64-bit size is good enough. Flags has to be unsigned, but flags is representing something different than image size. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 5/30/23 20:48, Eric Blake wrote:
On Tue, May 30, 2023 at 05:48:25PM +0200, Laszlo Ersek wrote:
On 5/30/23 16:56, Wouter Verhelst wrote:
On Tue, May 30, 2023 at 01:50:59PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
+struct nbd_structured_reply_block_status_ext_hdr { + uint32_t context_id; /* metadata context ID */ + uint32_t count; /* length of following array */
(1) I think that "length of following array" is confusing here. Per spec, this is "descriptor count" -- that's clearer. "Length" could be mistaken for "number of bytes".
Also, what was the justification for *not* making "count" uint64_t? (Just asking.) I do understand a server is permitted to respond with a block status reply that covers less than the requested range, so I understand a server, if it needs to, *can* truncate its response for the sake of fitting "count" into a uint32_t.
This is, as you say, the number of block descriptor messages we are going to send to the client. Each such message is at 8 bytes long.
16 bytes, isn't it? (uint64_t length, uint64_t status_flags)
Existing block status was 8 bytes per extent (uint32_t * 2), but yes, the extended extent info is 16 bytes (uint64_t * 2).
That would mean that by the time you overflow a uint32_t with the number of extents that are to be sent, you'll be promising to send 2^32 * 8 (i.e., 2^36) bytes of data, or 32 GiB.
(Or 64 GiB if we agree that sizeof(nbd_block_descriptor_ext)=16.)
But... yes, I'm aware this is exorbitant, practically speaking.
We already coded into the NBD spec (commit 926a51df) that a server SHOULD truncate its response rather than sending more than 2M extent entries; I picked that number because 32M (the existing payload cap in qemu) / 16 bytes is 2M extents (for non-extended headers, it actually caps things at 16M instead of 32M). In that commit, I pointed out that qemu actually caps things at 128k entries (payload cap 1M without extended headers), and nbdkit caps things at 1M entries.
[*]
My concern was that "practical considerations" must have been what led to the original 32-bit-only:
struct nbd_block_descriptor { uint32_t length; /* length of block */ uint32_t status_flags; /* block type (hole etc) */ } NBD_ATTRIBUTE_PACKED;
which is now proving too restrictive (being the basis for this entire work, IIUC).
A 2^(32+9) B = 2 TB image is not unthinkable. If the image used 512B sectors, and sectors alternated between being a hole and being allocated, then 2^32 extended descriptors would be justified.
May not be practical, but that's "policy"; the "mechanism" could still exist (if it doesn't come with undesirable costs).
That would be a ridiculously amount of data, and no user is going to wait for a client to finish parsing that amount of data without a hard reboot of their client.
Over a 10gbit/s connection, transferring 64GB should take on the order of a minute.
... *parsing* 4.3 billion entries is a different matter, of course ;)
OK!
The only change that would be reasonable here is to reduce the size of this field 16 to bits, rather than increasing it (but then we lose alignment, so that's also not a good idea)
Putting aside alignment even, I don't understand why reducing "count" to uint16_t would be reasonable. With the current 32-bit-only block descriptor, we already need to write loops in libnbd clients, because we can't cover the entire remote image in one API call [*]. If I understood Eric right earlier, the 64-bit extensions were supposed to remedy that -- but as it stands, clients will still need loops ("chunking") around block status fetching; is that right?
While the larger extents reduce the need for looping, it does not entirely eliminate it. For example, just because the server can now tell you that an image is entirely data in just one reply does not mean that it will actually do so - qemu in particular limits block status of a qcow2 file to reporting just one cluster at a time for consistency reasons, where even if you use the maximum size of 2M clusters, you can never get more than (2M/16)*2M = 256G status reported in a single request.
I don't understand the calculation. I can imagine the following interpretation: - QEMU never sends more than 128K block descriptors, and each descriptor covers one 2MB sized cluster --> 256 GB of the disk covered in one go. But I don't understand where the (2M/16) division comes from, even though the quotient is 128K. I can connect the constant "128K", and <https://github.com/NetworkBlockDevice/nbd/commit/926a51df>, to your paragraph [*] above, but not the division.
But without 64-bit lengths, you are guaranteed to need at least 2 (and possibly more) round trips to map out an entire 6G image, while with 64-bit lengths, you can often get an entire image map in one round trip.
Reducing 'count' to uint16_t for 64-bit responses would be possible if we wanted to hard-code a server limit of never sending more than 64k extents in one go; but above, I argued that existing servers currently do exceed that cap even for 32-bit responses.
OK, understood. Looping is still required, but we expect to see it less exercised in practice, with extended headers.
Let me emphasize again that I'm not challenging the spec; also I don't secretly wish for the patches to do more than required by the spec. I guess I'd like to understand the intent of the spec, plus the consequences for client applications.
[*] References (in this order!):
- https://github.com/libguestfs/virt-v2v/commit/27c056cdc6aa0 - https://gitlab.com/nbdkit/libnbd/-/commit/0e714a6e06e6 - https://github.com/libguestfs/virt-v2v/commit/a2afed32d8b1f
To be less cryptic, the first commit added "chunked" block status fetching to virt-v2v. Because our OCaml language libnbd mappings weren't proper at the time, that loop could move backwards and get stuck. We fixed that in the second commit (regarding the bindings) and then adapted virt-v2v's loop to the fixed bindings in the thirds commit. I've been hoping that such complexities could be eliminated in the future by virtue of not having to "chunk" the block status fetching.
We'll always have to deal with servers that send shorter replies than we asked for, and where consecutive replies may have the same status. The best the spec was able to do was recommend that the server return as much as it can, and without consecutive status, where feasible.
(
BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API.
At one point, I was playing with an idea to add a patch to the NBD spec to REQUIRE that an export be capped at 2^63-1 bytes (that is, the maximum of 'off_t'). It doesn't change any existing implementations, and actually frees us up to use signed 64-bit numbers where negative values are indeed error cases. I'll probably try to revisit my thoughts on that patch front, but don't think it holds up this series.
I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t.
I _did_ go with the (currently implicit) assumption that no server will ever expose larger than off_t can represent, and therefore a signed 64-bit size is good enough.
I'll have to see that patch later, but for now I think making that assumption explicit would be nice (assert, abort, etc).
Flags has to be unsigned, but flags is representing something different than image size.
I take it you mean "nbd_block_descriptor_ext.status_flags", which is a "uint64_t". If it's only used as a bitmap, that should be safe; first because some spec (dependent on metacontext) will introduce each bit one by one, and probably not start with bit#63; second because it makes no sense to sum bitmaps. However, Wouter mentions up-thread that (dependent on metacontext) status_flags could actually stand for a length too: 'Some metadata namespaces treat the "flags" field as an unsigned integer, to specify offsets or counters. I that context, the flags field should indeed be extended to 64 bits.' and then dealing with those particular metacontexts in OCaml could be a problem. It's probably best to catch unrepresentable values when converting between OCaml's int64 and C's uint64_t (both ways) in the generator as centrally as possible. Laszlo
On Wed, May 31, 2023 at 01:29:30PM +0200, Laszlo Ersek wrote:
Putting aside alignment even, I don't understand why reducing "count" to uint16_t would be reasonable. With the current 32-bit-only block descriptor, we already need to write loops in libnbd clients, because we can't cover the entire remote image in one API call [*]. If I understood Eric right earlier, the 64-bit extensions were supposed to remedy that -- but as it stands, clients will still need loops ("chunking") around block status fetching; is that right?
While the larger extents reduce the need for looping, it does not entirely eliminate it. For example, just because the server can now tell you that an image is entirely data in just one reply does not mean that it will actually do so - qemu in particular limits block status of a qcow2 file to reporting just one cluster at a time for consistency reasons, where even if you use the maximum size of 2M clusters, you can never get more than (2M/16)*2M = 256G status reported in a single request.
I don't understand the calculation. I can imagine the following interpretation:
- QEMU never sends more than 128K block descriptors, and each descriptor covers one 2MB sized cluster --> 256 GB of the disk covered in one go.
But I don't understand where the (2M/16) division comes from, even though the quotient is 128K.
Ah, I need to provide more backstory on the qcow2 format. A qcow2 image has a fixed cluster size, chosen between between 512 and 2M bytes. A smaller cluster size has less wasted space for small images, but uses more overhead. Each cluster has to be stored in an L1 map, where pages of the map are also a cluster in length, with 16 bytes per map entry. So if you pick a cluster size of 512, you get 512/16 or 32 entries per L1 page; if you pick a cluster size of 2M, you get 2M/16 or 128k entries per L1 page. When reporting block status, qemu reads at most one L1 page to then say how each cluster referenced from that page is mapped. https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/qcow2.txt#L4...
I can connect the constant "128K", and <https://github.com/NetworkBlockDevice/nbd/commit/926a51df>, to your paragraph [*] above, but not the division.
In this case, the qemu limit on reporting block status of at most one L1 map page at a time happens to have no relationship to the NBD constant of limiting block status reports to no more than 1M extents (8M bytes) in a single reply, nor the fact that qemu picked a cap of 1M bytes (128k extents) on its NBD reply regardless of whether the underlying image is qcow2 or some other format.
But without 64-bit lengths, you are guaranteed to need at least 2 (and possibly more) round trips to map out an entire 6G image, while with 64-bit lengths, you can often get an entire image map in one round trip.
Reducing 'count' to uint16_t for 64-bit responses would be possible if we wanted to hard-code a server limit of never sending more than 64k extents in one go; but above, I argued that existing servers currently do exceed that cap even for 32-bit responses.
OK, understood. Looping is still required, but we expect to see it less exercised in practice, with extended headers.
Let me emphasize again that I'm not challenging the spec; also I don't secretly wish for the patches to do more than required by the spec. I guess I'd like to understand the intent of the spec, plus the consequences for client applications.
[*] References (in this order!):
- https://github.com/libguestfs/virt-v2v/commit/27c056cdc6aa0 - https://gitlab.com/nbdkit/libnbd/-/commit/0e714a6e06e6 - https://github.com/libguestfs/virt-v2v/commit/a2afed32d8b1f
To be less cryptic, the first commit added "chunked" block status fetching to virt-v2v. Because our OCaml language libnbd mappings weren't proper at the time, that loop could move backwards and get stuck. We fixed that in the second commit (regarding the bindings) and then adapted virt-v2v's loop to the fixed bindings in the thirds commit. I've been hoping that such complexities could be eliminated in the future by virtue of not having to "chunk" the block status fetching.
We'll always have to deal with servers that send shorter replies than we asked for, and where consecutive replies may have the same status. The best the spec was able to do was recommend that the server return as much as it can, and without consecutive status, where feasible.
(
BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API.
At one point, I was playing with an idea to add a patch to the NBD spec to REQUIRE that an export be capped at 2^63-1 bytes (that is, the maximum of 'off_t'). It doesn't change any existing implementations, and actually frees us up to use signed 64-bit numbers where negative values are indeed error cases. I'll probably try to revisit my thoughts on that patch front, but don't think it holds up this series.
I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t.
I _did_ go with the (currently implicit) assumption that no server will ever expose larger than off_t can represent, and therefore a signed 64-bit size is good enough.
I'll have to see that patch later, but for now I think making that assumption explicit would be nice (assert, abort, etc).
Flags has to be unsigned, but flags is representing something different than image size.
I take it you mean "nbd_block_descriptor_ext.status_flags", which is a "uint64_t".
Yes
If it's only used as a bitmap, that should be safe; first because some spec (dependent on metacontext) will introduce each bit one by one, and probably not start with bit#63; second because it makes no sense to sum bitmaps.
However, Wouter mentions up-thread that (dependent on metacontext) status_flags could actually stand for a length too:
'Some metadata namespaces treat the "flags" field as an unsigned integer, to specify offsets or counters. I that context, the flags field should indeed be extended to 64 bits.'
and then dealing with those particular metacontexts in OCaml could be a problem.
It's probably best to catch unrepresentable values when converting between OCaml's int64 and C's uint64_t (both ways) in the generator as centrally as possible.
Indeed; that's where I'm hoping Rich may chime in with ideas for what is best for the C <=> OCaml translation. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 5/31/23 18:04, Eric Blake wrote:
On Wed, May 31, 2023 at 01:29:30PM +0200, Laszlo Ersek wrote:
Putting aside alignment even, I don't understand why reducing "count" to uint16_t would be reasonable. With the current 32-bit-only block descriptor, we already need to write loops in libnbd clients, because we can't cover the entire remote image in one API call [*]. If I understood Eric right earlier, the 64-bit extensions were supposed to remedy that -- but as it stands, clients will still need loops ("chunking") around block status fetching; is that right?
While the larger extents reduce the need for looping, it does not entirely eliminate it. For example, just because the server can now tell you that an image is entirely data in just one reply does not mean that it will actually do so - qemu in particular limits block status of a qcow2 file to reporting just one cluster at a time for consistency reasons, where even if you use the maximum size of 2M clusters, you can never get more than (2M/16)*2M = 256G status reported in a single request.
I don't understand the calculation. I can imagine the following interpretation:
- QEMU never sends more than 128K block descriptors, and each descriptor covers one 2MB sized cluster --> 256 GB of the disk covered in one go.
But I don't understand where the (2M/16) division comes from, even though the quotient is 128K.
Ah, I need to provide more backstory on the qcow2 format. A qcow2 image has a fixed cluster size, chosen between between 512 and 2M bytes. A smaller cluster size has less wasted space for small images, but uses more overhead. Each cluster has to be stored in an L1 map, where pages of the map are also a cluster in length, with 16 bytes per map entry. So if you pick a cluster size of 512, you get 512/16 or 32 entries per L1 page; if you pick a cluster size of 2M, you get 2M/16 or 128k entries per L1 page. When reporting block status, qemu reads at most one L1 page to then say how each cluster referenced from that page is mapped.
https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/qcow2.txt#L4...
I can connect the constant "128K", and <https://github.com/NetworkBlockDevice/nbd/commit/926a51df>, to your paragraph [*] above, but not the division.
In this case, the qemu limit on reporting block status of at most one L1 map page at a time happens to have no relationship to the NBD constant of limiting block status reports to no more than 1M extents (8M bytes) in a single reply, nor the fact that qemu picked a cap of 1M bytes (128k extents) on its NBD reply regardless of whether the underlying image is qcow2 or some other format.
Thanks! [...] Laszlo
On Tue, May 30, 2023 at 05:48:25PM +0200, Laszlo Ersek wrote:
BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API.
I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t.
I think that's OK because disk sizes are already limited to 2^63 - 1 by the kernel (and for qemu even less than that). The OCaml bindings return a (signed) int64 for NBD.get_size. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On 6/7/23 12:00, Richard W.M. Jones wrote:
On Tue, May 30, 2023 at 05:48:25PM +0200, Laszlo Ersek wrote:
BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API.
I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t.
I think that's OK because disk sizes are already limited to 2^63 - 1 by the kernel (and for qemu even less than that). The OCaml bindings return a (signed) int64 for NBD.get_size.
Under patch#7 yesterday, I made a proposal for "armoring" at least one instance / direction of the uint64_t <-> int64 conversion. It raised an interesting problem: raising OCaml exceptions in such C functions that are *not* directly called by the OCaml runtime. Comments would be much appreciated in that subthread! (On a tangent: I've also noticed we use CAMLparam0() & friends in some of our functions that are *not* directly called by the OCaml runtime. They certainly run on the OCaml runtime's stack, but there's at least one intervening stack frame where the C-language function is provided by us. Now I know we must use CAMLparam0() in our *outermost* such function, but what about the further functions (inner C-language functions) that our outermost function calls in turn? I think the inner functions are at liberty not to use CAMLparam0() -- otherwise, our functions couldn't even call normal C library functions!) Thanks, Laszlo
On Thu, Jun 08, 2023 at 01:48:41PM +0200, Laszlo Ersek wrote:
On 6/7/23 12:00, Richard W.M. Jones wrote:
On Tue, May 30, 2023 at 05:48:25PM +0200, Laszlo Ersek wrote:
BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API.
I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t.
I think that's OK because disk sizes are already limited to 2^63 - 1 by the kernel (and for qemu even less than that). The OCaml bindings return a (signed) int64 for NBD.get_size.
Under patch#7 yesterday, I made a proposal for "armoring" at least one instance / direction of the uint64_t <-> int64 conversion. It raised an interesting problem: raising OCaml exceptions in such C functions that are *not* directly called by the OCaml runtime. Comments would be much appreciated in that subthread!
I can't seem to find that thread (but also gmail-- split the messages randomly over the 3 different mailing lists because Google don't understand how email works). Do you have a link? But the answer is you can raise an exception anywhere since it's really just a longjmp. The only issue is if you need stack-allocated variables to be freed, which for OCaml values is handled by the CAMLlocal* macros and for C variables you need to be careful about and deal with yourself.
(On a tangent: I've also noticed we use CAMLparam0() & friends in some of our functions that are *not* directly called by the OCaml runtime. They certainly run on the OCaml runtime's stack, but there's at least one intervening stack frame where the C-language function is provided by us. Now I know we must use CAMLparam0() in our *outermost* such function, but what about the further functions (inner C-language functions) that our outermost function calls in turn? I think the inner functions are at liberty not to use CAMLparam0() -- otherwise, our functions couldn't even call normal C library functions!)
These macros just set up a linked list of frames. You don't need to use them in every function, only ones which are using OCaml values. The macros are fairly easy to understand by reading them: https://github.com/ocaml/ocaml/blob/864f772e5338dcf6be2093d5cc3ed6f7fbce16b7... When the GC runs it walks up the linked list of the current thread to find roots. The only tricky thing about it is making sure that at any point where the GC could run, each slot contains a valid entry and not some intermediate or uninitialized value, since this is precise (not conservative) garbage collection. This mechanism is only used by C code. In OCaml code there's a bitmap generated for each function showing which stack slots contain values (versus ints, return addresses, other stuff). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On 6/8/23 14:20, Richard W.M. Jones wrote:
On Thu, Jun 08, 2023 at 01:48:41PM +0200, Laszlo Ersek wrote:
On 6/7/23 12:00, Richard W.M. Jones wrote:
On Tue, May 30, 2023 at 05:48:25PM +0200, Laszlo Ersek wrote:
BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API.
I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t.
I think that's OK because disk sizes are already limited to 2^63 - 1 by the kernel (and for qemu even less than that). The OCaml bindings return a (signed) int64 for NBD.get_size.
Under patch#7 yesterday, I made a proposal for "armoring" at least one instance / direction of the uint64_t <-> int64 conversion. It raised an interesting problem: raising OCaml exceptions in such C functions that are *not* directly called by the OCaml runtime. Comments would be much appreciated in that subthread!
I can't seem to find that thread (but also gmail-- split the messages randomly over the 3 different mailing lists because Google don't understand how email works). Do you have a link?
It starts here (link to patch#7): http://mid.mail-archive.com/20230525130108.757242-8-eblake@redhat.com (The primary libguestfs archive works too, but mailman splits the archives on month boundaries, and so the patch is still in the May bucket, but my reply is in the June one: https://listman.redhat.com/archives/libguestfs/2023-May/031613.html https://listman.redhat.com/archives/libguestfs/2023-June/031736.html ) The particular topic is at the very end of my message.
But the answer is you can raise an exception anywhere since it's really just a longjmp. The only issue is if you need stack-allocated variables to be freed, which for OCaml values is handled by the CAMLlocal* macros and for C variables you need to be careful about and deal with yourself.
Understood! Thanks! So for example, in extent64_wrapper_locked(), if we exited the called nbd_internal_ocaml_alloc_extent64_array() function with caml_failwith(), the caml_copy_string() and caml_copy_int64() allocations, stored earlier into "CAMLlocal"s "metacontextv" and "offsetv", would not be leaked?
(On a tangent: I've also noticed we use CAMLparam0() & friends in some of our functions that are *not* directly called by the OCaml runtime. They certainly run on the OCaml runtime's stack, but there's at least one intervening stack frame where the C-language function is provided by us. Now I know we must use CAMLparam0() in our *outermost* such function, but what about the further functions (inner C-language functions) that our outermost function calls in turn? I think the inner functions are at liberty not to use CAMLparam0() -- otherwise, our functions couldn't even call normal C library functions!)
These macros just set up a linked list of frames. You don't need to use them in every function, only ones which are using OCaml values.
Ah, understood.
The macros are fairly easy to understand by reading them:
https://github.com/ocaml/ocaml/blob/864f772e5338dcf6be2093d5cc3ed6f7fbce16b7...
When the GC runs it walks up the linked list of the current thread to find roots. The only tricky thing about it is making sure that at any point where the GC could run, each slot contains a valid entry and not some intermediate or uninitialized value, since this is precise (not conservative) garbage collection.
Right, <https://v2.ocaml.org/manual/intfc.html> too contains a related warning: Rule 5  After a structured block (a block with tag less than No_scan_tag) is allocated with the low-level functions, all fields of this block must be filled with well-formed values before the next allocation operation. [...] Thankfully it also says, "You can ignore those rules if you stick to the simplified allocation function caml_alloc".
This mechanism is only used by C code. In OCaml code there's a bitmap generated for each function showing which stack slots contain values (versus ints, return addresses, other stuff).
I'm slightly interested in learning the OCaml runtime details, but at the same time I feel like not knowing them (and relying only on the docs) might allow me to write more "portable" code... Laszlo
On Thu, Jun 08, 2023 at 02:38:40PM +0200, Laszlo Ersek wrote:
On 6/8/23 14:20, Richard W.M. Jones wrote:
On Thu, Jun 08, 2023 at 01:48:41PM +0200, Laszlo Ersek wrote:
On 6/7/23 12:00, Richard W.M. Jones wrote:
On Tue, May 30, 2023 at 05:48:25PM +0200, Laszlo Ersek wrote:
BTW I'm foreseeing a problem: if the extended block descriptor can provide an unsigned 64-bit length, we're going to have trouble exposing that in OCaml, because OCaml only has signed 64-bit integers. So that's going to reproduce the same issue, only for OCaml callers of the *new* API.
I can see Eric's series includes patches like "ocaml: Add example for 64-bit extents" -- I've not looked at those yet; for now I'm just wondering what tricks we might need in the bindings generator. The method seen in the "middle patch" above won't work; we don't have a native OCaml "i128" type for example that we could use as an escape hatch, for representing C's uint64_t.
I think that's OK because disk sizes are already limited to 2^63 - 1 by the kernel (and for qemu even less than that). The OCaml bindings return a (signed) int64 for NBD.get_size.
Under patch#7 yesterday, I made a proposal for "armoring" at least one instance / direction of the uint64_t <-> int64 conversion. It raised an interesting problem: raising OCaml exceptions in such C functions that are *not* directly called by the OCaml runtime. Comments would be much appreciated in that subthread!
I can't seem to find that thread (but also gmail-- split the messages randomly over the 3 different mailing lists because Google don't understand how email works). Do you have a link?
It starts here (link to patch#7):
http://mid.mail-archive.com/20230525130108.757242-8-eblake@redhat.com
OK I see it now (in the reply). I have answered there.
So for example, in extent64_wrapper_locked(), if we exited the called nbd_internal_ocaml_alloc_extent64_array() function with caml_failwith(), the caml_copy_string() and caml_copy_int64() allocations, stored earlier into "CAMLlocal"s "metacontextv" and "offsetv", would not be leaked?
Correct. When unwinding the stack, those frame structs created by CAML* macros are unlinked. Then the heap variables will have no references and will be freed in the course of garbage collection.
(On a tangent: I've also noticed we use CAMLparam0() & friends in some of our functions that are *not* directly called by the OCaml runtime. They certainly run on the OCaml runtime's stack, but there's at least one intervening stack frame where the C-language function is provided by us. Now I know we must use CAMLparam0() in our *outermost* such function, but what about the further functions (inner C-language functions) that our outermost function calls in turn? I think the inner functions are at liberty not to use CAMLparam0() -- otherwise, our functions couldn't even call normal C library functions!)
These macros just set up a linked list of frames. You don't need to use them in every function, only ones which are using OCaml values.
Ah, understood.
The macros are fairly easy to understand by reading them:
https://github.com/ocaml/ocaml/blob/864f772e5338dcf6be2093d5cc3ed6f7fbce16b7...
When the GC runs it walks up the linked list of the current thread to find roots. The only tricky thing about it is making sure that at any point where the GC could run, each slot contains a valid entry and not some intermediate or uninitialized value, since this is precise (not conservative) garbage collection.
Right, <https://v2.ocaml.org/manual/intfc.html> too contains a related warning:
Rule 5  After a structured block (a block with tag less than No_scan_tag) is allocated with the low-level functions, all fields of this block must be filled with well-formed values before the next allocation operation. [...]
Thankfully it also says, "You can ignore those rules if you stick to the simplified allocation function caml_alloc".
This mechanism is only used by C code. In OCaml code there's a bitmap generated for each function showing which stack slots contain values (versus ints, return addresses, other stuff).
I'm slightly interested in learning the OCaml runtime details, but at the same time I feel like not knowing them (and relying only on the docs) might allow me to write more "portable" code...
Yes the real rules are quite subtle, and following the documentation is a good idea. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
Support sending 64-bit requests if extended headers were negotiated. This includes setting NBD_CMD_FLAG_PAYLOAD_LEN any time we send an extended NBD_CMD_WRITE; this is such a fundamental part of the protocol that for now it is easier to silently ignore whatever value the user passes in for that bit in the flags parameter of nbd_pwrite regardless of the current settings in set_strict_mode, rather than trying to force the user to pass in the correct value to match whether extended mode is negotiated. However, when we later add APIs to give the user more control for interoperability testing, it may be worth adding a new set_strict_mode control knob to explicitly enable the client to intentionally violate the protocol (the testsuite added in this patch would then be updated to match). At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted). Support for using FLAG_PAYLOAD_LEN with NBD_CMD_BLOCK_STATUS is less fundamental, and deserves to be in its own patch. Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 10 ++- generator/API.ml | 20 +++-- generator/states-issue-command.c | 29 ++++--- generator/states-reply-structured.c | 2 +- lib/rw.c | 17 +++-- tests/Makefile.am | 4 + tests/pwrite-extended.c | 112 ++++++++++++++++++++++++++++ .gitignore | 1 + 8 files changed, 169 insertions(+), 26 deletions(-) create mode 100644 tests/pwrite-extended.c diff --git a/lib/internal.h b/lib/internal.h index c71980ef..8a5f93d4 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -110,6 +110,9 @@ struct nbd_handle { char *tls_username; /* Username, NULL = use current username */ char *tls_psk_file; /* PSK filename, NULL = no PSK */ + /* Extended headers. */ + bool extended_headers; /* If we negotiated NBD_OPT_EXTENDED_HEADERS */ + /* Desired metadata contexts. */ bool request_sr; bool request_meta; @@ -263,7 +266,10 @@ struct nbd_handle { /* Issuing a command must use a buffer separate from sbuf, for the * case when we interrupt a request to service a reply. */ - struct nbd_request request; + union { + struct nbd_request compact; + struct nbd_request_ext extended; + } req; bool in_write_payload; bool in_write_shutdown; @@ -364,7 +370,7 @@ struct command { uint16_t type; uint64_t cookie; uint64_t offset; - uint32_t count; + uint64_t count; void *data; /* Buffer for read/write */ struct command_cb cb; bool initialized; /* For read, true if getting a hole may skip memset */ diff --git a/generator/API.ml b/generator/API.ml index 5fcb0e1f..02c1260d 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -198,11 +198,12 @@ let cmd_flags = flag_prefix = "CMD_FLAG"; guard = Some "((h->strict & LIBNBD_STRICT_FLAGS) || flags > UINT16_MAX)"; flags = [ - "FUA", 1 lsl 0; - "NO_HOLE", 1 lsl 1; - "DF", 1 lsl 2; - "REQ_ONE", 1 lsl 3; - "FAST_ZERO", 1 lsl 4; + "FUA", 1 lsl 0; + "NO_HOLE", 1 lsl 1; + "DF", 1 lsl 2; + "REQ_ONE", 1 lsl 3; + "FAST_ZERO", 1 lsl 4; + "PAYLOAD_LEN", 1 lsl 5; ] } let handshake_flags = { @@ -2507,7 +2508,7 @@ "pread_structured", { "pwrite", { default_call with args = [ BytesIn ("buf", "count"); UInt64 "offset" ]; - optargs = [ OFlags ("flags", cmd_flags, Some ["FUA"]) ]; + optargs = [ OFlags ("flags", cmd_flags, Some ["FUA"; "PAYLOAD_LEN"]) ]; ret = RErr; permitted_states = [ Connected ]; shortdesc = "write to the NBD server"; @@ -2530,7 +2531,10 @@ "pwrite", { C<LIBNBD_CMD_FLAG_FUA> meaning that the server should not return until the data has been committed to permanent storage (if that is supported - some servers cannot do this, see -L<nbd_can_fua(3)>)." +L<nbd_can_fua(3)>). For convenience, libnbd ignores the presence +or absence of the flag C<LIBNBD_CMD_FLAG_PAYLOAD_LEN> in C<flags>, +while correctly using the flag over the wire according to whether +extended headers were negotiated." ^ strict_call_description; see_also = [Link "can_fua"; Link "is_read_only"; Link "aio_pwrite"; Link "get_block_size"; @@ -3220,7 +3224,7 @@ "aio_pwrite", { default_call with args = [ BytesPersistIn ("buf", "count"); UInt64 "offset" ]; optargs = [ OClosure completion_closure; - OFlags ("flags", cmd_flags, Some ["FUA"]) ]; + OFlags ("flags", cmd_flags, Some ["FUA"; "PAYLOAD_LEN"]) ]; ret = RCookie; permitted_states = [ Connected ]; shortdesc = "write to the NBD server"; diff --git a/generator/states-issue-command.c b/generator/states-issue-command.c index 111e131c..79136b61 100644 --- a/generator/states-issue-command.c +++ b/generator/states-issue-command.c @@ -41,15 +41,24 @@ ISSUE_COMMAND.START: return 0; } - h->request.magic = htobe32 (NBD_REQUEST_MAGIC); - h->request.flags = htobe16 (cmd->flags); - h->request.type = htobe16 (cmd->type); - h->request.handle = htobe64 (cmd->cookie); - h->request.offset = htobe64 (cmd->offset); - h->request.count = htobe32 (cmd->count); + /* These fields are coincident between req.compact and req.extended */ + h->req.compact.flags = htobe16 (cmd->flags); + h->req.compact.type = htobe16 (cmd->type); + h->req.compact.handle = htobe64 (cmd->cookie); + h->req.compact.offset = htobe64 (cmd->offset); + if (h->extended_headers) { + h->req.extended.magic = htobe32 (NBD_EXTENDED_REQUEST_MAGIC); + h->req.extended.count = htobe64 (cmd->count); + h->wlen = sizeof (h->req.extended); + } + else { + assert (cmd->count <= UINT32_MAX); + h->req.compact.magic = htobe32 (NBD_REQUEST_MAGIC); + h->req.compact.count = htobe32 (cmd->count); + h->wlen = sizeof (h->req.compact); + } h->chunks_sent++; - h->wbuf = &h->request; - h->wlen = sizeof (h->request); + h->wbuf = &h->req; if (cmd->type == NBD_CMD_WRITE || cmd->next) h->wflags = MSG_MORE; SET_NEXT_STATE (%SEND_REQUEST); @@ -74,7 +83,7 @@ ISSUE_COMMAND.PREPARE_WRITE_PAYLOAD: assert (h->cmds_to_issue != NULL); cmd = h->cmds_to_issue; - assert (cmd->cookie == be64toh (h->request.handle)); + assert (cmd->cookie == be64toh (h->req.compact.handle)); if (cmd->type == NBD_CMD_WRITE) { h->wbuf = cmd->data; h->wlen = cmd->count; @@ -120,7 +129,7 @@ ISSUE_COMMAND.FINISH: assert (!h->wlen); assert (h->cmds_to_issue != NULL); cmd = h->cmds_to_issue; - assert (cmd->cookie == be64toh (h->request.handle)); + assert (cmd->cookie == be64toh (h->req.compact.handle)); h->cmds_to_issue = cmd->next; if (h->cmds_to_issue_tail == cmd) h->cmds_to_issue_tail = NULL; diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 6f96945a..df509216 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -34,7 +34,7 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length, offset + length > cmd->offset + cmd->count) { set_error (0, "range of structured reply is out of bounds, " "offset=%" PRIu64 ", cmd->offset=%" PRIu64 ", " - "length=%" PRIu32 ", cmd->count=%" PRIu32 ": " + "length=%" PRIu32 ", cmd->count=%" PRIu64 ": " "this is likely to be a bug in the NBD server", offset, cmd->offset, length, cmd->count); return false; diff --git a/lib/rw.c b/lib/rw.c index 3dc3499e..8b2bd4cc 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -223,13 +223,11 @@ nbd_internal_command_common (struct nbd_handle *h, } break; - /* Other commands are currently limited by the 32 bit field in the - * command structure on the wire, but in future we hope to support - * 64 bit values here with a change to the NBD protocol which is - * being discussed upstream. + /* Other commands are limited by the 32 bit field in the command + * structure on the wire, unless extended headers were negotiated. */ default: - if (count > UINT32_MAX) { + if (!h->extended_headers && count > UINT32_MAX) { set_error (ERANGE, "request too large: maximum request size is %" PRIu32, UINT32_MAX); goto err; @@ -358,6 +356,15 @@ nbd_unlocked_aio_pwrite (struct nbd_handle *h, const void *buf, return -1; } } + /* It is more convenient to manage PAYLOAD_LEN by what was negotiated + * than to require the user to have to set it correctly. + * TODO: Add new h->strict bit to allow intentional protocol violation + * for interoperability testing. + */ + if (h->extended_headers) + flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN; + else + flags &= ~LIBNBD_CMD_FLAG_PAYLOAD_LEN; SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_WRITE, offset, count, diff --git a/tests/Makefile.am b/tests/Makefile.am index 3a93251e..8b839bf5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -232,6 +232,7 @@ check_PROGRAMS += \ closure-lifetimes \ pread-initialize \ socket-activation-name \ + pwrite-extended \ $(NULL) TESTS += \ @@ -650,6 +651,9 @@ socket_activation_name_SOURCES = \ requires.h socket_activation_name_LDADD = $(top_builddir)/lib/libnbd.la +pwrite_extended_SOURCES = pwrite-extended.c +pwrite_extended_LDADD = $(top_builddir)/lib/libnbd.la + #---------------------------------------------------------------------- # Testing TLS support. diff --git a/tests/pwrite-extended.c b/tests/pwrite-extended.c new file mode 100644 index 00000000..f0b5a3f3 --- /dev/null +++ b/tests/pwrite-extended.c @@ -0,0 +1,112 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Check behavior of pwrite with PAYLOAD_LEN flag for extended headers. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <unistd.h> +#include <sys/stat.h> + +#include <libnbd.h> + +static char *progname; +static char buf[512]; + +static void +check (int experr, const char *prefix) +{ + const char *msg = nbd_get_error (); + int errnum = nbd_get_errno (); + + fprintf (stderr, "error: \"%s\"\n", msg); + fprintf (stderr, "errno: %d (%s)\n", errnum, strerror (errnum)); + if (strncmp (msg, prefix, strlen (prefix)) != 0) { + fprintf (stderr, "%s: test failed: missing context prefix: %s\n", + progname, msg); + exit (EXIT_FAILURE); + } + if (errnum != experr) { + fprintf (stderr, "%s: test failed: " + "expected errno = %d (%s), but got %d\n", + progname, experr, strerror (experr), errnum); + exit (EXIT_FAILURE); + } +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + const char *cmd[] = { + "nbdkit", "-s", "-v", "--exit-with-parent", "memory", "1048576", NULL + }; + uint32_t strict; + + progname = argv[0]; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Connect to the server. */ + if (nbd_connect_command (nbd, (char **)cmd) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* FIXME: future API addition to test if server negotiated extended mode. + * Until then, strict flags must ignore the PAYLOAD_LEN flag for pwrite, + * even though it is rejected for other commands. + */ + strict = nbd_get_strict_mode (nbd); + if (!(strict & LIBNBD_STRICT_FLAGS)) { + fprintf (stderr, "%s: test failed: " + "nbd_get_strict_mode did not have expected flag set\n", + argv[0]); + exit (EXIT_FAILURE); + } + if (nbd_aio_pread (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) != -1) { + fprintf (stderr, "%s: test failed: " + "nbd_aio_pread did not fail with unexpected flag\n", + argv[0]); + exit (EXIT_FAILURE); + } + check (EINVAL, "nbd_aio_pread: "); + + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +} diff --git a/.gitignore b/.gitignore index fe7feffa..bc7c2c37 100644 --- a/.gitignore +++ b/.gitignore @@ -250,6 +250,7 @@ Makefile.in /tests/pki/ /tests/pread-initialize /tests/private-data +/tests/pwrite-extended /tests/read-only-flag /tests/read-write-flag /tests/server-death -- 2.40.1
On 5/25/23 15:00, Eric Blake wrote:
Support sending 64-bit requests if extended headers were negotiated. This includes setting NBD_CMD_FLAG_PAYLOAD_LEN any time we send an extended NBD_CMD_WRITE; this is such a fundamental part of the protocol that for now it is easier to silently ignore whatever value the user passes in for that bit in the flags parameter of nbd_pwrite regardless of the current settings in set_strict_mode, rather than trying to force the user to pass in the correct value to match whether extended mode is negotiated. However, when we later add APIs to give the user more control for interoperability testing, it may be worth adding a new set_strict_mode control knob to explicitly enable the client to intentionally violate the protocol (the testsuite added in this patch would then be updated to match).
At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted).
Support for using FLAG_PAYLOAD_LEN with NBD_CMD_BLOCK_STATUS is less fundamental, and deserves to be in its own patch.
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 10 ++- generator/API.ml | 20 +++-- generator/states-issue-command.c | 29 ++++--- generator/states-reply-structured.c | 2 +- lib/rw.c | 17 +++-- tests/Makefile.am | 4 + tests/pwrite-extended.c | 112 ++++++++++++++++++++++++++++ .gitignore | 1 + 8 files changed, 169 insertions(+), 26 deletions(-) create mode 100644 tests/pwrite-extended.c
diff --git a/lib/internal.h b/lib/internal.h index c71980ef..8a5f93d4 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -110,6 +110,9 @@ struct nbd_handle { char *tls_username; /* Username, NULL = use current username */ char *tls_psk_file; /* PSK filename, NULL = no PSK */
+ /* Extended headers. */ + bool extended_headers; /* If we negotiated NBD_OPT_EXTENDED_HEADERS */ + /* Desired metadata contexts. */ bool request_sr; bool request_meta; @@ -263,7 +266,10 @@ struct nbd_handle { /* Issuing a command must use a buffer separate from sbuf, for the * case when we interrupt a request to service a reply. */ - struct nbd_request request; + union { + struct nbd_request compact; + struct nbd_request_ext extended; + } req; bool in_write_payload; bool in_write_shutdown;
@@ -364,7 +370,7 @@ struct command { uint16_t type; uint64_t cookie; uint64_t offset; - uint32_t count; + uint64_t count; void *data; /* Buffer for read/write */ struct command_cb cb; bool initialized; /* For read, true if getting a hole may skip memset */
(1) Are there places in the code where we currently assign this "count" field back to a uint32_t object, and assume truncation impossible?
diff --git a/generator/API.ml b/generator/API.ml index 5fcb0e1f..02c1260d 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -198,11 +198,12 @@ let cmd_flags = flag_prefix = "CMD_FLAG"; guard = Some "((h->strict & LIBNBD_STRICT_FLAGS) || flags > UINT16_MAX)"; flags = [ - "FUA", 1 lsl 0; - "NO_HOLE", 1 lsl 1; - "DF", 1 lsl 2; - "REQ_ONE", 1 lsl 3; - "FAST_ZERO", 1 lsl 4; + "FUA", 1 lsl 0; + "NO_HOLE", 1 lsl 1; + "DF", 1 lsl 2; + "REQ_ONE", 1 lsl 3; + "FAST_ZERO", 1 lsl 4; + "PAYLOAD_LEN", 1 lsl 5; ] } let handshake_flags = { @@ -2507,7 +2508,7 @@ "pread_structured", { "pwrite", { default_call with args = [ BytesIn ("buf", "count"); UInt64 "offset" ]; - optargs = [ OFlags ("flags", cmd_flags, Some ["FUA"]) ]; + optargs = [ OFlags ("flags", cmd_flags, Some ["FUA"; "PAYLOAD_LEN"]) ]; ret = RErr; permitted_states = [ Connected ]; shortdesc = "write to the NBD server"; @@ -2530,7 +2531,10 @@ "pwrite", { C<LIBNBD_CMD_FLAG_FUA> meaning that the server should not return until the data has been committed to permanent storage (if that is supported - some servers cannot do this, see -L<nbd_can_fua(3)>)." +L<nbd_can_fua(3)>). For convenience, libnbd ignores the presence +or absence of the flag C<LIBNBD_CMD_FLAG_PAYLOAD_LEN> in C<flags>, +while correctly using the flag over the wire according to whether +extended headers were negotiated." ^ strict_call_description; see_also = [Link "can_fua"; Link "is_read_only"; Link "aio_pwrite"; Link "get_block_size"; @@ -3220,7 +3224,7 @@ "aio_pwrite", { default_call with args = [ BytesPersistIn ("buf", "count"); UInt64 "offset" ]; optargs = [ OClosure completion_closure; - OFlags ("flags", cmd_flags, Some ["FUA"]) ]; + OFlags ("flags", cmd_flags, Some ["FUA"; "PAYLOAD_LEN"]) ]; ret = RCookie; permitted_states = [ Connected ]; shortdesc = "write to the NBD server"; diff --git a/generator/states-issue-command.c b/generator/states-issue-command.c index 111e131c..79136b61 100644 --- a/generator/states-issue-command.c +++ b/generator/states-issue-command.c @@ -41,15 +41,24 @@ ISSUE_COMMAND.START: return 0; }
- h->request.magic = htobe32 (NBD_REQUEST_MAGIC); - h->request.flags = htobe16 (cmd->flags); - h->request.type = htobe16 (cmd->type); - h->request.handle = htobe64 (cmd->cookie); - h->request.offset = htobe64 (cmd->offset); - h->request.count = htobe32 (cmd->count); + /* These fields are coincident between req.compact and req.extended */ + h->req.compact.flags = htobe16 (cmd->flags); + h->req.compact.type = htobe16 (cmd->type); + h->req.compact.handle = htobe64 (cmd->cookie); + h->req.compact.offset = htobe64 (cmd->offset);
What's more, this is a "by the book" common initial sequence! :)
+ if (h->extended_headers) { + h->req.extended.magic = htobe32 (NBD_EXTENDED_REQUEST_MAGIC); + h->req.extended.count = htobe64 (cmd->count); + h->wlen = sizeof (h->req.extended); + } + else { + assert (cmd->count <= UINT32_MAX); + h->req.compact.magic = htobe32 (NBD_REQUEST_MAGIC); + h->req.compact.count = htobe32 (cmd->count); + h->wlen = sizeof (h->req.compact); + } h->chunks_sent++; - h->wbuf = &h->request; - h->wlen = sizeof (h->request); + h->wbuf = &h->req; if (cmd->type == NBD_CMD_WRITE || cmd->next) h->wflags = MSG_MORE; SET_NEXT_STATE (%SEND_REQUEST); @@ -74,7 +83,7 @@ ISSUE_COMMAND.PREPARE_WRITE_PAYLOAD:
assert (h->cmds_to_issue != NULL); cmd = h->cmds_to_issue; - assert (cmd->cookie == be64toh (h->request.handle)); + assert (cmd->cookie == be64toh (h->req.compact.handle)); if (cmd->type == NBD_CMD_WRITE) { h->wbuf = cmd->data; h->wlen = cmd->count; @@ -120,7 +129,7 @@ ISSUE_COMMAND.FINISH: assert (!h->wlen); assert (h->cmds_to_issue != NULL); cmd = h->cmds_to_issue; - assert (cmd->cookie == be64toh (h->request.handle)); + assert (cmd->cookie == be64toh (h->req.compact.handle)); h->cmds_to_issue = cmd->next; if (h->cmds_to_issue_tail == cmd) h->cmds_to_issue_tail = NULL; diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 6f96945a..df509216 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -34,7 +34,7 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length, offset + length > cmd->offset + cmd->count) { set_error (0, "range of structured reply is out of bounds, " "offset=%" PRIu64 ", cmd->offset=%" PRIu64 ", " - "length=%" PRIu32 ", cmd->count=%" PRIu32 ": " + "length=%" PRIu32 ", cmd->count=%" PRIu64 ": " "this is likely to be a bug in the NBD server", offset, cmd->offset, length, cmd->count); return false; diff --git a/lib/rw.c b/lib/rw.c index 3dc3499e..8b2bd4cc 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -223,13 +223,11 @@ nbd_internal_command_common (struct nbd_handle *h, } break;
- /* Other commands are currently limited by the 32 bit field in the - * command structure on the wire, but in future we hope to support - * 64 bit values here with a change to the NBD protocol which is - * being discussed upstream. + /* Other commands are limited by the 32 bit field in the command + * structure on the wire, unless extended headers were negotiated. */ default: - if (count > UINT32_MAX) { + if (!h->extended_headers && count > UINT32_MAX) { set_error (ERANGE, "request too large: maximum request size is %" PRIu32, UINT32_MAX); goto err; @@ -358,6 +356,15 @@ nbd_unlocked_aio_pwrite (struct nbd_handle *h, const void *buf, return -1; } } + /* It is more convenient to manage PAYLOAD_LEN by what was negotiated + * than to require the user to have to set it correctly. + * TODO: Add new h->strict bit to allow intentional protocol violation + * for interoperability testing. + */ + if (h->extended_headers) + flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN; + else + flags &= ~LIBNBD_CMD_FLAG_PAYLOAD_LEN;
Nice -- I wanted to ask for: flags &= ~(uint32_t)LIBNBD_CMD_FLAG_PAYLOAD_LEN; due to LIBNBD_CMD_FLAG_PAYLOAD_LEN having type "int". However: in patch#3, what has type "int" is: +#define NBD_CMD_FLAG_PAYLOAD_LEN (1<<5) and here we have LIBNBD_CMD_FLAG_PAYLOAD_LEN instead -- and the latter has type unsigned int already, from your recent commit 69eecae2c03a ("api: Generate flag values as unsigned", 2022-11-11). And I think we're fine assuming that uint32_t is unsigned int.
SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_WRITE, offset, count, diff --git a/tests/Makefile.am b/tests/Makefile.am index 3a93251e..8b839bf5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -232,6 +232,7 @@ check_PROGRAMS += \ closure-lifetimes \ pread-initialize \ socket-activation-name \ + pwrite-extended \ $(NULL)
TESTS += \
(2) Incorrect indentation: two spaces rather than one tab.
@@ -650,6 +651,9 @@ socket_activation_name_SOURCES = \ requires.h socket_activation_name_LDADD = $(top_builddir)/lib/libnbd.la
+pwrite_extended_SOURCES = pwrite-extended.c +pwrite_extended_LDADD = $(top_builddir)/lib/libnbd.la + #---------------------------------------------------------------------- # Testing TLS support.
diff --git a/tests/pwrite-extended.c b/tests/pwrite-extended.c new file mode 100644 index 00000000..f0b5a3f3 --- /dev/null +++ b/tests/pwrite-extended.c @@ -0,0 +1,112 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Check behavior of pwrite with PAYLOAD_LEN flag for extended headers. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <unistd.h> +#include <sys/stat.h> + +#include <libnbd.h> + +static char *progname; +static char buf[512]; + +static void +check (int experr, const char *prefix) +{ + const char *msg = nbd_get_error (); + int errnum = nbd_get_errno (); + + fprintf (stderr, "error: \"%s\"\n", msg); + fprintf (stderr, "errno: %d (%s)\n", errnum, strerror (errnum)); + if (strncmp (msg, prefix, strlen (prefix)) != 0) { + fprintf (stderr, "%s: test failed: missing context prefix: %s\n", + progname, msg); + exit (EXIT_FAILURE); + } + if (errnum != experr) { + fprintf (stderr, "%s: test failed: " + "expected errno = %d (%s), but got %d\n", + progname, experr, strerror (experr), errnum); + exit (EXIT_FAILURE); + } +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + const char *cmd[] = { + "nbdkit", "-s", "-v", "--exit-with-parent", "memory", "1048576", NULL + }; + uint32_t strict; + + progname = argv[0]; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ());
(3) Minor inconsistency with check(): we're not printing "progname" here.
+ exit (EXIT_FAILURE); + } + + /* Connect to the server. */ + if (nbd_connect_command (nbd, (char **)cmd) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + }
(4) Another kind of inconsistency: we could use "progname" here, in place of argv[0]. (This applies to all other fprintf()s below.)
+ + /* FIXME: future API addition to test if server negotiated extended mode. + * Until then, strict flags must ignore the PAYLOAD_LEN flag for pwrite, + * even though it is rejected for other commands. + */ + strict = nbd_get_strict_mode (nbd); + if (!(strict & LIBNBD_STRICT_FLAGS)) { + fprintf (stderr, "%s: test failed: " + "nbd_get_strict_mode did not have expected flag set\n", + argv[0]); + exit (EXIT_FAILURE); + }
Not sure if I understand this check. Per <https://libguestfs.org/nbd_set_strict_mode.3.html>, I take it that LIBNBD_STRICT_FLAGS should be "on" by default. Are you enforcing that? And if so: is it your intent that, *even with* LIBNBD_STRICT_FLAGS, an invalid PAYLOAD_LEN is not rejected (as seen by the libnbd client app), but fixed up silently?
+ if (nbd_aio_pread (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) != -1) { + fprintf (stderr, "%s: test failed: " + "nbd_aio_pread did not fail with unexpected flag\n", + argv[0]); + exit (EXIT_FAILURE); + } + check (EINVAL, "nbd_aio_pread: ");
Ah, got it now. We do want APIs other than pwrite to fail.
+ + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + }
You could contract these into: if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1 || nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) { fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); exit (EXIT_FAILURE); }
+ + nbd_close (nbd); + exit (EXIT_SUCCESS); +}
In general, I think it's good practice to reach nbd_close() whenever nbd_create() succeeds (that is, on error paths as well, after nbd_create() succeeds). For example, if we connected to the server with systemd socket activation in this test, and (say) one of the pwrites failed, then we'd leak the unix domain socket in the filesystem (from the bind() in "generator/states-connect-socket-activation.c"). "sact_sockpath" is unlinked in nbd_close(). (As written, this test should not be affected, because, according to unix(7), unix domain sockets created with socketpair(2) are unnamed.)
diff --git a/.gitignore b/.gitignore index fe7feffa..bc7c2c37 100644 --- a/.gitignore +++ b/.gitignore @@ -250,6 +250,7 @@ Makefile.in /tests/pki/ /tests/pread-initialize /tests/private-data +/tests/pwrite-extended /tests/read-only-flag /tests/read-write-flag /tests/server-death
Thanks Laszlo
On Tue, May 30, 2023 at 04:06:32PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
Support sending 64-bit requests if extended headers were negotiated. This includes setting NBD_CMD_FLAG_PAYLOAD_LEN any time we send an extended NBD_CMD_WRITE; this is such a fundamental part of the protocol that for now it is easier to silently ignore whatever value the user passes in for that bit in the flags parameter of nbd_pwrite regardless of the current settings in set_strict_mode, rather than trying to force the user to pass in the correct value to match whether extended mode is negotiated. However, when we later add APIs to give the user more control for interoperability testing, it may be worth adding a new set_strict_mode control knob to explicitly enable the client to intentionally violate the protocol (the testsuite added in this patch would then be updated to match).
At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted).
Support for using FLAG_PAYLOAD_LEN with NBD_CMD_BLOCK_STATUS is less fundamental, and deserves to be in its own patch.
Signed-off-by: Eric Blake <eblake@redhat.com> ---
@@ -364,7 +370,7 @@ struct command { uint16_t type; uint64_t cookie; uint64_t offset; - uint32_t count; + uint64_t count; void *data; /* Buffer for read/write */ struct command_cb cb; bool initialized; /* For read, true if getting a hole may skip memset */
(1) Are there places in the code where we currently assign this "count" field back to a uint32_t object, and assume truncation impossible?
Grepping for '->count' in lib/ and generator/ shows we need to check at least: generator/states-reply-simple.c: h->rlen = cmd->count; generator/states-reply-simple.c: cmd->data_seen += cmd->count; which are adjustments to size_t and uint32_t variables respectively, in response to a server's reply to an NBD_CMD_READ command. But since we never send a server a read request larger than 64M, truncation and overflow are not possible in those lines of code (at most one simple reply is possible, and code in states-reply-structured.c ensures that cmd->data_seen is a saturating variable that never exceeds 2*MAX_REQUEST_SIZE). There is also pre-series: generator/states-issue-command.c: h->request.count = htobe32 (cmd->count); which is specifically updated in this patch to cover extended headers.
+++ b/generator/states-issue-command.c @@ -41,15 +41,24 @@ ISSUE_COMMAND.START: return 0; }
- h->request.magic = htobe32 (NBD_REQUEST_MAGIC); - h->request.flags = htobe16 (cmd->flags); - h->request.type = htobe16 (cmd->type); - h->request.handle = htobe64 (cmd->cookie); - h->request.offset = htobe64 (cmd->offset); - h->request.count = htobe32 (cmd->count); + /* These fields are coincident between req.compact and req.extended */ + h->req.compact.flags = htobe16 (cmd->flags); + h->req.compact.type = htobe16 (cmd->type); + h->req.compact.handle = htobe64 (cmd->cookie); + h->req.compact.offset = htobe64 (cmd->offset);
What's more, this is a "by the book" common initial sequence! :)
+ if (h->extended_headers) { + h->req.extended.magic = htobe32 (NBD_EXTENDED_REQUEST_MAGIC); + h->req.extended.count = htobe64 (cmd->count); + h->wlen = sizeof (h->req.extended); + } + else { + assert (cmd->count <= UINT32_MAX); + h->req.compact.magic = htobe32 (NBD_REQUEST_MAGIC); + h->req.compact.count = htobe32 (cmd->count); + h->wlen = sizeof (h->req.compact); + }
Indeed, and shows why my efforts to get a sane layout early in the series matter, even if it will cause me a bit more rebase churn here based on my response to your comments earlier in the series.
@@ -358,6 +356,15 @@ nbd_unlocked_aio_pwrite (struct nbd_handle *h, const void *buf, return -1; } } + /* It is more convenient to manage PAYLOAD_LEN by what was negotiated + * than to require the user to have to set it correctly. + * TODO: Add new h->strict bit to allow intentional protocol violation + * for interoperability testing. + */ + if (h->extended_headers) + flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN; + else + flags &= ~LIBNBD_CMD_FLAG_PAYLOAD_LEN;
Nice -- I wanted to ask for:
flags &= ~(uint32_t)LIBNBD_CMD_FLAG_PAYLOAD_LEN;
due to LIBNBD_CMD_FLAG_PAYLOAD_LEN having type "int".
However: in patch#3, what has type "int" is:
+#define NBD_CMD_FLAG_PAYLOAD_LEN (1<<5)
and here we have LIBNBD_CMD_FLAG_PAYLOAD_LEN instead -- and the latter has type unsigned int already, from your recent commit 69eecae2c03a ("api: Generate flag values as unsigned", 2022-11-11).
Still, worth a (separate) cleanup patch to nbd-protocol.h to prefer unsigned constants for the flag values where they are not generated.
And I think we're fine assuming that uint32_t is unsigned int.
Not true of all generic C platforms, but certainly true for the POSIX-like platforms we target (anyone that defines uint32_t as 'unsigned long' on a platform with 32-bit longs is unusual, but even then we should still be okay).
SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_WRITE, offset, count, diff --git a/tests/Makefile.am b/tests/Makefile.am index 3a93251e..8b839bf5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -232,6 +232,7 @@ check_PROGRAMS += \ closure-lifetimes \ pread-initialize \ socket-activation-name \ + pwrite-extended \ $(NULL)
TESTS += \
(2) Incorrect indentation: two spaces rather than one tab.
Arrgh. ./.editorconfig is supposed to do this correctly, but obviously its interaction with emacs is a bit botched when it comes to Makefile syntax. Will clean up.
+++ b/tests/pwrite-extended.c
+static void +check (int experr, const char *prefix) +{ + const char *msg = nbd_get_error (); + int errnum = nbd_get_errno (); + + fprintf (stderr, "error: \"%s\"\n", msg); + fprintf (stderr, "errno: %d (%s)\n", errnum, strerror (errnum)); + if (strncmp (msg, prefix, strlen (prefix)) != 0) { + fprintf (stderr, "%s: test failed: missing context prefix: %s\n", + progname, msg); + exit (EXIT_FAILURE); + } + if (errnum != experr) { + fprintf (stderr, "%s: test failed: " + "expected errno = %d (%s), but got %d\n", + progname, experr, strerror (experr), errnum); + exit (EXIT_FAILURE); + } +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + const char *cmd[] = { + "nbdkit", "-s", "-v", "--exit-with-parent", "memory", "1048576", NULL + }; + uint32_t strict; + + progname = argv[0]; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ());
(3) Minor inconsistency with check(): we're not printing "progname" here.
+ exit (EXIT_FAILURE); + } + + /* Connect to the server. */ + if (nbd_connect_command (nbd, (char **)cmd) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + }
(4) Another kind of inconsistency: we could use "progname" here, in place of argv[0].
(This applies to all other fprintf()s below.)
Probably copy-and-paste from other similar tests, but I don't mind cleaning those up.
+ + /* FIXME: future API addition to test if server negotiated extended mode. + * Until then, strict flags must ignore the PAYLOAD_LEN flag for pwrite, + * even though it is rejected for other commands. + */ + strict = nbd_get_strict_mode (nbd); + if (!(strict & LIBNBD_STRICT_FLAGS)) { + fprintf (stderr, "%s: test failed: " + "nbd_get_strict_mode did not have expected flag set\n", + argv[0]); + exit (EXIT_FAILURE); + }
Not sure if I understand this check. Per <https://libguestfs.org/nbd_set_strict_mode.3.html>, I take it that LIBNBD_STRICT_FLAGS should be "on" by default. Are you enforcing that? And if so: is it your intent that, *even with* LIBNBD_STRICT_FLAGS, an invalid PAYLOAD_LEN is not rejected (as seen by the libnbd client app), but fixed up silently?
Rather, until we can tell if the server negotiated extended mode, we are ASSUMING that the server did NOT negotiate it, and therefore we are in violation of the spec if we send the flag over the wire anyways. We can flag all other API where it is inappropriate to ever use...
+ if (nbd_aio_pread (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) != -1) { + fprintf (stderr, "%s: test failed: " + "nbd_aio_pread did not fail with unexpected flag\n", + argv[0]); + exit (EXIT_FAILURE); + } + check (EINVAL, "nbd_aio_pread: ");
Ah, got it now. We do want APIs other than pwrite to fail.
...but because we don't want to require clients to correctly decide when to pass or omit the flag to their API calls (by us masking out the user's choice and then hardcoding our actual wire behavior based on negotiated mode), passing the flag to pread works even when it would be technically wrong over the wire. The FIXME does get modified again later in the series, when I do add in support for detecting when the server supports extended headers.
+ + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + }
You could contract these into:
if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1 || nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) { fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); exit (EXIT_FAILURE); }
Sure.
+ + nbd_close (nbd); + exit (EXIT_SUCCESS); +}
In general, I think it's good practice to reach nbd_close() whenever nbd_create() succeeds (that is, on error paths as well, after nbd_create() succeeds). For example, if we connected to the server with systemd socket activation in this test, and (say) one of the pwrites failed, then we'd leak the unix domain socket in the filesystem (from the bind() in "generator/states-connect-socket-activation.c"). "sact_sockpath" is unlinked in nbd_close().
(As written, this test should not be affected, because, according to unix(7), unix domain sockets created with socketpair(2) are unnamed.)
Pre-existing in other tests, but a good observation for a followup patch. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 5/30/23 20:18, Eric Blake wrote:
On Tue, May 30, 2023 at 04:06:32PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
Support sending 64-bit requests if extended headers were negotiated. This includes setting NBD_CMD_FLAG_PAYLOAD_LEN any time we send an extended NBD_CMD_WRITE; this is such a fundamental part of the protocol that for now it is easier to silently ignore whatever value the user passes in for that bit in the flags parameter of nbd_pwrite regardless of the current settings in set_strict_mode, rather than trying to force the user to pass in the correct value to match whether extended mode is negotiated. However, when we later add APIs to give the user more control for interoperability testing, it may be worth adding a new set_strict_mode control knob to explicitly enable the client to intentionally violate the protocol (the testsuite added in this patch would then be updated to match).
At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted).
Support for using FLAG_PAYLOAD_LEN with NBD_CMD_BLOCK_STATUS is less fundamental, and deserves to be in its own patch.
Signed-off-by: Eric Blake <eblake@redhat.com> ---
@@ -364,7 +370,7 @@ struct command { uint16_t type; uint64_t cookie; uint64_t offset; - uint32_t count; + uint64_t count; void *data; /* Buffer for read/write */ struct command_cb cb; bool initialized; /* For read, true if getting a hole may skip memset */
(1) Are there places in the code where we currently assign this "count" field back to a uint32_t object, and assume truncation impossible?
Grepping for '->count' in lib/ and generator/ shows we need to check at least:
generator/states-reply-simple.c: h->rlen = cmd->count; generator/states-reply-simple.c: cmd->data_seen += cmd->count;
which are adjustments to size_t and uint32_t variables respectively, in response to a server's reply to an NBD_CMD_READ command. But since we never send a server a read request larger than 64M, truncation and overflow are not possible in those lines of code (at most one simple reply is possible, and code in states-reply-structured.c ensures that cmd->data_seen is a saturating variable that never exceeds 2*MAX_REQUEST_SIZE).
There is also pre-series:
generator/states-issue-command.c: h->request.count = htobe32 (cmd->count);
which is specifically updated in this patch to cover extended headers.
+++ b/generator/states-issue-command.c @@ -41,15 +41,24 @@ ISSUE_COMMAND.START: return 0; }
- h->request.magic = htobe32 (NBD_REQUEST_MAGIC); - h->request.flags = htobe16 (cmd->flags); - h->request.type = htobe16 (cmd->type); - h->request.handle = htobe64 (cmd->cookie); - h->request.offset = htobe64 (cmd->offset); - h->request.count = htobe32 (cmd->count); + /* These fields are coincident between req.compact and req.extended */ + h->req.compact.flags = htobe16 (cmd->flags); + h->req.compact.type = htobe16 (cmd->type); + h->req.compact.handle = htobe64 (cmd->cookie); + h->req.compact.offset = htobe64 (cmd->offset);
What's more, this is a "by the book" common initial sequence! :)
+ if (h->extended_headers) { + h->req.extended.magic = htobe32 (NBD_EXTENDED_REQUEST_MAGIC); + h->req.extended.count = htobe64 (cmd->count); + h->wlen = sizeof (h->req.extended); + } + else { + assert (cmd->count <= UINT32_MAX); + h->req.compact.magic = htobe32 (NBD_REQUEST_MAGIC); + h->req.compact.count = htobe32 (cmd->count); + h->wlen = sizeof (h->req.compact); + }
Indeed, and shows why my efforts to get a sane layout early in the series matter, even if it will cause me a bit more rebase churn here based on my response to your comments earlier in the series.
@@ -358,6 +356,15 @@ nbd_unlocked_aio_pwrite (struct nbd_handle *h, const void *buf, return -1; } } + /* It is more convenient to manage PAYLOAD_LEN by what was negotiated + * than to require the user to have to set it correctly. + * TODO: Add new h->strict bit to allow intentional protocol violation + * for interoperability testing. + */ + if (h->extended_headers) + flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN; + else + flags &= ~LIBNBD_CMD_FLAG_PAYLOAD_LEN;
Nice -- I wanted to ask for:
flags &= ~(uint32_t)LIBNBD_CMD_FLAG_PAYLOAD_LEN;
due to LIBNBD_CMD_FLAG_PAYLOAD_LEN having type "int".
However: in patch#3, what has type "int" is:
+#define NBD_CMD_FLAG_PAYLOAD_LEN (1<<5)
and here we have LIBNBD_CMD_FLAG_PAYLOAD_LEN instead -- and the latter has type unsigned int already, from your recent commit 69eecae2c03a ("api: Generate flag values as unsigned", 2022-11-11).
Still, worth a (separate) cleanup patch to nbd-protocol.h to prefer unsigned constants for the flag values where they are not generated.
And I think we're fine assuming that uint32_t is unsigned int.
Not true of all generic C platforms, but certainly true for the POSIX-like platforms we target (anyone that defines uint32_t as 'unsigned long' on a platform with 32-bit longs is unusual, but even then we should still be okay).
SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_WRITE, offset, count, diff --git a/tests/Makefile.am b/tests/Makefile.am index 3a93251e..8b839bf5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -232,6 +232,7 @@ check_PROGRAMS += \ closure-lifetimes \ pread-initialize \ socket-activation-name \ + pwrite-extended \ $(NULL)
TESTS += \
(2) Incorrect indentation: two spaces rather than one tab.
Arrgh. ./.editorconfig is supposed to do this correctly, but obviously its interaction with emacs is a bit botched when it comes to Makefile syntax. Will clean up.
+++ b/tests/pwrite-extended.c
+static void +check (int experr, const char *prefix) +{ + const char *msg = nbd_get_error (); + int errnum = nbd_get_errno (); + + fprintf (stderr, "error: \"%s\"\n", msg); + fprintf (stderr, "errno: %d (%s)\n", errnum, strerror (errnum)); + if (strncmp (msg, prefix, strlen (prefix)) != 0) { + fprintf (stderr, "%s: test failed: missing context prefix: %s\n", + progname, msg); + exit (EXIT_FAILURE); + } + if (errnum != experr) { + fprintf (stderr, "%s: test failed: " + "expected errno = %d (%s), but got %d\n", + progname, experr, strerror (experr), errnum); + exit (EXIT_FAILURE); + } +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + const char *cmd[] = { + "nbdkit", "-s", "-v", "--exit-with-parent", "memory", "1048576", NULL + }; + uint32_t strict; + + progname = argv[0]; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ());
(3) Minor inconsistency with check(): we're not printing "progname" here.
+ exit (EXIT_FAILURE); + } + + /* Connect to the server. */ + if (nbd_connect_command (nbd, (char **)cmd) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + }
(4) Another kind of inconsistency: we could use "progname" here, in place of argv[0].
(This applies to all other fprintf()s below.)
Probably copy-and-paste from other similar tests, but I don't mind cleaning those up.
+ + /* FIXME: future API addition to test if server negotiated extended mode. + * Until then, strict flags must ignore the PAYLOAD_LEN flag for pwrite, + * even though it is rejected for other commands. + */ + strict = nbd_get_strict_mode (nbd); + if (!(strict & LIBNBD_STRICT_FLAGS)) { + fprintf (stderr, "%s: test failed: " + "nbd_get_strict_mode did not have expected flag set\n", + argv[0]); + exit (EXIT_FAILURE); + }
Not sure if I understand this check. Per <https://libguestfs.org/nbd_set_strict_mode.3.html>, I take it that LIBNBD_STRICT_FLAGS should be "on" by default. Are you enforcing that? And if so: is it your intent that, *even with* LIBNBD_STRICT_FLAGS, an invalid PAYLOAD_LEN is not rejected (as seen by the libnbd client app), but fixed up silently?
Rather, until we can tell if the server negotiated extended mode, we are ASSUMING that the server did NOT negotiate it, and therefore we are in violation of the spec if we send the flag over the wire anyways.
OK.
We can flag all other API where it is inappropriate to ever use...
+ if (nbd_aio_pread (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) != -1) { + fprintf (stderr, "%s: test failed: " + "nbd_aio_pread did not fail with unexpected flag\n", + argv[0]); + exit (EXIT_FAILURE); + } + check (EINVAL, "nbd_aio_pread: ");
Ah, got it now. We do want APIs other than pwrite to fail.
...but because we don't want to require clients to correctly decide when to pass or omit the flag to their API calls (by us masking out the user's choice and then hardcoding our actual wire behavior based on negotiated mode), passing the flag to pread works even when it would be technically wrong over the wire.
I don't understand. What you describe here (= us fixing up the flag for the caller) applies to *pwrite*, not *pread*. Furthermore, the above check tests pread's behavior, and it expects pread to *fail*. In effect, my understanding of the test code is this: - assume extended headers have not been negotiated - require that the NBD connection be created such that it enforces flag validity on the client side (i.e., "strict mode" including "strict flags") - test that pread fails with PAYLOAD_LEN -- pread should fail *regardless* of extended headers having been negotiated, because (a) if extended headers are not in use, then the flag is altogether invalid, (b) even with extended headers, a read request does not accept the flag. Because we don't add "PAYLOAD_LEN" as a valid flag to pread in the generator code, the check for (b) is always active. - test that pwrite succeeds with PAYLOAD_LEN -- pwrite should succeed *regardless* of extended headers having been negotiated, because we set PAYLOAD_LEN internally, dependent on the extended headers; i.e., ignoring the user's argument. That is, I think I did manage to explain the test to myself, but your most recent answer confuses me again! :)
The FIXME does get modified again later in the series, when I do add in support for detecting when the server supports extended headers.
Right, I assume FIXME in the test code might be addressed together with the TODO in nbd_unlocked_aio_pwrite(). Once we know whether the server negotiated extended headers, *and* if the user asks for strictness regarding the PAYLOAD_LEN flag, we can enforce PAYLOAD_LEN's equivalence with extended headers in pwrite calls. Laszlo
+ + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + }
You could contract these into:
if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1 || nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) { fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); exit (EXIT_FAILURE); }
Sure.
+ + nbd_close (nbd); + exit (EXIT_SUCCESS); +}
In general, I think it's good practice to reach nbd_close() whenever nbd_create() succeeds (that is, on error paths as well, after nbd_create() succeeds). For example, if we connected to the server with systemd socket activation in this test, and (say) one of the pwrites failed, then we'd leak the unix domain socket in the filesystem (from the bind() in "generator/states-connect-socket-activation.c"). "sact_sockpath" is unlinked in nbd_close().
(As written, this test should not be affected, because, according to unix(7), unix domain sockets created with socketpair(2) are unnamed.)
Pre-existing in other tests, but a good observation for a followup patch.
On Wed, May 31, 2023 at 01:59:12PM +0200, Laszlo Ersek wrote:
+ + /* FIXME: future API addition to test if server negotiated extended mode. + * Until then, strict flags must ignore the PAYLOAD_LEN flag for pwrite, + * even though it is rejected for other commands. + */ + strict = nbd_get_strict_mode (nbd); + if (!(strict & LIBNBD_STRICT_FLAGS)) { + fprintf (stderr, "%s: test failed: " + "nbd_get_strict_mode did not have expected flag set\n", + argv[0]); + exit (EXIT_FAILURE); + }
Not sure if I understand this check. Per <https://libguestfs.org/nbd_set_strict_mode.3.html>, I take it that LIBNBD_STRICT_FLAGS should be "on" by default. Are you enforcing that? And if so: is it your intent that, *even with* LIBNBD_STRICT_FLAGS, an invalid PAYLOAD_LEN is not rejected (as seen by the libnbd client app), but fixed up silently?
Rather, until we can tell if the server negotiated extended mode, we are ASSUMING that the server did NOT negotiate it, and therefore we are in violation of the spec if we send the flag over the wire anyways.
OK.
We can flag all other API where it is inappropriate to ever use...
+ if (nbd_aio_pread (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) != -1) { + fprintf (stderr, "%s: test failed: " + "nbd_aio_pread did not fail with unexpected flag\n", + argv[0]); + exit (EXIT_FAILURE); + } + check (EINVAL, "nbd_aio_pread: ");
Ah, got it now. We do want APIs other than pwrite to fail.
...but because we don't want to require clients to correctly decide when to pass or omit the flag to their API calls (by us masking out the user's choice and then hardcoding our actual wire behavior based on negotiated mode), passing the flag to pread works even when it would be technically wrong over the wire.
I don't understand.
What you describe here (= us fixing up the flag for the caller) applies to *pwrite*, not *pread*. Furthermore, the above check tests pread's behavior, and it expects pread to *fail*.
I'll have to blame late-night email responses for this one. Yes, *pread* rejects the flag from the library client (never valid to use; if we bypass safety checks we can send it to the server which will also flag it); *pwrite" ignores the flag from the library client (whether or not we use it, and whether or not our use matches what is negotiated, what we send to the server uses what is negotiated).
In effect, my understanding of the test code is this:
- assume extended headers have not been negotiated
- require that the NBD connection be created such that it enforces flag validity on the client side (i.e., "strict mode" including "strict flags")
- test that pread fails with PAYLOAD_LEN -- pread should fail *regardless* of extended headers having been negotiated, because (a) if extended headers are not in use, then the flag is altogether invalid, (b) even with extended headers, a read request does not accept the flag. Because we don't add "PAYLOAD_LEN" as a valid flag to pread in the generator code, the check for (b) is always active.
- test that pwrite succeeds with PAYLOAD_LEN -- pwrite should succeed *regardless* of extended headers having been negotiated, because we set PAYLOAD_LEN internally, dependent on the extended headers; i.e., ignoring the user's argument.
That is, I think I did manage to explain the test to myself, but your most recent answer confuses me again! :)
Yeah, your explanation is correct, and I should quit writing emails at my bedtime.
The FIXME does get modified again later in the series, when I do add in support for detecting when the server supports extended headers.
Right, I assume FIXME in the test code might be addressed together with the TODO in nbd_unlocked_aio_pwrite(). Once we know whether the server negotiated extended headers, *and* if the user asks for strictness regarding the PAYLOAD_LEN flag, we can enforce PAYLOAD_LEN's equivalence with extended headers in pwrite calls.
I did NOT add a strictness flag in v3; but may do so in v4. You are right that it would require a user to opt-in to that strictness flag before we enforce PAYLOAD_LEN to match the use of extended headers. Unlike other strictness flags that are on by default, no sane user will opt in to this one except for unit tests of low-level protocol behaviors where it can be handy for forcing libnbd to send known-bad packets to see whether a server trying to implement extended headers has done so gracefully, hence, adding the strictness flag is lower priority. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Tue, May 30, 2023 at 01:18:22PM -0500, Eric Blake wrote:
+ /* It is more convenient to manage PAYLOAD_LEN by what was negotiated + * than to require the user to have to set it correctly. + * TODO: Add new h->strict bit to allow intentional protocol violation + * for interoperability testing. + */ + if (h->extended_headers) + flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN; + else + flags &= ~LIBNBD_CMD_FLAG_PAYLOAD_LEN;
Nice -- I wanted to ask for:
flags &= ~(uint32_t)LIBNBD_CMD_FLAG_PAYLOAD_LEN;
due to LIBNBD_CMD_FLAG_PAYLOAD_LEN having type "int".
However: in patch#3, what has type "int" is:
+#define NBD_CMD_FLAG_PAYLOAD_LEN (1<<5)
and here we have LIBNBD_CMD_FLAG_PAYLOAD_LEN instead -- and the latter has type unsigned int already, from your recent commit 69eecae2c03a ("api: Generate flag values as unsigned", 2022-11-11).
Still, worth a (separate) cleanup patch to nbd-protocol.h to prefer unsigned constants for the flag values where they are not generated.
I pushed a preliminary commit 65011cf6 for libnbd along those lines, and will copy the same changes to nbd-protocol.h over to nbdkit shortly. v4 of this series will be rebased on that. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Thu, May 25, 2023 at 08:00:50AM -0500, Eric Blake wrote:
Support sending 64-bit requests if extended headers were negotiated. This includes setting NBD_CMD_FLAG_PAYLOAD_LEN any time we send an extended NBD_CMD_WRITE; this is such a fundamental part of the protocol that for now it is easier to silently ignore whatever value the user passes in for that bit in the flags parameter of nbd_pwrite regardless of the current settings in set_strict_mode, rather than trying to force the user to pass in the correct value to match whether extended mode is negotiated. However, when we later add APIs to give the user more control for interoperability testing, it may be worth adding a new set_strict_mode control knob to explicitly enable the client to intentionally violate the protocol (the testsuite added in this patch would then be updated to match).
In hindsight it's unfortunate that the flags parameter of the library ABI functions like nbd_pread, nbd_pwrite etc got directly mapped to the command flags used on the wire. But given that it is, I agree with this approach.
At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted).
Support for using FLAG_PAYLOAD_LEN with NBD_CMD_BLOCK_STATUS is less fundamental, and deserves to be in its own patch.
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 10 ++- generator/API.ml | 20 +++-- generator/states-issue-command.c | 29 ++++--- generator/states-reply-structured.c | 2 +- lib/rw.c | 17 +++-- tests/Makefile.am | 4 + tests/pwrite-extended.c | 112 ++++++++++++++++++++++++++++ .gitignore | 1 + 8 files changed, 169 insertions(+), 26 deletions(-) create mode 100644 tests/pwrite-extended.c
diff --git a/lib/internal.h b/lib/internal.h index c71980ef..8a5f93d4 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -110,6 +110,9 @@ struct nbd_handle { char *tls_username; /* Username, NULL = use current username */ char *tls_psk_file; /* PSK filename, NULL = no PSK */
+ /* Extended headers. */ + bool extended_headers; /* If we negotiated NBD_OPT_EXTENDED_HEADERS */ + /* Desired metadata contexts. */ bool request_sr; bool request_meta; @@ -263,7 +266,10 @@ struct nbd_handle { /* Issuing a command must use a buffer separate from sbuf, for the * case when we interrupt a request to service a reply. */ - struct nbd_request request; + union { + struct nbd_request compact; + struct nbd_request_ext extended; + } req; bool in_write_payload; bool in_write_shutdown;
@@ -364,7 +370,7 @@ struct command { uint16_t type; uint64_t cookie; uint64_t offset; - uint32_t count; + uint64_t count; void *data; /* Buffer for read/write */ struct command_cb cb; bool initialized; /* For read, true if getting a hole may skip memset */ diff --git a/generator/API.ml b/generator/API.ml index 5fcb0e1f..02c1260d 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -198,11 +198,12 @@ let cmd_flags = flag_prefix = "CMD_FLAG"; guard = Some "((h->strict & LIBNBD_STRICT_FLAGS) || flags > UINT16_MAX)"; flags = [ - "FUA", 1 lsl 0; - "NO_HOLE", 1 lsl 1; - "DF", 1 lsl 2; - "REQ_ONE", 1 lsl 3; - "FAST_ZERO", 1 lsl 4; + "FUA", 1 lsl 0; + "NO_HOLE", 1 lsl 1; + "DF", 1 lsl 2; + "REQ_ONE", 1 lsl 3; + "FAST_ZERO", 1 lsl 4; + "PAYLOAD_LEN", 1 lsl 5; ] } let handshake_flags = { @@ -2507,7 +2508,7 @@ "pread_structured", { "pwrite", { default_call with args = [ BytesIn ("buf", "count"); UInt64 "offset" ]; - optargs = [ OFlags ("flags", cmd_flags, Some ["FUA"]) ]; + optargs = [ OFlags ("flags", cmd_flags, Some ["FUA"; "PAYLOAD_LEN"]) ]; ret = RErr; permitted_states = [ Connected ]; shortdesc = "write to the NBD server"; @@ -2530,7 +2531,10 @@ "pwrite", { C<LIBNBD_CMD_FLAG_FUA> meaning that the server should not return until the data has been committed to permanent storage (if that is supported - some servers cannot do this, see -L<nbd_can_fua(3)>)." +L<nbd_can_fua(3)>). For convenience, libnbd ignores the presence +or absence of the flag C<LIBNBD_CMD_FLAG_PAYLOAD_LEN> in C<flags>, +while correctly using the flag over the wire according to whether +extended headers were negotiated." ^ strict_call_description; see_also = [Link "can_fua"; Link "is_read_only"; Link "aio_pwrite"; Link "get_block_size"; @@ -3220,7 +3224,7 @@ "aio_pwrite", { default_call with args = [ BytesPersistIn ("buf", "count"); UInt64 "offset" ]; optargs = [ OClosure completion_closure; - OFlags ("flags", cmd_flags, Some ["FUA"]) ]; + OFlags ("flags", cmd_flags, Some ["FUA"; "PAYLOAD_LEN"]) ]; ret = RCookie; permitted_states = [ Connected ]; shortdesc = "write to the NBD server"; diff --git a/generator/states-issue-command.c b/generator/states-issue-command.c index 111e131c..79136b61 100644 --- a/generator/states-issue-command.c +++ b/generator/states-issue-command.c @@ -41,15 +41,24 @@ ISSUE_COMMAND.START: return 0; }
- h->request.magic = htobe32 (NBD_REQUEST_MAGIC); - h->request.flags = htobe16 (cmd->flags); - h->request.type = htobe16 (cmd->type); - h->request.handle = htobe64 (cmd->cookie); - h->request.offset = htobe64 (cmd->offset); - h->request.count = htobe32 (cmd->count); + /* These fields are coincident between req.compact and req.extended */ + h->req.compact.flags = htobe16 (cmd->flags); + h->req.compact.type = htobe16 (cmd->type); + h->req.compact.handle = htobe64 (cmd->cookie); + h->req.compact.offset = htobe64 (cmd->offset); + if (h->extended_headers) { + h->req.extended.magic = htobe32 (NBD_EXTENDED_REQUEST_MAGIC); + h->req.extended.count = htobe64 (cmd->count); + h->wlen = sizeof (h->req.extended); + } + else { + assert (cmd->count <= UINT32_MAX); + h->req.compact.magic = htobe32 (NBD_REQUEST_MAGIC); + h->req.compact.count = htobe32 (cmd->count); + h->wlen = sizeof (h->req.compact); + } h->chunks_sent++; - h->wbuf = &h->request; - h->wlen = sizeof (h->request); + h->wbuf = &h->req; if (cmd->type == NBD_CMD_WRITE || cmd->next) h->wflags = MSG_MORE; SET_NEXT_STATE (%SEND_REQUEST); @@ -74,7 +83,7 @@ ISSUE_COMMAND.PREPARE_WRITE_PAYLOAD:
assert (h->cmds_to_issue != NULL); cmd = h->cmds_to_issue; - assert (cmd->cookie == be64toh (h->request.handle)); + assert (cmd->cookie == be64toh (h->req.compact.handle)); if (cmd->type == NBD_CMD_WRITE) { h->wbuf = cmd->data; h->wlen = cmd->count; @@ -120,7 +129,7 @@ ISSUE_COMMAND.FINISH: assert (!h->wlen); assert (h->cmds_to_issue != NULL); cmd = h->cmds_to_issue; - assert (cmd->cookie == be64toh (h->request.handle)); + assert (cmd->cookie == be64toh (h->req.compact.handle)); h->cmds_to_issue = cmd->next; if (h->cmds_to_issue_tail == cmd) h->cmds_to_issue_tail = NULL; diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 6f96945a..df509216 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -34,7 +34,7 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length, offset + length > cmd->offset + cmd->count) { set_error (0, "range of structured reply is out of bounds, " "offset=%" PRIu64 ", cmd->offset=%" PRIu64 ", " - "length=%" PRIu32 ", cmd->count=%" PRIu32 ": " + "length=%" PRIu32 ", cmd->count=%" PRIu64 ": " "this is likely to be a bug in the NBD server", offset, cmd->offset, length, cmd->count); return false; diff --git a/lib/rw.c b/lib/rw.c index 3dc3499e..8b2bd4cc 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -223,13 +223,11 @@ nbd_internal_command_common (struct nbd_handle *h, } break;
- /* Other commands are currently limited by the 32 bit field in the - * command structure on the wire, but in future we hope to support - * 64 bit values here with a change to the NBD protocol which is - * being discussed upstream. + /* Other commands are limited by the 32 bit field in the command + * structure on the wire, unless extended headers were negotiated. */ default: - if (count > UINT32_MAX) { + if (!h->extended_headers && count > UINT32_MAX) { set_error (ERANGE, "request too large: maximum request size is %" PRIu32, UINT32_MAX); goto err; @@ -358,6 +356,15 @@ nbd_unlocked_aio_pwrite (struct nbd_handle *h, const void *buf, return -1; } } + /* It is more convenient to manage PAYLOAD_LEN by what was negotiated + * than to require the user to have to set it correctly. + * TODO: Add new h->strict bit to allow intentional protocol violation + * for interoperability testing. + */ + if (h->extended_headers) + flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN; + else + flags &= ~LIBNBD_CMD_FLAG_PAYLOAD_LEN;
SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_WRITE, offset, count, diff --git a/tests/Makefile.am b/tests/Makefile.am index 3a93251e..8b839bf5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -232,6 +232,7 @@ check_PROGRAMS += \ closure-lifetimes \ pread-initialize \ socket-activation-name \ + pwrite-extended \
Spaces vs tabs here, as Laszlo already pointed out.
$(NULL)
TESTS += \ @@ -650,6 +651,9 @@ socket_activation_name_SOURCES = \ requires.h socket_activation_name_LDADD = $(top_builddir)/lib/libnbd.la
+pwrite_extended_SOURCES = pwrite-extended.c +pwrite_extended_LDADD = $(top_builddir)/lib/libnbd.la + #---------------------------------------------------------------------- # Testing TLS support.
diff --git a/tests/pwrite-extended.c b/tests/pwrite-extended.c new file mode 100644 index 00000000..f0b5a3f3 --- /dev/null +++ b/tests/pwrite-extended.c @@ -0,0 +1,112 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Check behavior of pwrite with PAYLOAD_LEN flag for extended headers. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <unistd.h> +#include <sys/stat.h> + +#include <libnbd.h> + +static char *progname; +static char buf[512]; + +static void +check (int experr, const char *prefix) +{ + const char *msg = nbd_get_error (); + int errnum = nbd_get_errno (); + + fprintf (stderr, "error: \"%s\"\n", msg); + fprintf (stderr, "errno: %d (%s)\n", errnum, strerror (errnum)); + if (strncmp (msg, prefix, strlen (prefix)) != 0) { + fprintf (stderr, "%s: test failed: missing context prefix: %s\n", + progname, msg); + exit (EXIT_FAILURE); + } + if (errnum != experr) { + fprintf (stderr, "%s: test failed: " + "expected errno = %d (%s), but got %d\n", + progname, experr, strerror (experr), errnum); + exit (EXIT_FAILURE); + } +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + const char *cmd[] = { + "nbdkit", "-s", "-v", "--exit-with-parent", "memory", "1048576", NULL + }; + uint32_t strict; + + progname = argv[0]; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Connect to the server. */ + if (nbd_connect_command (nbd, (char **)cmd) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* FIXME: future API addition to test if server negotiated extended mode. + * Until then, strict flags must ignore the PAYLOAD_LEN flag for pwrite, + * even though it is rejected for other commands. + */ + strict = nbd_get_strict_mode (nbd); + if (!(strict & LIBNBD_STRICT_FLAGS)) { + fprintf (stderr, "%s: test failed: " + "nbd_get_strict_mode did not have expected flag set\n", + argv[0]); + exit (EXIT_FAILURE); + } + if (nbd_aio_pread (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) != -1) { + fprintf (stderr, "%s: test failed: " + "nbd_aio_pread did not fail with unexpected flag\n", + argv[0]); + exit (EXIT_FAILURE); + } + check (EINVAL, "nbd_aio_pread: "); + + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) { + fprintf (stderr, "%s: %s\n", argv[0], nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +} diff --git a/.gitignore b/.gitignore index fe7feffa..bc7c2c37 100644 --- a/.gitignore +++ b/.gitignore @@ -250,6 +250,7 @@ Makefile.in /tests/pki/ /tests/pread-initialize /tests/private-data +/tests/pwrite-extended /tests/read-only-flag /tests/read-write-flag /tests/server-death
Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Support receiving headers for 64-bit replies if extended headers were negotiated. We already insist that the server not send us too much payload in one reply, so we can exploit that and merge the 64-bit length back into a normalized 32-bit field for the rest of the payload length calculations. The NBD protocol specifically documents that extended mode takes precedence over structured replies, and that there are no simple replies in extended mode. We can also take advantage that the handle field is in the same offset in all the various reply types. Note that if we negotiate extended headers, but a non-compliant server replies with a non-extended header, this patch will stall waiting for the server to send more bytes rather than noticing that the magic number is wrong (for aio operations, you'll get a magic number mismatch once you send a second command that elicits a reply; but for blocking operations, we basically deadlock). The easy alternative would be to read just the first 4 bytes of magic, then determine how many more bytes to expect; but that would require more states and syscalls, and not worth it since the typical server will be compliant. The other alternative is what the next patch implements: teaching REPLY.RECV_REPLY to handle short reads that were at least long enough to transmit magic to specifically look for magic number mismatch. At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted). Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 1 + generator/states-reply-structured.c | 52 +++++++++++++++++++---------- generator/states-reply.c | 34 ++++++++++++------- 3 files changed, 58 insertions(+), 29 deletions(-) diff --git a/lib/internal.h b/lib/internal.h index 8a5f93d4..e4e21a4d 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -243,6 +243,7 @@ struct nbd_handle { union { struct nbd_simple_reply simple; struct nbd_structured_reply structured; + struct nbd_extended_reply extended; } hdr; union { struct nbd_structured_reply_offset_data offset_data; diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index df509216..c53aed7b 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -45,14 +45,20 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length, STATE_MACHINE { REPLY.STRUCTURED_REPLY.START: - /* We've only read the simple_reply. The structured_reply is longer, - * so read the remaining part. + /* If we have extended headers, we've already read the entire header. + * Otherwise, we've only read enough for a simple_reply; since structured + * replies are longer, read the remaining part. */ - h->rbuf = &h->sbuf; - h->rbuf = (char *)h->rbuf + sizeof h->sbuf.reply.hdr.simple; - h->rlen = sizeof h->sbuf.reply.hdr.structured; - h->rlen -= sizeof h->sbuf.reply.hdr.simple; - SET_NEXT_STATE (%RECV_REMAINING); + if (h->extended_headers) { + assert (h->rbuf == sizeof h->sbuf.reply.hdr.extended + (char*)&h->sbuf); + SET_NEXT_STATE (%CHECK); + } + else { + assert (h->rbuf == sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf); + h->rlen = sizeof h->sbuf.reply.hdr.structured - + sizeof h->sbuf.reply.hdr.simple; + SET_NEXT_STATE (%RECV_REMAINING); + } return 0; REPLY.STRUCTURED_REPLY.RECV_REMAINING: @@ -69,11 +75,18 @@ REPLY.STRUCTURED_REPLY.RECV_REMAINING: REPLY.STRUCTURED_REPLY.CHECK: struct command *cmd = h->reply_cmd; uint16_t flags, type; - uint32_t length; + uint64_t length; + uint64_t offset = -1; + assert (cmd); flags = be16toh (h->sbuf.reply.hdr.structured.flags); type = be16toh (h->sbuf.reply.hdr.structured.type); - length = be32toh (h->sbuf.reply.hdr.structured.length); + if (h->extended_headers) { + length = be64toh (h->sbuf.reply.hdr.extended.length); + offset = be64toh (h->sbuf.reply.hdr.extended.offset); + } + else + length = be32toh (h->sbuf.reply.hdr.structured.length); /* Reject a server that replies with too much information, but don't * reject a single structured reply to NBD_CMD_READ on the largest @@ -83,13 +96,18 @@ REPLY.STRUCTURED_REPLY.CHECK: * not worth keeping the connection alive. */ if (length > MAX_REQUEST_SIZE + sizeof h->sbuf.reply.payload.offset_data) { - set_error (0, "invalid server reply length %" PRIu32, length); + set_error (0, "invalid server reply length %" PRIu64, length); SET_NEXT_STATE (%.DEAD); return 0; } + /* For convenience, we now normalize extended replies into compact, + * doable since we validated that payload length fits in 32 bits. + */ + h->sbuf.reply.hdr.structured.length = length; /* Skip an unexpected structured reply, including to an unknown cookie. */ - if (cmd == NULL || !h->structured_replies) + if (cmd == NULL || !h->structured_replies || + (h->extended_headers && offset != cmd->offset)) goto resync; switch (type) { @@ -168,7 +186,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ assert (length >= sizeof h->sbuf.reply.payload.error.error.error); assert (cmd); @@ -207,7 +225,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR_MESSAGE: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ msglen = be16toh (h->sbuf.reply.payload.error.error.len); type = be16toh (h->sbuf.reply.hdr.structured.type); @@ -307,7 +325,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ offset = be64toh (h->sbuf.reply.payload.offset_data.offset); assert (cmd); /* guaranteed by CHECK */ @@ -346,7 +364,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ offset = be64toh (h->sbuf.reply.payload.offset_data.offset); assert (cmd); /* guaranteed by CHECK */ @@ -426,7 +444,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); @@ -460,7 +478,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); diff --git a/generator/states-reply.c b/generator/states-reply.c index 31e4bd2f..4e9f2dde 100644 --- a/generator/states-reply.c +++ b/generator/states-reply.c @@ -62,15 +62,19 @@ REPLY.START: */ ssize_t r; - /* We read all replies initially as if they are simple replies, but - * check the magic in CHECK_SIMPLE_OR_STRUCTURED_REPLY below. - * This works because the structured_reply header is larger. + /* With extended headers, there is only one size to read, so we can do it + * all in one syscall. But for older structured replies, we don't know if + * we have a simple or structured reply until we read the magic number, + * requiring a two-part read with CHECK_SIMPLE_OR_STRUCTURED_REPLY below. */ assert (h->reply_cmd == NULL); assert (h->rlen == 0); h->rbuf = &h->sbuf.reply.hdr; - h->rlen = sizeof h->sbuf.reply.hdr.simple; + if (h->extended_headers) + h->rlen = sizeof h->sbuf.reply.hdr.extended; + else + h->rlen = sizeof h->sbuf.reply.hdr.simple; r = h->sock->ops->recv (h, h->sock, h->rbuf, h->rlen); if (r == -1) { @@ -116,16 +120,22 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: uint64_t cookie; magic = be32toh (h->sbuf.reply.hdr.simple.magic); - if (magic == NBD_SIMPLE_REPLY_MAGIC) { + switch (magic) { + case NBD_SIMPLE_REPLY_MAGIC: + if (h->extended_headers) + goto invalid; SET_NEXT_STATE (%SIMPLE_REPLY.START); - } - else if (magic == NBD_STRUCTURED_REPLY_MAGIC) { + break; + case NBD_STRUCTURED_REPLY_MAGIC: + case NBD_EXTENDED_REPLY_MAGIC: + if ((magic == NBD_STRUCTURED_REPLY_MAGIC) == h->extended_headers) + goto invalid; SET_NEXT_STATE (%STRUCTURED_REPLY.START); - } - else { - /* We've probably lost synchronization. */ - SET_NEXT_STATE (%.DEAD); - set_error (0, "invalid reply magic 0x%" PRIx32, magic); + break; + default: + invalid: + SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); #if 0 /* uncomment to see desynchronized data */ nbd_internal_hexdump (&h->sbuf.reply.hdr.simple, sizeof (h->sbuf.reply.hdr.simple), -- 2.40.1
On 5/25/23 15:00, Eric Blake wrote:
Support receiving headers for 64-bit replies if extended headers were negotiated. We already insist that the server not send us too much payload in one reply, so we can exploit that and merge the 64-bit length back into a normalized 32-bit field for the rest of the payload length calculations. The NBD protocol specifically documents that extended mode takes precedence over structured replies, and that there are no simple replies in extended mode. We can also take advantage that the handle field is in the same offset in all the various reply types.
(1) We might want to replace "handle" with "cookie" at this point.
Note that if we negotiate extended headers, but a non-compliant server replies with a non-extended header, this patch will stall waiting for the server to send more bytes
Ah, yes. If we negotiate extended headers, we set "rlen" to "sizeof extended" in REPLY.START, which is larger than either of "sizeof simple" and "sizeof structured". Therefore we'll get stuck in REPLY.RECV_REPLY. Correct?
rather than noticing that the magic number is wrong (for aio operations, you'll get a magic number mismatch once you send a second command that elicits a reply; but for blocking operations, we basically deadlock). The easy alternative would be to read just the first 4 bytes of magic, then determine how many more bytes to expect; but that would require more states and syscalls, and not worth it since the typical server will be compliant.
Not liking this "not worth it" argument. But...
The other alternative is what the next patch implements: teaching REPLY.RECV_REPLY to handle short reads that were at least long enough to transmit magic to specifically look for magic number mismatch.
... that sounds OK!
At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted).
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 1 + generator/states-reply-structured.c | 52 +++++++++++++++++++---------- generator/states-reply.c | 34 ++++++++++++------- 3 files changed, 58 insertions(+), 29 deletions(-)
Probably best to reorder the files in this patch for review:
diff --git a/lib/internal.h b/lib/internal.h index 8a5f93d4..e4e21a4d 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -243,6 +243,7 @@ struct nbd_handle { union { struct nbd_simple_reply simple; struct nbd_structured_reply structured; + struct nbd_extended_reply extended; } hdr; union { struct nbd_structured_reply_offset_data offset_data;
Seems OK.
diff --git a/generator/states-reply.c b/generator/states-reply.c index 31e4bd2f..4e9f2dde 100644 --- a/generator/states-reply.c +++ b/generator/states-reply.c @@ -62,15 +62,19 @@ REPLY.START: */ ssize_t r;
- /* We read all replies initially as if they are simple replies, but - * check the magic in CHECK_SIMPLE_OR_STRUCTURED_REPLY below. - * This works because the structured_reply header is larger. + /* With extended headers, there is only one size to read, so we can do it + * all in one syscall. But for older structured replies, we don't know if + * we have a simple or structured reply until we read the magic number, + * requiring a two-part read with CHECK_SIMPLE_OR_STRUCTURED_REPLY below. */ assert (h->reply_cmd == NULL); assert (h->rlen == 0);
h->rbuf = &h->sbuf.reply.hdr; - h->rlen = sizeof h->sbuf.reply.hdr.simple; + if (h->extended_headers) + h->rlen = sizeof h->sbuf.reply.hdr.extended; + else + h->rlen = sizeof h->sbuf.reply.hdr.simple;
r = h->sock->ops->recv (h, h->sock, h->rbuf, h->rlen); if (r == -1) {
The comment "This works because the structured_reply header is larger" is being removed, which makes the non-ext branch even less comprehensible than before. (2) Can we add the STATIC_ASSERT() here, stating that "sizeof simple" is smaller than "sizeof structured"?
@@ -116,16 +120,22 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: uint64_t cookie;
magic = be32toh (h->sbuf.reply.hdr.simple.magic); - if (magic == NBD_SIMPLE_REPLY_MAGIC) { + switch (magic) { + case NBD_SIMPLE_REPLY_MAGIC: + if (h->extended_headers) + goto invalid; SET_NEXT_STATE (%SIMPLE_REPLY.START); - } - else if (magic == NBD_STRUCTURED_REPLY_MAGIC) { + break; + case NBD_STRUCTURED_REPLY_MAGIC: + case NBD_EXTENDED_REPLY_MAGIC: + if ((magic == NBD_STRUCTURED_REPLY_MAGIC) == h->extended_headers)
Heh, I love this ((a == b) == c) "equivalence" form! :)
+ goto invalid; SET_NEXT_STATE (%STRUCTURED_REPLY.START); - } - else { - /* We've probably lost synchronization. */ - SET_NEXT_STATE (%.DEAD); - set_error (0, "invalid reply magic 0x%" PRIx32, magic); + break; + default: + invalid: + SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); #if 0 /* uncomment to see desynchronized data */ nbd_internal_hexdump (&h->sbuf.reply.hdr.simple, sizeof (h->sbuf.reply.hdr.simple),
My apologies, but I find *this* placement of "invalid" terrible. I thought the "goto invalid" statements were jumping to the end of the function. Now I see they jump effectively to another case label. (3) How about this (on top of your patch, to be squashed): /--------------------- | diff --git a/generator/states-reply.c b/generator/states-reply.c | index 4e9f2ddeb567..ea7bd4e7db91 100644 | --- a/generator/states-reply.c | +++ b/generator/states-reply.c | @@ -133,15 +133,8 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: | SET_NEXT_STATE (%STRUCTURED_REPLY.START); | break; | default: | - invalid: | - SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ | - set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); | -#if 0 /* uncomment to see desynchronized data */ | - nbd_internal_hexdump (&h->sbuf.reply.hdr.simple, | - sizeof (h->sbuf.reply.hdr.simple), | - stderr); | -#endif | - return 0; | + /* We've probably lost synchronization. */ | + goto invalid: | } | | /* NB: This works for both simple and structured replies because the | @@ -159,6 +152,16 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: | h->reply_cmd = cmd; | return 0; | | +invalid: | + SET_NEXT_STATE (%.DEAD); | + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); | +#if 0 /* uncomment to see desynchronized data */ | + nbd_internal_hexdump (&h->sbuf.reply.hdr.simple, | + sizeof (h->sbuf.reply.hdr.simple), | + stderr); | +#endif | + return 0; | + | REPLY.FINISH_COMMAND: | struct command *prev_cmd, *cmd; | uint64_t cookie; \--------------------- ... At the same time, even with this "cleanup" for the labels, I find it relatively confusing (albeit correct, as far as I can tell!) that in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY, we first check the magic number(s) and *then* whether we negotiated extended headers, whereas in all the other state handlers, the order (= condition nesting) is the opposite: we first check extended headers, and deal with any other objects second. This makes the assert()s in REPLY.STRUCTURED_REPLY.START especially tricky to demonstrate -- I think I've managed to do it, it looks correct, it's just hard. So that's a half- (or quarter-)hearted proposal to investigate how REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY looked if there too, "extended_headers" were the outer check. Anyway, I don't feel strongly about this. :) (4) Still in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY, we have a comment saying /* NB: This works for both simple and structured replies because the * handle (our cookie) is stored at the same offset. */ The code under it applies to extended headers too, so the comment should be updated. (There's a similar comment in REPLY.FINISH_COMMAND, too; I'm not sure if that state is relevant for extended headers.) Then:
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index df509216..c53aed7b 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -45,14 +45,20 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length,
STATE_MACHINE { REPLY.STRUCTURED_REPLY.START: - /* We've only read the simple_reply. The structured_reply is longer, - * so read the remaining part. + /* If we have extended headers, we've already read the entire header. + * Otherwise, we've only read enough for a simple_reply; since structured + * replies are longer, read the remaining part. */
(5) Ah, the word "simple_reply", which this change preserves, is a leftover. It should be updated in patch#2, "internal: Refactor layout of replies in sbuf", where the "simple_reply" field is replaced with "reply.hdr.simple" in "sbuf". I guess I didn't notice this omission in patch#2 because the context there only shows the "so read the remaining part" line of the comment, and "simple_reply" is on the preceding line. And then this patch too should refer to "reply.hdr.simple", in the new comment. Alternatively, perhaps use the type name (struct tag, actually): "nbd_simple_reply".
- h->rbuf = &h->sbuf; - h->rbuf = (char *)h->rbuf + sizeof h->sbuf.reply.hdr.simple; - h->rlen = sizeof h->sbuf.reply.hdr.structured; - h->rlen -= sizeof h->sbuf.reply.hdr.simple; - SET_NEXT_STATE (%RECV_REMAINING); + if (h->extended_headers) { + assert (h->rbuf == sizeof h->sbuf.reply.hdr.extended + (char*)&h->sbuf); + SET_NEXT_STATE (%CHECK); + } + else { + assert (h->rbuf == sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf); + h->rlen = sizeof h->sbuf.reply.hdr.structured - + sizeof h->sbuf.reply.hdr.simple; + SET_NEXT_STATE (%RECV_REMAINING); + } return 0;
REPLY.STRUCTURED_REPLY.RECV_REMAINING:
This looks OK, but can we make it less verbose? How about: /---------------------- | diff --git a/lib/internal.h b/lib/internal.h | index e4e21a4d1ffa..a630b2511ff7 100644 | --- a/lib/internal.h | +++ b/lib/internal.h | @@ -240,7 +240,7 @@ struct nbd_handle { | } or; | struct nbd_export_name_option_reply export_name_reply; | struct { | - union { | + union reply_header { | struct nbd_simple_reply simple; | struct nbd_structured_reply structured; | struct nbd_extended_reply extended; | diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c | index c53aed7bb859..852cb5b6053c 100644 | --- a/generator/states-reply-structured.c | +++ b/generator/states-reply-structured.c | @@ -49,14 +49,14 @@ REPLY.STRUCTURED_REPLY.START: | * Otherwise, we've only read enough for a simple_reply; since structured | * replies are longer, read the remaining part. | */ | + union reply_header *hdr = &h->sbuf.reply.hdr; | if (h->extended_headers) { | - assert (h->rbuf == sizeof h->sbuf.reply.hdr.extended + (char*)&h->sbuf); | + assert (h->rbuf == sizeof hdr->extended + (char*)&h->sbuf); | SET_NEXT_STATE (%CHECK); | } | else { | - assert (h->rbuf == sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf); | - h->rlen = sizeof h->sbuf.reply.hdr.structured - | - sizeof h->sbuf.reply.hdr.simple; | + assert (h->rbuf == sizeof hdr->simple + (char*)&h->sbuf); | + h->rlen = sizeof hdr->structured - sizeof hdr->simple; | SET_NEXT_STATE (%RECV_REMAINING); | } | return 0; \---------------------- Anyway, feel free to ignore this -- I can see two counter-arguments myself: - the rest of the code remains just as verbose, - one could argue that the addition sizeof hdr->simple + (char*)&h->sbuf is *more* confusing than sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf Then:
@@ -69,11 +75,18 @@ REPLY.STRUCTURED_REPLY.RECV_REMAINING: REPLY.STRUCTURED_REPLY.CHECK: struct command *cmd = h->reply_cmd; uint16_t flags, type; - uint32_t length; + uint64_t length; + uint64_t offset = -1;
(6) I disagree with initializing the local variable "offset" here. Below (in the rest of REPLY.STRUCTURED_REPLY.CHECK), we only read "offset" back if "extended_headers" is set. But if "extended_headers" is set, we also store a value to "offset", before the read. Initializing "offset" to -1 suggests that the code might otherwise read an indeterminate value from "offset" -- but that's not the case.
+ assert (cmd);
(7) What guarantees this? Is it the lookup code at the end of REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY? But that code can set "reply_cmd" to NULL, in case the cookie is not found; and the cookie lookup there defers to FINISH for diagnosing the unassociated reply.
flags = be16toh (h->sbuf.reply.hdr.structured.flags); type = be16toh (h->sbuf.reply.hdr.structured.type); - length = be32toh (h->sbuf.reply.hdr.structured.length); + if (h->extended_headers) { + length = be64toh (h->sbuf.reply.hdr.extended.length); + offset = be64toh (h->sbuf.reply.hdr.extended.offset); + } + else + length = be32toh (h->sbuf.reply.hdr.structured.length);
/* Reject a server that replies with too much information, but don't * reject a single structured reply to NBD_CMD_READ on the largest @@ -83,13 +96,18 @@ REPLY.STRUCTURED_REPLY.CHECK: * not worth keeping the connection alive. */ if (length > MAX_REQUEST_SIZE + sizeof h->sbuf.reply.payload.offset_data) { - set_error (0, "invalid server reply length %" PRIu32, length); + set_error (0, "invalid server reply length %" PRIu64, length); SET_NEXT_STATE (%.DEAD); return 0; } + /* For convenience, we now normalize extended replies into compact, + * doable since we validated that payload length fits in 32 bits. + */ + h->sbuf.reply.hdr.structured.length = length;
(8) I'm very confused by this. For an extended reply, this will overwrite the "offset" field. I understand we have stolen that field already, above; but that's little solace, especially without specific comments.
/* Skip an unexpected structured reply, including to an unknown cookie. */ - if (cmd == NULL || !h->structured_replies) + if (cmd == NULL || !h->structured_replies || + (h->extended_headers && offset != cmd->offset)) goto resync;
switch (type) {
(9) Preserving the (cmd == NULL) sub-condition, plus the reference to "unkown cookie" in the comment, looks like a logic bug to me: it can never evaluate to "true", given the assert() I'm questioning above at (7). Alternatively, the assert() is wrong. (10) The comment only talks about "unexpected structured reply", but the condition now deals with extended headers too, and I don't know how those relate to each other. What I'm trying to express is that (a) checking "structured_replies" *here*, but (b) checking "extended_headers" against magic numbers in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY, at the same time, seems inconsistent. If we get a structured reply after *not* having negotiated them, we should be able to catch that in the exact same location -- i.e., where we check magic numbers -- where we can also realize that we're getting an extended reply in spite of *not* having negotiated *those*. In other words, the treatment differs, and I don't know why: if we get an unexpected structured reply, we skip it and "resync", whereas if we get an unexpected extended reply, we move to the DEAD state. I think it's fine to skip replies that refer to unknown cookies, but *reply format* violations should be fatal. Then, this is not a request for an update, just a comment: I don't understand why the spec introduced the "offset" field at all. It does not seem to carry additional information beyond the cookie. So we can certainly check that the response's offset matches the command's offset (the code looks OK), but the larger purpose is unclear. (And this is probably also the reason why you get away with corrupting "nbd_extended_reply.offset" at (8) -- the field is never again needed, beyond the sanity check performed here.)
@@ -168,7 +186,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ assert (length >= sizeof h->sbuf.reply.payload.error.error.error); assert (cmd);
@@ -207,7 +225,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR_MESSAGE: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ msglen = be16toh (h->sbuf.reply.payload.error.error.len); type = be16toh (h->sbuf.reply.hdr.structured.type);
@@ -307,7 +325,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ offset = be64toh (h->sbuf.reply.payload.offset_data.offset);
assert (cmd); /* guaranteed by CHECK */ @@ -346,7 +364,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ offset = be64toh (h->sbuf.reply.payload.offset_data.offset);
assert (cmd); /* guaranteed by CHECK */ @@ -426,7 +444,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */
assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); @@ -460,7 +478,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */
assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS);
(11) This is all fallout from (8). The commit message does document it (in the first paragraph), but I don't understand where we *benefit* from stuffing "nbd_extended_reply.length" into "nbd_structured_reply.length". Regarding downsides thereof, I can see two: - as I mentioned, "nbd_extended_reply.offset" becomes unusable, - "nbd_structured_reply.length" will no longer be big-endian but host-endian, which arguably does not match the other fields' endianness in the scratch space (= sbuf). I feel this back-stuffing brings the stashed header into an inconsistent state that is specific to a subset of the automaton's states. I'd rather introduce an entirely new field to "sbuf.reply" -- between "sbuf.reply.hdr" and "sbuf.reply.payload" --, if we really need to cache a host-endian length value, regardless of which kind of reply header we got. Thanks! Laszlo
On Thu, Jun 01, 2023 at 11:04:05AM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
Support receiving headers for 64-bit replies if extended headers were negotiated. We already insist that the server not send us too much payload in one reply, so we can exploit that and merge the 64-bit length back into a normalized 32-bit field for the rest of the payload length calculations. The NBD protocol specifically documents that extended mode takes precedence over structured replies, and that there are no simple replies in extended mode. We can also take advantage that the handle field is in the same offset in all the various reply types.
(1) We might want to replace "handle" with "cookie" at this point.
Yep, I'm already rebasing the series to do that, now that c1df4df9 landed.
Note that if we negotiate extended headers, but a non-compliant server replies with a non-extended header, this patch will stall waiting for the server to send more bytes
Ah, yes. If we negotiate extended headers, we set "rlen" to "sizeof extended" in REPLY.START, which is larger than either of "sizeof simple" and "sizeof structured". Therefore we'll get stuck in REPLY.RECV_REPLY. Correct?
Yes.
rather than noticing that the magic number is wrong (for aio operations, you'll get a magic number mismatch once you send a second command that elicits a reply; but for blocking operations, we basically deadlock). The easy alternative would be to read just the first 4 bytes of magic, then determine how many more bytes to expect; but that would require more states and syscalls, and not worth it since the typical server will be compliant.
Not liking this "not worth it" argument. But...
The other alternative is what the next patch implements: teaching REPLY.RECV_REPLY to handle short reads that were at least long enough to transmit magic to specifically look for magic number mismatch.
... that sounds OK!
If you haven't guessed already, I actually did hit this bug (my initial qemu patches sent the wrong reply type, and it took me several minutes to figure out why libnbd was hung), so I was also pleasantly surprised at how easy the next patch turned out to be, after first trying (and failing) to go down the rabbit hole mentioned here of adding yet more states. But, as you mentioned earlier in the series, it MAY be worth rearranging states anyways, to specifically route extended header parsing differently from structured header parsing. So I'm still playing with that, and seeing what may happen to this patch as fallout.
At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted).
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 1 + generator/states-reply-structured.c | 52 +++++++++++++++++++---------- generator/states-reply.c | 34 ++++++++++++------- 3 files changed, 58 insertions(+), 29 deletions(-)
Probably best to reorder the files in this patch for review:
I see what you mean: because of the state hierarchy, it is probably worth tweaking the git orderfile to favor files nearer the top of the state tree first, rather than strict alphabetical ordering of *.c. I may just push a patch for that without needing review, as it doesn't affect library semantics at all. ...
+++ b/generator/states-reply.c @@ -62,15 +62,19 @@ REPLY.START: */ ssize_t r;
- /* We read all replies initially as if they are simple replies, but - * check the magic in CHECK_SIMPLE_OR_STRUCTURED_REPLY below. - * This works because the structured_reply header is larger. + /* With extended headers, there is only one size to read, so we can do it + * all in one syscall. But for older structured replies, we don't know if + * we have a simple or structured reply until we read the magic number, + * requiring a two-part read with CHECK_SIMPLE_OR_STRUCTURED_REPLY below. */ assert (h->reply_cmd == NULL); assert (h->rlen == 0);
h->rbuf = &h->sbuf.reply.hdr; - h->rlen = sizeof h->sbuf.reply.hdr.simple; + if (h->extended_headers) + h->rlen = sizeof h->sbuf.reply.hdr.extended; + else + h->rlen = sizeof h->sbuf.reply.hdr.simple;
r = h->sock->ops->recv (h, h->sock, h->rbuf, h->rlen); if (r == -1) {
The comment "This works because the structured_reply header is larger" is being removed, which makes the non-ext branch even less comprehensible than before.
(2) Can we add the STATIC_ASSERT() here, stating that "sizeof simple" is smaller than "sizeof structured"?
Yep, I'm already rebasing onto some additional asserts along those lines, based on your reviews earlier in the series.
@@ -116,16 +120,22 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: uint64_t cookie;
magic = be32toh (h->sbuf.reply.hdr.simple.magic); - if (magic == NBD_SIMPLE_REPLY_MAGIC) { + switch (magic) { + case NBD_SIMPLE_REPLY_MAGIC: + if (h->extended_headers) + goto invalid; SET_NEXT_STATE (%SIMPLE_REPLY.START); - } - else if (magic == NBD_STRUCTURED_REPLY_MAGIC) { + break; + case NBD_STRUCTURED_REPLY_MAGIC: + case NBD_EXTENDED_REPLY_MAGIC: + if ((magic == NBD_STRUCTURED_REPLY_MAGIC) == h->extended_headers)
Heh, I love this ((a == b) == c) "equivalence" form! :)
I still do a double-take every time I see 'a < b < c' in languages (like python) that support it as shorthand for 'a < b && b < c'. Even weirder is when you see 'a < b > c'. But yes, C's non-transitive == can be a surprise for the uninitiated.
+ goto invalid; SET_NEXT_STATE (%STRUCTURED_REPLY.START); - } - else { - /* We've probably lost synchronization. */ - SET_NEXT_STATE (%.DEAD); - set_error (0, "invalid reply magic 0x%" PRIx32, magic); + break; + default: + invalid: + SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); #if 0 /* uncomment to see desynchronized data */ nbd_internal_hexdump (&h->sbuf.reply.hdr.simple, sizeof (h->sbuf.reply.hdr.simple),
My apologies, but I find *this* placement of "invalid" terrible. I thought the "goto invalid" statements were jumping to the end of the function. Now I see they jump effectively to another case label.
(3) How about this (on top of your patch, to be squashed):
That part of the patch pre-dates other reviews I've seen from you on the same topic (this patch series has been percolating on my local builds for a long time), so I'm not surprised by your request, and I'm happy to squash in your improvement. I may have copied from other similar code in the generator/states-*.c files, if so, I'll probably do a separate cleanup patch first.
... At the same time, even with this "cleanup" for the labels, I find it relatively confusing (albeit correct, as far as I can tell!) that in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY, we first check the magic number(s) and *then* whether we negotiated extended headers, whereas in all the other state handlers, the order (= condition nesting) is the opposite: we first check extended headers, and deal with any other objects second. This makes the assert()s in REPLY.STRUCTURED_REPLY.START especially tricky to demonstrate -- I think I've managed to do it, it looks correct, it's just hard. So that's a half- (or quarter-)hearted proposal to investigate how REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY looked if there too, "extended_headers" were the outer check. Anyway, I don't feel strongly about this. :)
(4) Still in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY, we have a comment saying
/* NB: This works for both simple and structured replies because the * handle (our cookie) is stored at the same offset. */
The code under it applies to extended headers too, so the comment should be updated.
(There's a similar comment in REPLY.FINISH_COMMAND, too; I'm not sure if that state is relevant for extended headers.)
Indeed, when rebasing, I need to remember to grep (to cover comments) rather than merely relying on the compiler (which only covers code) for renames.
Then:
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index df509216..c53aed7b 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -45,14 +45,20 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length,
STATE_MACHINE { REPLY.STRUCTURED_REPLY.START: - /* We've only read the simple_reply. The structured_reply is longer, - * so read the remaining part. + /* If we have extended headers, we've already read the entire header. + * Otherwise, we've only read enough for a simple_reply; since structured + * replies are longer, read the remaining part. */
(5) Ah, the word "simple_reply", which this change preserves, is a leftover. It should be updated in patch#2, "internal: Refactor layout of replies in sbuf", where the "simple_reply" field is replaced with "reply.hdr.simple" in "sbuf".
I guess I didn't notice this omission in patch#2 because the context there only shows the "so read the remaining part" line of the comment, and "simple_reply" is on the preceding line.
And then this patch too should refer to "reply.hdr.simple", in the new comment.
Alternatively, perhaps use the type name (struct tag, actually): "nbd_simple_reply".
Will fix in the appropriate place(s).
- h->rbuf = &h->sbuf; - h->rbuf = (char *)h->rbuf + sizeof h->sbuf.reply.hdr.simple; - h->rlen = sizeof h->sbuf.reply.hdr.structured; - h->rlen -= sizeof h->sbuf.reply.hdr.simple; - SET_NEXT_STATE (%RECV_REMAINING); + if (h->extended_headers) { + assert (h->rbuf == sizeof h->sbuf.reply.hdr.extended + (char*)&h->sbuf); + SET_NEXT_STATE (%CHECK); + } + else { + assert (h->rbuf == sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf); + h->rlen = sizeof h->sbuf.reply.hdr.structured - + sizeof h->sbuf.reply.hdr.simple; + SET_NEXT_STATE (%RECV_REMAINING); + } return 0;
REPLY.STRUCTURED_REPLY.RECV_REMAINING:
This looks OK, but can we make it less verbose? How about:
/---------------------- | diff --git a/lib/internal.h b/lib/internal.h | index e4e21a4d1ffa..a630b2511ff7 100644 | --- a/lib/internal.h | +++ b/lib/internal.h | @@ -240,7 +240,7 @@ struct nbd_handle { | } or; | struct nbd_export_name_option_reply export_name_reply; | struct { | - union { | + union reply_header {
Oh cool. I keep forgetting that you can declare a type name usable at the top level even while declaring a nested struct. Yeah, doing that probably has some nice line length improvements.
| struct nbd_simple_reply simple; | struct nbd_structured_reply structured; | struct nbd_extended_reply extended; | diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c | index c53aed7bb859..852cb5b6053c 100644 | --- a/generator/states-reply-structured.c | +++ b/generator/states-reply-structured.c | @@ -49,14 +49,14 @@ REPLY.STRUCTURED_REPLY.START: | * Otherwise, we've only read enough for a simple_reply; since structured | * replies are longer, read the remaining part. | */ | + union reply_header *hdr = &h->sbuf.reply.hdr; | if (h->extended_headers) { | - assert (h->rbuf == sizeof h->sbuf.reply.hdr.extended + (char*)&h->sbuf); | + assert (h->rbuf == sizeof hdr->extended + (char*)&h->sbuf); | SET_NEXT_STATE (%CHECK); | } | else { | - assert (h->rbuf == sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf); | - h->rlen = sizeof h->sbuf.reply.hdr.structured - | - sizeof h->sbuf.reply.hdr.simple; | + assert (h->rbuf == sizeof hdr->simple + (char*)&h->sbuf); | + h->rlen = sizeof hdr->structured - sizeof hdr->simple; | SET_NEXT_STATE (%RECV_REMAINING); | } | return 0; \----------------------
Anyway, feel free to ignore this -- I can see two counter-arguments myself:
- the rest of the code remains just as verbose,
Not necessarily - once I add in more STATIC_ASSERTs, being able to refer to individual nested types without having to go all the way through struct nbd_handle or union sbuf will be shorter.
- one could argue that the addition
sizeof hdr->simple + (char*)&h->sbuf
is *more* confusing than
sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf
That argument still remains - it's easier to see that we expect a particular offset from sbuf when the use of sbuf in the offset calculation is not hidden several lines away in the intermediate initialization. I'll think about it for this line.
Then:
@@ -69,11 +75,18 @@ REPLY.STRUCTURED_REPLY.RECV_REMAINING: REPLY.STRUCTURED_REPLY.CHECK: struct command *cmd = h->reply_cmd; uint16_t flags, type; - uint32_t length; + uint64_t length; + uint64_t offset = -1;
(6) I disagree with initializing the local variable "offset" here.
Below (in the rest of REPLY.STRUCTURED_REPLY.CHECK), we only read "offset" back if "extended_headers" is set. But if "extended_headers" is set, we also store a value to "offset", before the read.
Initializing "offset" to -1 suggests that the code might otherwise read an indeterminate value from "offset" -- but that's not the case.
I'll have to double-check; I thought that offset was read even for structured replies (where there really isn't an offset read from the wire), but my rebasing efforts may have changed that.
+ assert (cmd);
(7) What guarantees this?
Is it the lookup code at the end of REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY?
But that code can set "reply_cmd" to NULL, in case the cookie is not found; and the cookie lookup there defers to FINISH for diagnosing the unassociated reply.
Hmmm. Reading ahead, I see what you wrote in (9). This may have been something I copied from other states, but I'll have to double check whether it still makes sense.
flags = be16toh (h->sbuf.reply.hdr.structured.flags); type = be16toh (h->sbuf.reply.hdr.structured.type); - length = be32toh (h->sbuf.reply.hdr.structured.length); + if (h->extended_headers) { + length = be64toh (h->sbuf.reply.hdr.extended.length); + offset = be64toh (h->sbuf.reply.hdr.extended.offset); + } + else + length = be32toh (h->sbuf.reply.hdr.structured.length);
/* Reject a server that replies with too much information, but don't * reject a single structured reply to NBD_CMD_READ on the largest @@ -83,13 +96,18 @@ REPLY.STRUCTURED_REPLY.CHECK: * not worth keeping the connection alive. */ if (length > MAX_REQUEST_SIZE + sizeof h->sbuf.reply.payload.offset_data) { - set_error (0, "invalid server reply length %" PRIu32, length); + set_error (0, "invalid server reply length %" PRIu64, length); SET_NEXT_STATE (%.DEAD); return 0; } + /* For convenience, we now normalize extended replies into compact, + * doable since we validated that payload length fits in 32 bits. + */ + h->sbuf.reply.hdr.structured.length = length;
(8) I'm very confused by this. For an extended reply, this will overwrite the "offset" field.
At one point, I considered normalizing in the opposite direction (take structured replies and widen them into the extended header form); it required touching more lines of code, so I didn't pursue it at the time. But I can see how compressing things down (intentionally throwing away information we will no longer reference) can be confusing without good comments, so at a minimum, I will be adding comments, if not outright going with the widening rather than narrowing approach.
I understand we have stolen that field already, above; but that's little solace, especially without specific comments.
/* Skip an unexpected structured reply, including to an unknown cookie. */ - if (cmd == NULL || !h->structured_replies) + if (cmd == NULL || !h->structured_replies || + (h->extended_headers && offset != cmd->offset)) goto resync;
switch (type) {
(9) Preserving the (cmd == NULL) sub-condition, plus the reference to "unkown cookie" in the comment, looks like a logic bug to me: it can never evaluate to "true", given the assert() I'm questioning above at (7).
Alternatively, the assert() is wrong.
Off-hand, I think the assert is correct and I do have a dead condition here; but I'll have to convince myself of that. Since the condition is pre-existing, it may be worth a separate patch adding just the assertion and removing the 'cmd == NULL' check here, if it is correct.
(10) The comment only talks about "unexpected structured reply", but the condition now deals with extended headers too, and I don't know how those relate to each other.
What I'm trying to express is that
(a) checking "structured_replies" *here*, but
(b) checking "extended_headers" against magic numbers in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY,
at the same time, seems inconsistent. If we get a structured reply after *not* having negotiated them, we should be able to catch that in the exact same location -- i.e., where we check magic numbers -- where we can also realize that we're getting an extended reply in spite of *not* having negotiated *those*.
In other words, the treatment differs, and I don't know why: if we get an unexpected structured reply, we skip it and "resync", whereas if we get an unexpected extended reply, we move to the DEAD state.
I think it's fine to skip replies that refer to unknown cookies, but *reply format* violations should be fatal.
I think this all stems from me trying to be as generous as possible with bad servers: if we get a wrong reply type per the protocol but it is a reply type we know how to handle, AND we haven't consumed too many bytes for the size of that reply, then we can tolerate the server's bug. If it is the wrong type, and we already read too many bytes for what the right type would be, we can't (easily) undo the fact that we read too much. If we did not negotiate extended headers, we start by reading the length of a simple header; if the magic says it was instead an extended header, we can tolerate the server's mistake by reading the rest of that packet. But I definitely see your point about consolidating the decisions to be local to one state, rather than split across two files, and aiming for consistent handling. I'm liking your idea earlier in the series about reworking the state engine to FULLY parse the header (whether simple, structured, or extended) in states-reply.c, and only then move to states-reply-structured.c if the header was structured or extended, rather than splitting the header parse across two files.
Then, this is not a request for an update, just a comment: I don't understand why the spec introduced the "offset" field at all. It does not seem to carry additional information beyond the cookie. So we can certainly check that the response's offset matches the command's offset (the code looks OK), but the larger purpose is unclear. (And this is probably also the reason why you get away with corrupting "nbd_extended_reply.offset" at (8) -- the field is never again needed, beyond the sanity check performed here.)
The evolution of that field: in my first draft, I wanted power-of-2 sizing to the reply header (both request and reply having a header of exactly 32 bytes); this left dead space in the reply packet, which I originally tried to mandate to be zero-filled. But when looking at the types again, I noticed that if the 'offset' field occupies the same relative place in the request and reply struct, a server can rewrite the reply in the same memory as it had saved for the request (I don't know that I actually implemented it that way in qemu as server, but it is possible for some other server). There's also NBD_REPLY_TYPE_OFFSET_DATA and NBD_REPLY_TYPE_OFFSET_HOLE (used in response to NBD_CMD_READ), which each return an offset field in order to handle the case where a single client read request was split across multiple server replies. When we first specified structured replies, the spec was ambiguous: if I issue NBD_CMD_READ(length=1k, offset=0) but the server replies with the pair OFFSET_DATA(length=512, offset=0) and OFFSET_HOLE(length=512, offset=512), it's obvious how to reconstruct the original buffer (read into the first 512 bytes, memset the second 512 bytes). But if I issue NBD_CMD_READ(length=1k, offset=512), should the server reply OFFSET_DATA(length=512, offset=512) and OFFSET_HOLE(length=512, offset=1k) (that is, the reply offsets are absolute to the overall image), or with OFFSET_DATA(length=512, offset=0) and OFFSET_HOLE(length=512, offset=512) (that is, the reply offsets are relative to the start of the 1k buffer of the read request)? The initial implementations of qemu and nbdkit disagreed, and the spec ended up settling in favor of the former (see nbd.git commit 587ba722). As a result, when receiving an absolute offset in a data reply, the client has to place the data into buffer + (reply_offset - request_offset) (that is, convert the server's absolute offset back into a buffer-relative offset), which requires knowing what the reqeust offset was; having the server return this offset in extended replies may make it easier to implement a client that doesn't have to store its request offset separately. Since libnbd also handles non-extended headers (and has to hang on to the request offset for those), you are right that we end up merely validating that the server's reply offset was expected, and then throwing it away as it didn't add anything further than a validation that we haven't lost sync with the server.
@@ -168,7 +186,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ assert (length >= sizeof h->sbuf.reply.payload.error.error.error); assert (cmd);
@@ -207,7 +225,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR_MESSAGE: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ msglen = be16toh (h->sbuf.reply.payload.error.error.len); type = be16toh (h->sbuf.reply.hdr.structured.type);
@@ -307,7 +325,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ offset = be64toh (h->sbuf.reply.payload.offset_data.offset);
assert (cmd); /* guaranteed by CHECK */ @@ -346,7 +364,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ offset = be64toh (h->sbuf.reply.payload.offset_data.offset);
assert (cmd); /* guaranteed by CHECK */ @@ -426,7 +444,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */
assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); @@ -460,7 +478,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */
assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS);
(11) This is all fallout from (8). The commit message does document it (in the first paragraph), but I don't understand where we *benefit* from stuffing "nbd_extended_reply.length" into "nbd_structured_reply.length".
The benefit is that all later code only has to deal with nbd_structured_reply (rather than adding yet more 'if (h->extended_reply)' in all subsequent states). Conversely, if I had widened instead of narrowed, then all later code would only need to doea with nbd_extended_reply.
Regarding downsides thereof, I can see two:
- as I mentioned, "nbd_extended_reply.offset" becomes unusable,
- "nbd_structured_reply.length" will no longer be big-endian but host-endian, which arguably does not match the other fields' endianness in the scratch space (= sbuf).
I feel this back-stuffing brings the stashed header into an inconsistent state that is specific to a subset of the automaton's states.
I'd rather introduce an entirely new field to "sbuf.reply" -- between "sbuf.reply.hdr" and "sbuf.reply.payload" --, if we really need to cache a host-endian length value, regardless of which kind of reply header we got.
You may be on to something. Leaving the wire reply ALWAYS in network endian order, and adding a few bytes to nbd_reply to stash a normalized host-endian decoded value, would certainly be less confusing than having to figure out "which parts of my wire reply are still network-endian vs. natively converted". It may make the overall structure a few bytes larger, but that's probably in the noise (since our nbd_handle already accomodatese maximum-length NBD strings of 4k, adding up to another 32 bytes for fully-normalized host-endian form isn't bad). I'll keep that in mind for v4 (this is not the only place that needs normalization; handling 32- vs. 64-bit block status replies also needs normalization; the tradeoff between minimal storage by normalizing in place vs. maintainability by normalizing into a copy has interesting implications). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 6/1/23 15:00, Eric Blake wrote:
On Thu, Jun 01, 2023 at 11:04:05AM +0200, Laszlo Ersek wrote:
Probably best to reorder the files in this patch for review:
I see what you mean: because of the state hierarchy, it is probably worth tweaking the git orderfile to favor files nearer the top of the state tree first, rather than strict alphabetical ordering of *.c. I may just push a patch for that without needing review, as it doesn't affect library semantics at all.
Ouch, I'm sorry, I meant that *I* should rearrange the hunks in the patch for review! I didn't mean to ask you for any orderfile changes! :) Of course if you can do that: high five! :)
...
+++ b/generator/states-reply.c @@ -62,15 +62,19 @@ REPLY.START: */ ssize_t r;
- /* We read all replies initially as if they are simple replies, but - * check the magic in CHECK_SIMPLE_OR_STRUCTURED_REPLY below. - * This works because the structured_reply header is larger. + /* With extended headers, there is only one size to read, so we can do it + * all in one syscall. But for older structured replies, we don't know if + * we have a simple or structured reply until we read the magic number, + * requiring a two-part read with CHECK_SIMPLE_OR_STRUCTURED_REPLY below. */ assert (h->reply_cmd == NULL); assert (h->rlen == 0);
h->rbuf = &h->sbuf.reply.hdr; - h->rlen = sizeof h->sbuf.reply.hdr.simple; + if (h->extended_headers) + h->rlen = sizeof h->sbuf.reply.hdr.extended; + else + h->rlen = sizeof h->sbuf.reply.hdr.simple;
r = h->sock->ops->recv (h, h->sock, h->rbuf, h->rlen); if (r == -1) {
The comment "This works because the structured_reply header is larger" is being removed, which makes the non-ext branch even less comprehensible than before.
(2) Can we add the STATIC_ASSERT() here, stating that "sizeof simple" is smaller than "sizeof structured"?
Yep, I'm already rebasing onto some additional asserts along those lines, based on your reviews earlier in the series.
@@ -116,16 +120,22 @@ REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY: uint64_t cookie;
magic = be32toh (h->sbuf.reply.hdr.simple.magic); - if (magic == NBD_SIMPLE_REPLY_MAGIC) { + switch (magic) { + case NBD_SIMPLE_REPLY_MAGIC: + if (h->extended_headers) + goto invalid; SET_NEXT_STATE (%SIMPLE_REPLY.START); - } - else if (magic == NBD_STRUCTURED_REPLY_MAGIC) { + break; + case NBD_STRUCTURED_REPLY_MAGIC: + case NBD_EXTENDED_REPLY_MAGIC: + if ((magic == NBD_STRUCTURED_REPLY_MAGIC) == h->extended_headers)
Heh, I love this ((a == b) == c) "equivalence" form! :)
I still do a double-take every time I see 'a < b < c' in languages (like python) that support it as shorthand for 'a < b && b < c'. Even weirder is when you see 'a < b > c'. But yes, C's non-transitive == can be a surprise for the uninitiated.
+ goto invalid; SET_NEXT_STATE (%STRUCTURED_REPLY.START); - } - else { - /* We've probably lost synchronization. */ - SET_NEXT_STATE (%.DEAD); - set_error (0, "invalid reply magic 0x%" PRIx32, magic); + break; + default: + invalid: + SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); #if 0 /* uncomment to see desynchronized data */ nbd_internal_hexdump (&h->sbuf.reply.hdr.simple, sizeof (h->sbuf.reply.hdr.simple),
My apologies, but I find *this* placement of "invalid" terrible. I thought the "goto invalid" statements were jumping to the end of the function. Now I see they jump effectively to another case label.
(3) How about this (on top of your patch, to be squashed):
That part of the patch pre-dates other reviews I've seen from you on the same topic (this patch series has been percolating on my local builds for a long time), so I'm not surprised by your request, and I'm happy to squash in your improvement. I may have copied from other similar code in the generator/states-*.c files, if so, I'll probably do a separate cleanup patch first.
... At the same time, even with this "cleanup" for the labels, I find it relatively confusing (albeit correct, as far as I can tell!) that in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY, we first check the magic number(s) and *then* whether we negotiated extended headers, whereas in all the other state handlers, the order (= condition nesting) is the opposite: we first check extended headers, and deal with any other objects second. This makes the assert()s in REPLY.STRUCTURED_REPLY.START especially tricky to demonstrate -- I think I've managed to do it, it looks correct, it's just hard. So that's a half- (or quarter-)hearted proposal to investigate how REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY looked if there too, "extended_headers" were the outer check. Anyway, I don't feel strongly about this. :)
(4) Still in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY, we have a comment saying
/* NB: This works for both simple and structured replies because the * handle (our cookie) is stored at the same offset. */
The code under it applies to extended headers too, so the comment should be updated.
(There's a similar comment in REPLY.FINISH_COMMAND, too; I'm not sure if that state is relevant for extended headers.)
Indeed, when rebasing, I need to remember to grep (to cover comments) rather than merely relying on the compiler (which only covers code) for renames.
Then:
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index df509216..c53aed7b 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -45,14 +45,20 @@ structured_reply_in_bounds (uint64_t offset, uint32_t length,
STATE_MACHINE { REPLY.STRUCTURED_REPLY.START: - /* We've only read the simple_reply. The structured_reply is longer, - * so read the remaining part. + /* If we have extended headers, we've already read the entire header. + * Otherwise, we've only read enough for a simple_reply; since structured + * replies are longer, read the remaining part. */
(5) Ah, the word "simple_reply", which this change preserves, is a leftover. It should be updated in patch#2, "internal: Refactor layout of replies in sbuf", where the "simple_reply" field is replaced with "reply.hdr.simple" in "sbuf".
I guess I didn't notice this omission in patch#2 because the context there only shows the "so read the remaining part" line of the comment, and "simple_reply" is on the preceding line.
And then this patch too should refer to "reply.hdr.simple", in the new comment.
Alternatively, perhaps use the type name (struct tag, actually): "nbd_simple_reply".
Will fix in the appropriate place(s).
- h->rbuf = &h->sbuf; - h->rbuf = (char *)h->rbuf + sizeof h->sbuf.reply.hdr.simple; - h->rlen = sizeof h->sbuf.reply.hdr.structured; - h->rlen -= sizeof h->sbuf.reply.hdr.simple; - SET_NEXT_STATE (%RECV_REMAINING); + if (h->extended_headers) { + assert (h->rbuf == sizeof h->sbuf.reply.hdr.extended + (char*)&h->sbuf); + SET_NEXT_STATE (%CHECK); + } + else { + assert (h->rbuf == sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf); + h->rlen = sizeof h->sbuf.reply.hdr.structured - + sizeof h->sbuf.reply.hdr.simple; + SET_NEXT_STATE (%RECV_REMAINING); + } return 0;
REPLY.STRUCTURED_REPLY.RECV_REMAINING:
This looks OK, but can we make it less verbose? How about:
/---------------------- | diff --git a/lib/internal.h b/lib/internal.h | index e4e21a4d1ffa..a630b2511ff7 100644 | --- a/lib/internal.h | +++ b/lib/internal.h | @@ -240,7 +240,7 @@ struct nbd_handle { | } or; | struct nbd_export_name_option_reply export_name_reply; | struct { | - union { | + union reply_header {
Oh cool. I keep forgetting that you can declare a type name usable at the top level even while declaring a nested struct. Yeah, doing that probably has some nice line length improvements.
| struct nbd_simple_reply simple; | struct nbd_structured_reply structured; | struct nbd_extended_reply extended; | diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c | index c53aed7bb859..852cb5b6053c 100644 | --- a/generator/states-reply-structured.c | +++ b/generator/states-reply-structured.c | @@ -49,14 +49,14 @@ REPLY.STRUCTURED_REPLY.START: | * Otherwise, we've only read enough for a simple_reply; since structured | * replies are longer, read the remaining part. | */ | + union reply_header *hdr = &h->sbuf.reply.hdr; | if (h->extended_headers) { | - assert (h->rbuf == sizeof h->sbuf.reply.hdr.extended + (char*)&h->sbuf); | + assert (h->rbuf == sizeof hdr->extended + (char*)&h->sbuf); | SET_NEXT_STATE (%CHECK); | } | else { | - assert (h->rbuf == sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf); | - h->rlen = sizeof h->sbuf.reply.hdr.structured - | - sizeof h->sbuf.reply.hdr.simple; | + assert (h->rbuf == sizeof hdr->simple + (char*)&h->sbuf); | + h->rlen = sizeof hdr->structured - sizeof hdr->simple; | SET_NEXT_STATE (%RECV_REMAINING); | } | return 0; \----------------------
Anyway, feel free to ignore this -- I can see two counter-arguments myself:
- the rest of the code remains just as verbose,
Not necessarily - once I add in more STATIC_ASSERTs, being able to refer to individual nested types without having to go all the way through struct nbd_handle or union sbuf will be shorter.
- one could argue that the addition
sizeof hdr->simple + (char*)&h->sbuf
is *more* confusing than
sizeof h->sbuf.reply.hdr.simple + (char*)&h->sbuf
That argument still remains - it's easier to see that we expect a particular offset from sbuf when the use of sbuf in the offset calculation is not hidden several lines away in the intermediate initialization. I'll think about it for this line.
Then:
@@ -69,11 +75,18 @@ REPLY.STRUCTURED_REPLY.RECV_REMAINING: REPLY.STRUCTURED_REPLY.CHECK: struct command *cmd = h->reply_cmd; uint16_t flags, type; - uint32_t length; + uint64_t length; + uint64_t offset = -1;
(6) I disagree with initializing the local variable "offset" here.
Below (in the rest of REPLY.STRUCTURED_REPLY.CHECK), we only read "offset" back if "extended_headers" is set. But if "extended_headers" is set, we also store a value to "offset", before the read.
Initializing "offset" to -1 suggests that the code might otherwise read an indeterminate value from "offset" -- but that's not the case.
I'll have to double-check; I thought that offset was read even for structured replies (where there really isn't an offset read from the wire), but my rebasing efforts may have changed that.
+ assert (cmd);
(7) What guarantees this?
Is it the lookup code at the end of REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY?
But that code can set "reply_cmd" to NULL, in case the cookie is not found; and the cookie lookup there defers to FINISH for diagnosing the unassociated reply.
Hmmm. Reading ahead, I see what you wrote in (9). This may have been something I copied from other states, but I'll have to double check whether it still makes sense.
flags = be16toh (h->sbuf.reply.hdr.structured.flags); type = be16toh (h->sbuf.reply.hdr.structured.type); - length = be32toh (h->sbuf.reply.hdr.structured.length); + if (h->extended_headers) { + length = be64toh (h->sbuf.reply.hdr.extended.length); + offset = be64toh (h->sbuf.reply.hdr.extended.offset); + } + else + length = be32toh (h->sbuf.reply.hdr.structured.length);
/* Reject a server that replies with too much information, but don't * reject a single structured reply to NBD_CMD_READ on the largest @@ -83,13 +96,18 @@ REPLY.STRUCTURED_REPLY.CHECK: * not worth keeping the connection alive. */ if (length > MAX_REQUEST_SIZE + sizeof h->sbuf.reply.payload.offset_data) { - set_error (0, "invalid server reply length %" PRIu32, length); + set_error (0, "invalid server reply length %" PRIu64, length); SET_NEXT_STATE (%.DEAD); return 0; } + /* For convenience, we now normalize extended replies into compact, + * doable since we validated that payload length fits in 32 bits. + */ + h->sbuf.reply.hdr.structured.length = length;
(8) I'm very confused by this. For an extended reply, this will overwrite the "offset" field.
At one point, I considered normalizing in the opposite direction (take structured replies and widen them into the extended header form); it required touching more lines of code, so I didn't pursue it at the time. But I can see how compressing things down (intentionally throwing away information we will no longer reference) can be confusing without good comments, so at a minimum, I will be adding comments, if not outright going with the widening rather than narrowing approach.
I understand we have stolen that field already, above; but that's little solace, especially without specific comments.
/* Skip an unexpected structured reply, including to an unknown cookie. */ - if (cmd == NULL || !h->structured_replies) + if (cmd == NULL || !h->structured_replies || + (h->extended_headers && offset != cmd->offset)) goto resync;
switch (type) {
(9) Preserving the (cmd == NULL) sub-condition, plus the reference to "unkown cookie" in the comment, looks like a logic bug to me: it can never evaluate to "true", given the assert() I'm questioning above at (7).
Alternatively, the assert() is wrong.
Off-hand, I think the assert is correct and I do have a dead condition here; but I'll have to convince myself of that. Since the condition is pre-existing, it may be worth a separate patch adding just the assertion and removing the 'cmd == NULL' check here, if it is correct.
(10) The comment only talks about "unexpected structured reply", but the condition now deals with extended headers too, and I don't know how those relate to each other.
What I'm trying to express is that
(a) checking "structured_replies" *here*, but
(b) checking "extended_headers" against magic numbers in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY,
at the same time, seems inconsistent. If we get a structured reply after *not* having negotiated them, we should be able to catch that in the exact same location -- i.e., where we check magic numbers -- where we can also realize that we're getting an extended reply in spite of *not* having negotiated *those*.
In other words, the treatment differs, and I don't know why: if we get an unexpected structured reply, we skip it and "resync", whereas if we get an unexpected extended reply, we move to the DEAD state.
I think it's fine to skip replies that refer to unknown cookies, but *reply format* violations should be fatal.
I think this all stems from me trying to be as generous as possible with bad servers: if we get a wrong reply type per the protocol but it is a reply type we know how to handle, AND we haven't consumed too many bytes for the size of that reply, then we can tolerate the server's bug. If it is the wrong type, and we already read too many bytes for what the right type would be, we can't (easily) undo the fact that we read too much.
If we did not negotiate extended headers, we start by reading the length of a simple header; if the magic says it was instead an extended header, we can tolerate the server's mistake by reading the rest of that packet. But I definitely see your point about consolidating the decisions to be local to one state, rather than split across two files, and aiming for consistent handling.
Right, I found your note about commit b31e7bac in patch#6, and I found it hard to agree with the motivation behind it :) When reading patch#6, I immediately felt it was somehow related to my point (10) here, under patch#5. I couldn't figure out how exactly though, as the affected files were different!
I'm liking your idea earlier in the series about reworking the state engine to FULLY parse the header (whether simple, structured, or extended) in states-reply.c, and only then move to states-reply-structured.c if the header was structured or extended, rather than splitting the header parse across two files.
I don't want to generate unnecessary work for you, but you too might find future maintenance easier this way. From my perspective, I already have to ping-pong between these two C files; not easy. :)
Then, this is not a request for an update, just a comment: I don't understand why the spec introduced the "offset" field at all. It does not seem to carry additional information beyond the cookie. So we can certainly check that the response's offset matches the command's offset (the code looks OK), but the larger purpose is unclear. (And this is probably also the reason why you get away with corrupting "nbd_extended_reply.offset" at (8) -- the field is never again needed, beyond the sanity check performed here.)
The evolution of that field: in my first draft, I wanted power-of-2 sizing to the reply header (both request and reply having a header of exactly 32 bytes); this left dead space in the reply packet, which I originally tried to mandate to be zero-filled. But when looking at the types again, I noticed that if the 'offset' field occupies the same relative place in the request and reply struct, a server can rewrite the reply in the same memory as it had saved for the request (I don't know that I actually implemented it that way in qemu as server, but it is possible for some other server).
There's also NBD_REPLY_TYPE_OFFSET_DATA and NBD_REPLY_TYPE_OFFSET_HOLE (used in response to NBD_CMD_READ), which each return an offset field in order to handle the case where a single client read request was split across multiple server replies. When we first specified structured replies, the spec was ambiguous: if I issue NBD_CMD_READ(length=1k, offset=0) but the server replies with the pair OFFSET_DATA(length=512, offset=0) and OFFSET_HOLE(length=512, offset=512), it's obvious how to reconstruct the original buffer (read into the first 512 bytes, memset the second 512 bytes). But if I issue NBD_CMD_READ(length=1k, offset=512), should the server reply OFFSET_DATA(length=512, offset=512) and OFFSET_HOLE(length=512, offset=1k) (that is, the reply offsets are absolute to the overall image), or with OFFSET_DATA(length=512, offset=0) and OFFSET_HOLE(length=512, offset=512) (that is, the reply offsets are relative to the start of the 1k buffer of the read request)? The initial implementations of qemu and nbdkit disagreed, and the spec ended up settling in favor of the former (see nbd.git commit 587ba722).
Interesting: while reading your comparison, I immediately found the read request relative reply offsets far-fetched and cumbersome, and the absolute reply offsets "natural". :)
As a result, when receiving an absolute offset in a data reply, the client has to place the data into buffer + (reply_offset - request_offset) (that is, convert the server's absolute offset back into a buffer-relative offset), which requires knowing what the reqeust offset was; having the server return this offset in extended replies may make it easier to implement a client that doesn't have to store its request offset separately. Since libnbd also handles non-extended headers (and has to hang on to the request offset for those), you are right that we end up merely validating that the server's reply offset was expected, and then throwing it away as it didn't add anything further than a validation that we haven't lost sync with the server.
Thanks for clarifying! So a less "legacy compatible" NBD client could insist on extended headers (refuse working with servers not providing extended headers), and then such a client might need the "offset" field for real. (On a tangent: I wonder if it were *secure* to trust the server's "offset" field in the reply, for addressing client memory!)
@@ -168,7 +186,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ assert (length >= sizeof h->sbuf.reply.payload.error.error.error); assert (cmd);
@@ -207,7 +225,7 @@ REPLY.STRUCTURED_REPLY.RECV_ERROR_MESSAGE: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ msglen = be16toh (h->sbuf.reply.payload.error.error.len); type = be16toh (h->sbuf.reply.hdr.structured.type);
@@ -307,7 +325,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ offset = be64toh (h->sbuf.reply.payload.offset_data.offset);
assert (cmd); /* guaranteed by CHECK */ @@ -346,7 +364,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_DATA_DATA: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ offset = be64toh (h->sbuf.reply.payload.offset_data.offset);
assert (cmd); /* guaranteed by CHECK */ @@ -426,7 +444,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */
assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); @@ -460,7 +478,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: SET_NEXT_STATE (%.READY); return 0; case 0: - length = be32toh (h->sbuf.reply.hdr.structured.length); + length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */
assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS);
(11) This is all fallout from (8). The commit message does document it (in the first paragraph), but I don't understand where we *benefit* from stuffing "nbd_extended_reply.length" into "nbd_structured_reply.length".
The benefit is that all later code only has to deal with nbd_structured_reply (rather than adding yet more 'if (h->extended_reply)' in all subsequent states). Conversely, if I had widened instead of narrowed, then all later code would only need to doea with nbd_extended_reply.
Hmmm. That's a good point. I guess I didn't think of it because it would mean (for example) handling extended block descriptors in a payload that *seemingly* followed a structured (not extended) reply header. I've not yet gotten used to the idea that header and payload are going to be treated independently, after initial consistency checks.
Regarding downsides thereof, I can see two:
- as I mentioned, "nbd_extended_reply.offset" becomes unusable,
- "nbd_structured_reply.length" will no longer be big-endian but host-endian, which arguably does not match the other fields' endianness in the scratch space (= sbuf).
I feel this back-stuffing brings the stashed header into an inconsistent state that is specific to a subset of the automaton's states.
I'd rather introduce an entirely new field to "sbuf.reply" -- between "sbuf.reply.hdr" and "sbuf.reply.payload" --, if we really need to cache a host-endian length value, regardless of which kind of reply header we got.
You may be on to something. Leaving the wire reply ALWAYS in network endian order, and adding a few bytes to nbd_reply to stash a normalized host-endian decoded value, would certainly be less confusing than having to figure out "which parts of my wire reply are still network-endian vs. natively converted". It may make the overall structure a few bytes larger, but that's probably in the noise (since our nbd_handle already accomodatese maximum-length NBD strings of 4k, adding up to another 32 bytes for fully-normalized host-endian form isn't bad). I'll keep that in mind for v4 (this is not the only place that needs normalization; handling 32- vs. 64-bit block status replies also needs normalization; the tradeoff between minimal storage by normalizing in place vs. maintainability by normalizing into a copy has interesting implications).
Thanks! Laszlo
On Thu, Jun 01, 2023 at 08:00:55AM -0500, Eric Blake wrote:
@@ -83,13 +96,18 @@ REPLY.STRUCTURED_REPLY.CHECK: * not worth keeping the connection alive. */ if (length > MAX_REQUEST_SIZE + sizeof h->sbuf.reply.payload.offset_data) { - set_error (0, "invalid server reply length %" PRIu32, length); + set_error (0, "invalid server reply length %" PRIu64, length); SET_NEXT_STATE (%.DEAD); return 0; } + /* For convenience, we now normalize extended replies into compact, + * doable since we validated that payload length fits in 32 bits. + */ + h->sbuf.reply.hdr.structured.length = length;
(8) I'm very confused by this. For an extended reply, this will overwrite the "offset" field.
At one point, I considered normalizing in the opposite direction (take structured replies and widen them into the extended header form); it required touching more lines of code, so I didn't pursue it at the time. But I can see how compressing things down (intentionally throwing away information we will no longer reference) can be confusing without good comments, so at a minimum, I will be adding comments, if not outright going with the widening rather than narrowing approach.
In fact, doing it in place is bad. Later code in this function can trigger a transition to state RESYNC, which assumes sbuf.reply.hdr.structured.length still holds the wire value, not the normalized value. I'm fixing it up by creating h->payload_left, initialized in START to the endian-corrected wire value, and then decremented as various states peel off parts of the payload, so that FINISH can then assert all payload has been accounted for. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Thu, Jun 01, 2023 at 11:04:05AM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
@@ -69,11 +75,18 @@ REPLY.STRUCTURED_REPLY.RECV_REMAINING: REPLY.STRUCTURED_REPLY.CHECK: struct command *cmd = h->reply_cmd; uint16_t flags, type; - uint32_t length; + uint64_t length; + uint64_t offset = -1;
(6) I disagree with initializing the local variable "offset" here.
Below (in the rest of REPLY.STRUCTURED_REPLY.CHECK), we only read "offset" back if "extended_headers" is set. But if "extended_headers" is set, we also store a value to "offset", before the read.
Initializing "offset" to -1 suggests that the code might otherwise read an indeterminate value from "offset" -- but that's not the case.
You may find that the compiler will give a warning. It's usually not good about dealing with the case where a variable being initialized + used depends on another variable being true. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On 6/7/23 16:55, Richard W.M. Jones wrote:
On Thu, Jun 01, 2023 at 11:04:05AM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
@@ -69,11 +75,18 @@ REPLY.STRUCTURED_REPLY.RECV_REMAINING: REPLY.STRUCTURED_REPLY.CHECK: struct command *cmd = h->reply_cmd; uint16_t flags, type; - uint32_t length; + uint64_t length; + uint64_t offset = -1;
(6) I disagree with initializing the local variable "offset" here.
Below (in the rest of REPLY.STRUCTURED_REPLY.CHECK), we only read "offset" back if "extended_headers" is set. But if "extended_headers" is set, we also store a value to "offset", before the read.
Initializing "offset" to -1 suggests that the code might otherwise read an indeterminate value from "offset" -- but that's not the case.
You may find that the compiler will give a warning. It's usually not good about dealing with the case where a variable being initialized + used depends on another variable being true.
Good point; that reminds me we used to encounter that issue specifically in IA32 edk2 builds, but only when using old gcc (RHEL-7 era, gcc-4.8.5). Of course it might still happen today. If that's the case, can we comment the code that we initialize "offset" for shutting up the compiler? Thanks! Laszlo
One of the benefits of extended replies is that we can do a fixed-length read for the entire header of every server reply, which is fewer syscalls than the split-read approach required by structured replies. But one of the drawbacks of doing a large read is that if the server is non-compliant (not a problem for normal servers, but something I hit rather more than I'd like to admit while developing extended header support in servers), nbd_pwrite() and friends will deadlock if the server replies with the wrong header. Add in some code to catch that failure mode and move the state machine to DEAD sooner, to make it easier to diagnose the fault in the server. Unlike in the case of an unexpected simply reply from a structured server (where we never over-read, and can therefore commit b31e7bac can merely fail the command with EPROTO and successfully move on to the next server reply), in this case we really do have to move to DEAD: in addition to having already read the 16 or 20 bytes that the server sent in its (short) reply for this command, we may have already read the initial bytes of the server's next reply, but we have no way to push those extra bytes back onto our read stream for parsing on our next pass through the state machine. Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/states-reply.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/generator/states-reply.c b/generator/states-reply.c index 4e9f2dde..d4710d91 100644 --- a/generator/states-reply.c +++ b/generator/states-reply.c @@ -109,7 +109,28 @@ REPLY.START: REPLY.RECV_REPLY: switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; - case 1: SET_NEXT_STATE (%.READY); return 0; + case 1: SET_NEXT_STATE (%.READY); + /* Special case: if we have a short read, but got at least far + * enough to decode the magic number, we can check if the server + * is matching our expectations. This lets us avoid deadlocking if + * a buggy server sends only 16 bytes of a simple reply, and is + * waiting for our next command, while we are blocked waiting for + * the server to send 32 bytes of an extended reply. + */ + if (h->extended_headers && + (char *)h->rbuf >= (char *)&h->sbuf.reply.hdr.extended.flags) { + uint32_t magic = be32toh (h->sbuf.reply.hdr.extended.magic); + if (magic != NBD_EXTENDED_REPLY_MAGIC) { + SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); +#if 0 /* uncomment to see desynchronized data */ + nbd_internal_hexdump (&h->sbuf.reply.hdr.extended.flags, + sizeof (h->sbuf.reply.hdr.extended.flags), + stderr); +#endif + } + } + return 0; case 0: SET_NEXT_STATE (%CHECK_SIMPLE_OR_STRUCTURED_REPLY); } return 0; -- 2.40.1
On 5/25/23 15:00, Eric Blake wrote:
One of the benefits of extended replies is that we can do a fixed-length read for the entire header of every server reply, which is fewer syscalls than the split-read approach required by structured replies.
(Totally tangential comment: recvmsg() could scatter the incoming traffic into non-contiguous fields of a non-packed structure. But I don't know if TLS has anything similar. The current "linear receive" approach is probably the least demanding of the underlying socket abstractions. At the same time it requires us to use packed structs, and/or multiple syscalls.)
But one of the drawbacks of doing a large read is that if the server is non-compliant (not a problem for normal servers, but something I hit rather more than I'd like to admit while developing extended header support in servers),
Haha, so this is where it's coming from! :) I can totally relate.
nbd_pwrite() and friends will deadlock if the server replies with the wrong header. Add in some code to catch that failure mode and move the state machine to DEAD sooner, to make it easier to diagnose the fault in the server.
Unlike in the case of an unexpected simply reply from a structured
(1) s/simply/simple/
server (where we never over-read, and can therefore
(2) At this point my English parser gets thrown off:
commit b31e7bac can merely fail the command with EPROTO and successfully move on to the next server reply),
resync here:
in this case we really do have to move to DEAD: in addition to having already read the 16 or 20 bytes that the server sent in its (short) reply for this command, we may have already read the initial bytes of the server's next reply, but we have no way to push those extra bytes back onto our read stream for parsing on our next pass through the state machine.
Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/states-reply.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/generator/states-reply.c b/generator/states-reply.c index 4e9f2dde..d4710d91 100644 --- a/generator/states-reply.c +++ b/generator/states-reply.c @@ -109,7 +109,28 @@ REPLY.START: REPLY.RECV_REPLY: switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; - case 1: SET_NEXT_STATE (%.READY); return 0; + case 1: SET_NEXT_STATE (%.READY); + /* Special case: if we have a short read, but got at least far + * enough to decode the magic number, we can check if the server + * is matching our expectations. This lets us avoid deadlocking if + * a buggy server sends only 16 bytes of a simple reply, and is + * waiting for our next command, while we are blocked waiting for + * the server to send 32 bytes of an extended reply. + */
(4) Slight inconsistency between commit message and code comment: the former mentions "20 bytes", but the latter doesn't mention "structured reply".
+ if (h->extended_headers && + (char *)h->rbuf >= (char *)&h->sbuf.reply.hdr.extended.flags) {
(.) I wonder if (address-of-magic + size-of magic) might be more readable / greppable. Just in case. Feel free to ignore.
+ uint32_t magic = be32toh (h->sbuf.reply.hdr.extended.magic); + if (magic != NBD_EXTENDED_REPLY_MAGIC) { + SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); +#if 0 /* uncomment to see desynchronized data */ + nbd_internal_hexdump (&h->sbuf.reply.hdr.extended.flags, + sizeof (h->sbuf.reply.hdr.extended.flags), + stderr); +#endif + } + } + return 0;
(5) This could be factored out to a helper function, to share the "invalid:" label logic with the previous patch. (6) Commencing a dump from "flags" onward looks questionable. At this point, the "flags" field need not to be complete, or even started -- all we know is that the "magic" field *before" "flags" is complete. I think we should dump "h->sbuf.reply.hdr.simple", like in patch#5. (.) Side comment (so no bullet number assigned): because we can take multiple iterations on RECV_REPLY, we may end up checking the "magic" field multiple times. Not very costly, just something to be aware of. (7) Assume that we have a short read first, but then complete REPLY.RECV_REPLY successfully, and move to REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY. Then, the following condition will have been caught (excluded) under RECV_REPLY: (extended_headers && magic != NBD_EXTENDED_REPLY_MAGIC) [1] Consequently, the same condition will never hold in REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY. Now consider REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY, from the previous patch: magic = be32toh (h->sbuf.reply.hdr.simple.magic); switch (magic) { case NBD_SIMPLE_REPLY_MAGIC: if (h->extended_headers) goto invalid; This branch becomes dead code. The condition h->extended_headers && magic == NBD_SIMPLE_REPLY_MAGIC would satisfy [1], therefore it can't happen here. Further: SET_NEXT_STATE (%SIMPLE_REPLY.START); break; case NBD_STRUCTURED_REPLY_MAGIC: case NBD_EXTENDED_REPLY_MAGIC: if ((magic == NBD_STRUCTURED_REPLY_MAGIC) == h->extended_headers) goto invalid; While this lovely condition is satisfied by "true == true", that is, by: h->extended_headers && magic == NBD_STRUCTURED_REPLY_MAGIC [2] the same condition [2] satisfies [1] too, and therefore it can never occur. The only way for satisfying this (magic == NBD_STRUCTURED_REPLY_MAGIC) == h->extended_headers condition would be the "false == false" case, that is: !h->extended_headers && magic == NBD_EXTENDED_REPLY_MAGIC [3] (note that (magic != NBD_STRUCTURED_REPLY_MAGIC) is equivalent to (magic == NBD_EXTENDED_REPLY_MAGIC) due to how the switch statement is structured,) and then we should use that expression, i.e., [3], because it is more readable than the double-equality (i.e., equivalence) form. *HOWEVER*! :) If we do *not* see some short-but-together-long-enough reads first, but see a full read at once, then our "pre-check" in RECV_REPLY does not get activated at all! And then all the conditions in CHECK_SIMPLE_OR_STRUCTURED_REPLY remain necessary. So this is my long-winded way to ask that:
case 0: SET_NEXT_STATE (%CHECK_SIMPLE_OR_STRUCTURED_REPLY); } return 0;
you please consider adding a comment here, saying that, while the short read special case *does* short-circuit a number of checks under CHECK_SIMPLE_OR_STRUCTURED_REPLY, all the checks in CHECK_SIMPLE_OR_STRUCTURED_REPLY remain required if under RECV_REPLY we get a full read immediately. Thanks! Laszlo
On Thu, Jun 01, 2023 at 03:42:30PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
One of the benefits of extended replies is that we can do a fixed-length read for the entire header of every server reply, which is fewer syscalls than the split-read approach required by structured replies.
(Totally tangential comment: recvmsg() could scatter the incoming traffic into non-contiguous fields of a non-packed structure. But I don't know if TLS has anything similar. The current "linear receive" approach is probably the least demanding of the underlying socket abstractions. At the same time it requires us to use packed structs, and/or multiple syscalls.)
But one of the drawbacks of doing a large read is that if the server is non-compliant (not a problem for normal servers, but something I hit rather more than I'd like to admit while developing extended header support in servers),
Haha, so this is where it's coming from! :) I can totally relate.
nbd_pwrite() and friends will deadlock if the server replies with the wrong header. Add in some code to catch that failure mode and move the state machine to DEAD sooner, to make it easier to diagnose the fault in the server.
Unlike in the case of an unexpected simply reply from a structured
(1) s/simply/simple/
Yep.
server (where we never over-read, and can therefore
(2) At this point my English parser gets thrown off:
commit b31e7bac can merely fail the command with EPROTO and successfully move on to the next server reply),
Fixing the parenthetical as follows: (where, since commit b31e7bac, we can merely fail the command with EPROTO and successfully move on to the next server reply, because we didn't read too many bytes yet)
resync here:
in this case we really do have to move to DEAD: in addition to having already read the 16 or 20 bytes that the server sent in its (short) reply for this command, we may have already read the initial bytes of the server's next reply, but we have no way to push those extra bytes back onto our read stream for parsing on our next pass through the state machine.
Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/states-reply.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/generator/states-reply.c b/generator/states-reply.c index 4e9f2dde..d4710d91 100644 --- a/generator/states-reply.c +++ b/generator/states-reply.c @@ -109,7 +109,28 @@ REPLY.START: REPLY.RECV_REPLY: switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; - case 1: SET_NEXT_STATE (%.READY); return 0; + case 1: SET_NEXT_STATE (%.READY); + /* Special case: if we have a short read, but got at least far + * enough to decode the magic number, we can check if the server + * is matching our expectations. This lets us avoid deadlocking if + * a buggy server sends only 16 bytes of a simple reply, and is + * waiting for our next command, while we are blocked waiting for + * the server to send 32 bytes of an extended reply. + */
(4) Slight inconsistency between commit message and code comment: the former mentions "20 bytes", but the latter doesn't mention "structured reply".
Did I miss (3)? But yes, I can improve this comment.
+ if (h->extended_headers && + (char *)h->rbuf >= (char *)&h->sbuf.reply.hdr.extended.flags) {
(.) I wonder if (address-of-magic + size-of magic) might be more readable / greppable. Just in case.
Feel free to ignore.
Actually, I agree that it is nicer (although a bit longer).
+ uint32_t magic = be32toh (h->sbuf.reply.hdr.extended.magic); + if (magic != NBD_EXTENDED_REPLY_MAGIC) { + SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); +#if 0 /* uncomment to see desynchronized data */ + nbd_internal_hexdump (&h->sbuf.reply.hdr.extended.flags, + sizeof (h->sbuf.reply.hdr.extended.flags), + stderr); +#endif + } + } + return 0;
(5) This could be factored out to a helper function, to share the "invalid:" label logic with the previous patch.
(6) Commencing a dump from "flags" onward looks questionable. At this point, the "flags" field need not to be complete, or even started -- all we know is that the "magic" field *before" "flags" is complete.
I think we should dump "h->sbuf.reply.hdr.simple", like in patch#5.
Yep, I noticed that while addressing your point (4), just before reading your point (6). Bug dumping the full hdr.simple is also too much; really, given the preconditions above, all we can dump is hdr.magic. At which point, the hexdump is not providing us any further information than what the set_error() call already output. I'm just deleting the #if 0 segment, which in turn eliminates the need to address point (5), as the two invalid: labels are no longer identical.
(.) Side comment (so no bullet number assigned): because we can take multiple iterations on RECV_REPLY, we may end up checking the "magic" field multiple times. Not very costly, just something to be aware of.
Indeed. In the common case, short reads are rare, and even when they do happen, it is even more rare to hit it more than once per packet. I wonder if nc or similar has a way to force a server's response to be flushed after every byte, to see the performance impact of maximum network overhead. But your observation means you are correctly aware of a larger design aspect of the state machine: all code handling short reads (case 1 of recv_into_rbuf) must be re-startable (no messing with h->rbuf, for example). Stateful changes can only occur when the read is complete (case 0) or irretrievably failed (case -1).
(7) Assume that we have a short read first, but then complete REPLY.RECV_REPLY successfully, and move to REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY.
...
*HOWEVER*! :) If we do *not* see some short-but-together-long-enough reads first, but see a full read at once, then our "pre-check" in RECV_REPLY does not get activated at all! And then all the conditions in CHECK_SIMPLE_OR_STRUCTURED_REPLY remain necessary.
So this is my long-winded way to ask that:
case 0: SET_NEXT_STATE (%CHECK_SIMPLE_OR_STRUCTURED_REPLY); } return 0;
you please consider adding a comment here, saying that, while the short read special case *does* short-circuit a number of checks under CHECK_SIMPLE_OR_STRUCTURED_REPLY, all the checks in CHECK_SIMPLE_OR_STRUCTURED_REPLY remain required if under RECV_REPLY we get a full read immediately.
Good idea. Added for v4. -- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
On 7/21/23 14:50, Eric Blake wrote:
On Thu, Jun 01, 2023 at 03:42:30PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
One of the benefits of extended replies is that we can do a fixed-length read for the entire header of every server reply, which is fewer syscalls than the split-read approach required by structured replies.
(Totally tangential comment: recvmsg() could scatter the incoming traffic into non-contiguous fields of a non-packed structure. But I don't know if TLS has anything similar. The current "linear receive" approach is probably the least demanding of the underlying socket abstractions. At the same time it requires us to use packed structs, and/or multiple syscalls.)
But one of the drawbacks of doing a large read is that if the server is non-compliant (not a problem for normal servers, but something I hit rather more than I'd like to admit while developing extended header support in servers),
Haha, so this is where it's coming from! :) I can totally relate.
nbd_pwrite() and friends will deadlock if the server replies with the wrong header. Add in some code to catch that failure mode and move the state machine to DEAD sooner, to make it easier to diagnose the fault in the server.
Unlike in the case of an unexpected simply reply from a structured
(1) s/simply/simple/
Yep.
server (where we never over-read, and can therefore
(2) At this point my English parser gets thrown off:
commit b31e7bac can merely fail the command with EPROTO and successfully move on to the next server reply),
Fixing the parenthetical as follows:
(where, since commit b31e7bac, we can merely fail the command with EPROTO and successfully move on to the next server reply, because we didn't read too many bytes yet)
resync here:
in this case we really do have to move to DEAD: in addition to having already read the 16 or 20 bytes that the server sent in its (short) reply for this command, we may have already read the initial bytes of the server's next reply, but we have no way to push those extra bytes back onto our read stream for parsing on our next pass through the state machine.
Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/states-reply.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/generator/states-reply.c b/generator/states-reply.c index 4e9f2dde..d4710d91 100644 --- a/generator/states-reply.c +++ b/generator/states-reply.c @@ -109,7 +109,28 @@ REPLY.START: REPLY.RECV_REPLY: switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; - case 1: SET_NEXT_STATE (%.READY); return 0; + case 1: SET_NEXT_STATE (%.READY); + /* Special case: if we have a short read, but got at least far + * enough to decode the magic number, we can check if the server + * is matching our expectations. This lets us avoid deadlocking if + * a buggy server sends only 16 bytes of a simple reply, and is + * waiting for our next command, while we are blocked waiting for + * the server to send 32 bytes of an extended reply. + */
(4) Slight inconsistency between commit message and code comment: the former mentions "20 bytes", but the latter doesn't mention "structured reply".
Did I miss (3)? But yes, I can improve this comment.
My mistake, probably due to over-editing! :) Thanks Laszlo
+ if (h->extended_headers && + (char *)h->rbuf >= (char *)&h->sbuf.reply.hdr.extended.flags) {
(.) I wonder if (address-of-magic + size-of magic) might be more readable / greppable. Just in case.
Feel free to ignore.
Actually, I agree that it is nicer (although a bit longer).
+ uint32_t magic = be32toh (h->sbuf.reply.hdr.extended.magic); + if (magic != NBD_EXTENDED_REPLY_MAGIC) { + SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */ + set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic); +#if 0 /* uncomment to see desynchronized data */ + nbd_internal_hexdump (&h->sbuf.reply.hdr.extended.flags, + sizeof (h->sbuf.reply.hdr.extended.flags), + stderr); +#endif + } + } + return 0;
(5) This could be factored out to a helper function, to share the "invalid:" label logic with the previous patch.
(6) Commencing a dump from "flags" onward looks questionable. At this point, the "flags" field need not to be complete, or even started -- all we know is that the "magic" field *before" "flags" is complete.
I think we should dump "h->sbuf.reply.hdr.simple", like in patch#5.
Yep, I noticed that while addressing your point (4), just before reading your point (6). Bug dumping the full hdr.simple is also too much; really, given the preconditions above, all we can dump is hdr.magic. At which point, the hexdump is not providing us any further information than what the set_error() call already output. I'm just deleting the #if 0 segment, which in turn eliminates the need to address point (5), as the two invalid: labels are no longer identical.
(.) Side comment (so no bullet number assigned): because we can take multiple iterations on RECV_REPLY, we may end up checking the "magic" field multiple times. Not very costly, just something to be aware of.
Indeed. In the common case, short reads are rare, and even when they do happen, it is even more rare to hit it more than once per packet. I wonder if nc or similar has a way to force a server's response to be flushed after every byte, to see the performance impact of maximum network overhead.
But your observation means you are correctly aware of a larger design aspect of the state machine: all code handling short reads (case 1 of recv_into_rbuf) must be re-startable (no messing with h->rbuf, for example). Stateful changes can only occur when the read is complete (case 0) or irretrievably failed (case -1).
(7) Assume that we have a short read first, but then complete REPLY.RECV_REPLY successfully, and move to REPLY.CHECK_SIMPLE_OR_STRUCTURED_REPLY.
...
*HOWEVER*! :) If we do *not* see some short-but-together-long-enough reads first, but see a full read at once, then our "pre-check" in RECV_REPLY does not get activated at all! And then all the conditions in CHECK_SIMPLE_OR_STRUCTURED_REPLY remain necessary.
So this is my long-winded way to ask that:
case 0: SET_NEXT_STATE (%CHECK_SIMPLE_OR_STRUCTURED_REPLY); } return 0;
you please consider adding a comment here, saying that, while the short read special case *does* short-circuit a number of checks under CHECK_SIMPLE_OR_STRUCTURED_REPLY, all the checks in CHECK_SIMPLE_OR_STRUCTURED_REPLY remain required if under RECV_REPLY we get a full read immediately.
Good idea. Added for v4.
The existing nbd_block_status() callback is permanently stuck with an array of uint32_t pairs (len/2 extents), which is both constrained on maximum extent size (no larger than 4G) and on the status flags (must fit in 32 bits). While the "base:allocation" metacontext will never exceed 32 bits, it is not hard to envision other extension metacontexts where a 64-bit status would be useful (for example, Zoned Block Devices expressing a 64-bit offset[1]). Exposing 64-bit extents will require a new API; we now have the decision of whether to copy the existing API practice of returning a bare array containing len/2 extent pairs, or with a saner idea of an array of structs with len extents. Returning an array of structs is actually easier to map to various language bindings, particularly since we know the length field will generally fit in 63-bits (fallout from the fact that NBD exports are effectively capped in size by signed off_t), but where the status field must represent a full 64 bits (assuming that the user wants to connect to a metadata extension that utilizes 64 bits, rather than existing metadata contexts that only expose 32 bits). This patch introduces the struct we plan to use in the new API, along with language bindings. The bindings for Python and OCaml were relatively straightforward; the Golang bindings took a bit more effort for me to write. Temporary unused attributes are needed to keep the compiler happy until a later patch exposes a new API using the new callback type. Note that 'struct nbd_block_descriptor_ext' in lib/nbd-protocol.h is exactly the same as what we want to use in C. But it is easier to stick a new public type in <libnbd.h> than to figure out how to expose just part of a header we only want to use internally. [1] https://zonedstorage.io/docs/linux/zbd-api Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/API.mli | 1 + generator/API.ml | 12 +++++++++++- generator/C.ml | 24 +++++++++++++++++++++--- generator/GoLang.ml | 24 ++++++++++++++++++++++++ generator/OCaml.ml | 24 +++++++++++++++++++++--- generator/Python.ml | 21 ++++++++++++++++++++- ocaml/helpers.c | 20 ++++++++++++++++++++ ocaml/nbd-c.h | 1 + golang/handle.go | 6 ++++++ 9 files changed, 125 insertions(+), 8 deletions(-) diff --git a/generator/API.mli b/generator/API.mli index c5bba8c2..96626c9b 100644 --- a/generator/API.mli +++ b/generator/API.mli @@ -52,6 +52,7 @@ and | BytesPersistOut of string * string | Closure of closure (** function pointer + void *opaque *) | Enum of string * enum (** enum/union type, int in C *) +| Extent64 of string (** extent descriptor, struct nbd_extent in C *) | Fd of string (** file descriptor *) | Flags of string * flags (** flags, uint32_t in C *) | Int of string (** small int *) diff --git a/generator/API.ml b/generator/API.ml index 02c1260d..1da1ec53 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -42,6 +42,7 @@ | BytesPersistOut of string * string | Closure of closure | Enum of string * enum +| Extent64 of string | Fd of string | Flags of string * flags | Int of string @@ -157,6 +158,14 @@ let extent_closure = "nr_entries"); CBMutable (Int "error") ] } +let extent64_closure = { + cbname = "extent64"; + cbargs = [ CBString "metacontext"; + CBUInt64 "offset"; + CBArrayAndLen (Extent64 "entries", + "nr_entries"); + CBMutable (Int "error") ] +} let list_closure = { cbname = "list"; cbargs = [ CBString "name"; CBString "description" ] @@ -166,7 +175,8 @@ let context_closure = cbargs = [ CBString "name" ] } let all_closures = [ chunk_closure; completion_closure; - debug_closure; extent_closure; list_closure; + debug_closure; extent_closure; extent64_closure; + list_closure; context_closure ] (* Enums. *) diff --git a/generator/C.ml b/generator/C.ml index f772117c..d8472153 100644 --- a/generator/C.ml +++ b/generator/C.ml @@ -94,6 +94,7 @@ let | Closure { cbname } -> [ sprintf "%s_callback" cbname; sprintf "%s_user_data" cbname ] | Enum (n, _) -> [n] + | Extent64 n -> [n] | Fd n -> [n] | Flags (n, _) -> [n] | Int n -> [n] @@ -129,7 +130,7 @@ let (* list of strings should be marked as non-null *) | StringList n -> [ true ] (* numeric and other non-pointer types are not able to be null *) - | Bool _ | Closure _ | Enum _ | Fd _ | Flags _ + | Bool _ | Closure _ | Enum _ | Extent64 _ | Fd _ | Flags _ | Int _ | Int64 _ | SizeT _ | UInt _ | UInt32 _ | UInt64 _ | UIntPtr _ -> [ false ] @@ -191,6 +192,9 @@ and | Enum (n, _) -> if types then pr "int "; pr "%s" n + | Extent64 n -> + if types then pr "nbd_extent "; + pr "%s" n | Flags (n, _) -> if types then pr "uint32_t "; pr "%s" n @@ -305,6 +309,11 @@ and pr "%s, " n; if types then pr "size_t "; pr "%s" len + | CBArrayAndLen (Extent64 n, len) -> + if types then pr "nbd_extent *"; + pr "%s, " n; + if types then pr "size_t "; + pr "%s" len | CBArrayAndLen _ -> assert false | CBBytesIn (n, len) -> if types then pr "const void *"; @@ -477,6 +486,13 @@ let pr "extern int nbd_get_errno (void);\n"; pr "#define LIBNBD_HAVE_NBD_GET_ERRNO 1\n"; pr "\n"; + pr "/* This is used in the callback for nbd_block_status_64.\n"; + pr " */\n"; + pr "typedef struct {\n"; + pr " uint64_t length;\n"; + pr " uint64_t flags;\n"; + pr "} nbd_extent;\n"; + pr "\n"; print_closure_structs (); List.iter ( fun (name, { args; optargs; ret }) -> @@ -740,7 +756,7 @@ let pr " char *%s_printable =\n" n; pr " nbd_internal_printable_string_list (%s);\n" n | BytesOut _ | BytesPersistOut _ - | Bool _ | Closure _ | Enum _ | Flags _ | Fd _ | Int _ + | Bool _ | Closure _ | Enum _ | Extent64 _ | Flags _ | Fd _ | Int _ | Int64 _ | SizeT _ | SockAddrAndLen _ | UInt _ | UInt32 _ | UInt64 _ | UIntPtr _ -> () ) args; @@ -766,6 +782,7 @@ let pr " %s=\\\"%%s\\\" %s=%%zu" n count | Closure { cbname } -> pr " %s=%%s" cbname | Enum (n, _) -> pr " %s=%%d" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> pr " %s=0x%%x" n | Fd n | Int n -> pr " %s=%%d" n | Int64 n -> pr " %s=%%\"PRIi64\"" n @@ -804,6 +821,7 @@ let pr "%s_printable ? %s_printable : \"\", %s" n n count | Closure _ -> pr "\"<fun>\"" | Enum (n, _) -> pr "%s" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> pr "%s" n | Fd n | Int n | Int64 n | SizeT n -> pr "%s" n | SockAddrAndLen (_, len) -> pr "(int) %s" len @@ -836,7 +854,7 @@ let | StringList n -> pr " free (%s_printable);\n" n | BytesOut _ | BytesPersistOut _ - | Bool _ | Closure _ | Enum _ | Flags _ | Fd _ | Int _ + | Bool _ | Closure _ | Enum _ | Extent64 _ | Flags _ | Fd _ | Int _ | Int64 _ | SizeT _ | SockAddrAndLen _ | UInt _ | UInt32 _ | UInt64 _ | UIntPtr _ -> () ) args; diff --git a/generator/GoLang.ml b/generator/GoLang.ml index 4ab6b262..8922812b 100644 --- a/generator/GoLang.ml +++ b/generator/GoLang.ml @@ -49,6 +49,7 @@ let | BytesPersistOut (n, len) -> n | Closure { cbname } -> cbname | Enum (n, _) -> n + | Extent64 n -> n | Fd n -> n | Flags (n, _) -> n | Int n -> n @@ -71,6 +72,7 @@ let | BytesPersistOut _ -> "AioBuffer" | Closure { cbname } -> sprintf "%sCallback" (camel_case cbname) | Enum (_, { enum_prefix }) -> camel_case enum_prefix + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd _ -> "int" | Flags (_, { flag_prefix }) -> camel_case flag_prefix | Int _ -> "int" @@ -262,6 +264,7 @@ let pr " c_%s.user_data = C.alloc_cbid(C.long(%s_cbid))\n" cbname cbname | Enum (n, _) -> pr " c_%s := C.int (%s)\n" n n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd n -> pr " c_%s := C.int (%s)\n" n n | Flags (n, _) -> @@ -334,6 +337,7 @@ let | BytesPersistOut (n, len) -> pr ", c_%s, c_%s" n len | Closure { cbname } -> pr ", c_%s" cbname | Enum (n, _) -> pr ", c_%s" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd n -> pr ", c_%s" n | Flags (n, _) -> pr ", c_%s" n | Int n -> pr ", c_%s" n @@ -524,6 +528,18 @@ let copy(ret, s) return ret } + +func copy_extent_array (entries *C.nbd_extent, count C.size_t) []LibnbdExtent { + unsafePtr := unsafe.Pointer(entries) + arrayPtr := (*[1 << 20]C.nbd_extent)(unsafePtr) + slice := arrayPtr[:count:count] + ret := make([]LibnbdExtent, count) + for i := 0; i < int (count); i++ { + ret[i].Length = uint64 (slice[i].length) + ret[i].Flags = uint64 (slice[i].flags) + } + return ret +} "; List.iter ( @@ -537,6 +553,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, _) -> pr "%s []uint32" n; + | CBArrayAndLen (Extent64 n, _) -> + pr "%s []LibnbdExtent" n; | CBBytesIn (n, len) -> pr "%s []byte" n; | CBInt n -> @@ -563,6 +581,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "%s *C.uint32_t, %s C.size_t" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "%s *C.nbd_extent, %s C.size_t" n count | CBBytesIn (n, len) -> pr "%s unsafe.Pointer, %s C.size_t" n len | CBInt n -> @@ -605,6 +625,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "copy_uint32_array (%s, %s)" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "copy_extent_array (%s, %s)" n count | CBBytesIn (n, len) -> pr "C.GoBytes (%s, C.int (%s))" n len | CBInt n -> @@ -756,6 +778,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "uint32_t *%s, size_t %s" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "nbd_extent *%s, size_t %s" n count | CBBytesIn (n, len) -> pr "void *%s, size_t %s" n len | CBInt n -> diff --git a/generator/OCaml.ml b/generator/OCaml.ml index edb81f25..ea3b044a 100644 --- a/generator/OCaml.ml +++ b/generator/OCaml.ml @@ -44,6 +44,7 @@ and | Closure { cbargs } -> sprintf "(%s)" (ocaml_closuredecl_to_string cbargs) | Enum (_, { enum_prefix }) -> sprintf "%s.t" enum_prefix + | Extent64 _ -> "extent" | Fd _ -> "Unix.file_descr" | Flags (_, { flag_prefix }) -> sprintf "%s.t list" flag_prefix | Int _ -> "int" @@ -102,6 +103,7 @@ let | BytesPersistOut (n, len) -> n | Closure { cbname } -> cbname | Enum (n, _) -> n + | Extent64 n -> n | Fd n -> n | Flags (n, _) -> n | Int n -> n @@ -147,6 +149,9 @@ let type cookie = int64 +type extent = int64 * int64 +(** Length and flags of an extent in block_status_64 callback. *) + "; List.iter ( @@ -268,6 +273,7 @@ let exception Error of string * Unix.error option exception Closed of string type cookie = int64 +type extent = int64 * int64 (* Give the exceptions names so that they can be raised from the C code. *) let () = @@ -498,7 +504,8 @@ let let argnames = List.map ( function - | CBArrayAndLen (UInt32 n, _) | CBBytesIn (n, _) + | CBArrayAndLen (UInt32 n, _) | CBArrayAndLen (Extent64 n, _) + | CBBytesIn (n, _) | CBInt n | CBInt64 n | CBMutable (Int n) | CBString n | CBUInt n | CBUInt64 n -> n ^ "v" @@ -531,6 +538,14 @@ let pr "%s %s,\n" indent n; pr "%s %s\n" indent count; pr "%s);\n" indent + | CBArrayAndLen (Extent64 n, count) -> + pr " %sv = " n; + let fncol = output_column () in + let indent = spaces fncol in + pr "nbd_internal_ocaml_alloc_extent64_array (\n"; + pr "%s %s,\n" indent n; + pr "%s %s\n" indent count; + pr "%s);\n" indent | CBBytesIn (n, len) -> pr " %sv = caml_alloc_initialized_string (%s, %s);\n" n len n | CBInt n | CBUInt n -> @@ -554,7 +569,7 @@ let List.iter ( function - | CBArrayAndLen (UInt32 _, _) + | CBArrayAndLen (_, _) | CBBytesIn _ | CBInt _ | CBInt64 _ @@ -563,7 +578,7 @@ let | CBUInt64 _ -> () | CBMutable (Int n) -> pr " *%s = Int_val (Field (%sv, 0));\n" n n - | CBArrayAndLen _ | CBMutable _ -> assert false + | CBMutable _ -> assert false ) cbargs; pr " if (Is_exception_result (rv)) {\n"; @@ -578,6 +593,7 @@ let pr "}\n"; pr "\n"; pr "static int\n"; + pr "__attribute__ ((unused)) /* XXX temporary hack */\n"; pr "%s_wrapper " cbname; C.print_cbarg_list ~wrap:true cbargs; pr "\n"; @@ -695,6 +711,7 @@ let pr " %s_callback.free = free_user_data;\n" cbname | Enum (n, { enum_prefix }) -> pr " int %s = %s_val (%sv);\n" n enum_prefix n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd n -> pr " /* OCaml Unix.file_descr is just an int, at least on Unix. */\n"; pr " int %s = Int_val (%sv);\n" n n @@ -780,6 +797,7 @@ let | BytesPersistOut _ | Closure _ | Enum _ + | Extent64 _ | Fd _ | Flags _ | Int _ diff --git a/generator/Python.ml b/generator/Python.ml index c81878de..ccac5fed 100644 --- a/generator/Python.ml +++ b/generator/Python.ml @@ -158,6 +158,7 @@ let let print_python_closure_wrapper { cbname; cbargs } = pr "/* Wrapper for %s callback. */\n" cbname; pr "static int\n"; + pr "__attribute__ ((unused)) /* XXX temporary hack */\n"; pr "%s_wrapper " cbname; C.print_cbarg_list ~wrap:true cbargs; pr "\n"; @@ -170,6 +171,7 @@ let List.iter ( function | CBArrayAndLen (UInt32 n, _) + | CBArrayAndLen (Extent64 n, _) | CBBytesIn (n, _) | CBMutable (Int n) -> pr " PyObject *py_%s = NULL;\n" n @@ -187,6 +189,16 @@ let pr " if (!py_e_%s) { PyErr_PrintEx (0); goto out; }\n" n; pr " PyList_SET_ITEM (py_%s, i_%s, py_e_%s);\n" n n n; pr " }\n" + | CBArrayAndLen (Extent64 n, len) -> + pr " py_%s = PyList_New (%s);\n" n len; + pr " size_t i_%s;\n" n; + pr " for (i_%s = 0; i_%s < %s; ++i_%s) {\n" n n len n; + pr " PyObject *py_e_%s = Py_BuildValue (\"OO\",\n" n; + pr " PyLong_FromUnsignedLong (%s[i_%s].length),\n" n n; + pr " PyLong_FromUnsignedLong (%s[i_%s].flags));\n" n n; + pr " if (!py_e_%s) { PyErr_PrintEx (0); goto out; }\n" n; + pr " PyList_SET_ITEM (py_%s, i_%s, py_e_%s);\n" n n n; + pr " }\n" | CBBytesIn (n, len) -> pr " py_%s = nbd_internal_py_get_subview (data->view, %s, %s);\n" n n len; pr " if (!py_%s) { PyErr_PrintEx (0); goto out; }\n" n @@ -206,6 +218,7 @@ let List.map ( function | CBArrayAndLen (UInt32 n, _) -> "O", sprintf "py_%s" n + | CBArrayAndLen (Extent64 n, _) -> "O", sprintf "py_%s" n | CBBytesIn (n, _) -> "O", sprintf "py_%s" n | CBInt n -> "i", n | CBInt64 n -> "L", n @@ -254,7 +267,8 @@ let pr " out:\n"; List.iter ( function - | CBArrayAndLen (UInt32 n, _) -> + | CBArrayAndLen (UInt32 n, _) + | CBArrayAndLen (Extent64 n, _) -> pr " Py_XDECREF (py_%s);\n" n | CBBytesIn (n, _) -> pr " Py_XDECREF (py_%s);\n" n @@ -307,6 +321,7 @@ let pr ".callback = %s_wrapper, .free = free_user_data" cbname); pr " };\n" | Enum (n, _) -> pr " int %s;\n" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> pr " uint32_t %s_u32;\n" n; pr " unsigned int %s; /* really uint32_t */\n" n @@ -363,6 +378,7 @@ let "O", sprintf "&%s" n, sprintf "py_%s->buf, py_%s->len" n n | Closure { cbname } -> "O", sprintf "&py_%s_fn" cbname, cbname | Enum (n, _) -> "i", sprintf "&%s" n, n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> "I", sprintf "&%s" n, sprintf "%s_u32" n | Fd n | Int n -> "i", sprintf "&%s" n, n | Int64 n -> "L", sprintf "&%s" n, sprintf "%s_i64" n @@ -454,6 +470,7 @@ let pr " if (!chunk_user_data->view) goto out;\n" ) | Enum _ -> () + | Extent64 _ -> () | Flags (n, _) -> pr " %s_u32 = %s;\n" n n | Fd _ | Int _ -> () | Int64 n -> pr " %s_i64 = %s;\n" n n @@ -552,6 +569,7 @@ let | Closure { cbname } -> pr " free_user_data (%s_user_data);\n" cbname | Enum _ -> () + | Extent64 _ -> () | Flags _ -> () | Fd _ | Int _ -> () | Int64 _ -> () @@ -881,6 +899,7 @@ let | BytesPersistOut (n, _) -> n, None | Closure { cbname } -> cbname, None | Enum (n, _) -> n, None + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> n, None | Fd n | Int n -> n, None | Int64 n -> n, None diff --git a/ocaml/helpers.c b/ocaml/helpers.c index 3361a696..09666daf 100644 --- a/ocaml/helpers.c +++ b/ocaml/helpers.c @@ -133,6 +133,26 @@ nbd_internal_ocaml_alloc_i64_from_u32_array (uint32_t *a, size_t len) CAMLreturn (rv); } +value +nbd_internal_ocaml_alloc_extent64_array (nbd_extent *a, size_t len) +{ + CAMLparam0 (); + CAMLlocal3 (s, v, rv); + size_t i; + + rv = caml_alloc (len, 0); + for (i = 0; i < len; ++i) { + s = caml_alloc (2, 0); + v = caml_copy_int64 (a[i].length); + Store_field (s, 0, v); + v = caml_copy_int64 (a[i].flags); + Store_field (s, 1, v); + Store_field (rv, i, s); + } + + CAMLreturn (rv); +} + /* Convert a Unix.sockaddr to a C struct sockaddr. */ void nbd_internal_unix_sockaddr_to_sa (value sockaddrv, diff --git a/ocaml/nbd-c.h b/ocaml/nbd-c.h index e3abb912..adcdd15a 100644 --- a/ocaml/nbd-c.h +++ b/ocaml/nbd-c.h @@ -62,6 +62,7 @@ extern void nbd_internal_ocaml_raise_closed (const char *func) Noreturn; extern const char **nbd_internal_ocaml_string_list (value); extern value nbd_internal_ocaml_alloc_i64_from_u32_array (uint32_t *, size_t); +extern value nbd_internal_ocaml_alloc_extent64_array (nbd_extent *, size_t); extern void nbd_internal_unix_sockaddr_to_sa (value, struct sockaddr_storage *, socklen_t *); extern void nbd_internal_ocaml_exception_in_wrapper (const char *, value); diff --git a/golang/handle.go b/golang/handle.go index 5fe4ed4f..0b126005 100644 --- a/golang/handle.go +++ b/golang/handle.go @@ -58,6 +58,12 @@ func (h *Libnbd) String() string { return "&Libnbd{}" } +/* Used for block status callback. */ +type LibnbdExtent struct { + Length uint64 // length of the extent + Flags uint64 // flags describing properties of the extent +} + /* All functions (except Close) return ([result,] LibnbdError). */ type LibnbdError struct { Op string // operation which failed -- 2.40.1
On 5/25/23 15:00, Eric Blake wrote:
The existing nbd_block_status() callback is permanently stuck with an array of uint32_t pairs (len/2 extents), which is both constrained on
( so the terminology is: - we have "len" uint32_t elements in the array, - one extent is one uint32_t pair, - hence we have len/2 extents in the array )
maximum extent size (no larger than 4G) and on the status flags (must fit in 32 bits). While the "base:allocation" metacontext will never exceed 32 bits, it is not hard to envision other extension metacontexts where a 64-bit status would be useful (for example, Zoned Block Devices expressing a 64-bit offset[1]). Exposing 64-bit extents will require a new API; we now have the decision of whether to copy the existing API practice of returning a bare array containing len/2 extent pairs,
The term "extent pairs" is inconsistent with the above terminology. (1) We should say: - "len/2 extents", or - "len/2 uint64_t pairs".
or with a saner idea
(2) s/or with/or go with/ ?
of an array of structs with len extents.
This sounds OK; if the element type of the array is a struct, then one struct describes one extent, and then the "len" elements of the array describe "len" extents.
Returning an array of structs is actually easier to map to various language bindings, particularly since we know the length field will generally fit in 63-bits (fallout from the fact that NBD exports are effectively capped in size by signed off_t), but where the status field must represent a full 64 bits (assuming that the user wants to connect to a metadata extension that utilizes 64 bits, rather than existing metadata contexts that only expose 32 bits).
This is a good argument, but I needed to chew on the "but" conjunction for a good while until I understood the point. The "but" seemed out of place at first; only later did I realize it was actually the core of the argument ("easier to map"). The point is that for each extent, we have two properties that are same in size but that differ in meaning (hence in type), and *therefore* a struct is better than two adjacent, raw uint64_t elements. We're going to have two uint64_t fields, but conceptually, that's one off_t field (just serialized as a uint64_t) and a genuine uint64_t field, for status. (3) Can you append a sentence, "That is, conceptually, the fields have different types, so a structure is best to express them"? Or something similar?
This patch introduces the struct we plan to use in the new API, along with language bindings. The bindings for Python and OCaml were relatively straightforward; the Golang bindings took a bit more effort for me to write. Temporary unused attributes are needed to keep the compiler happy until a later patch exposes a new API using the new callback type.
(4) Can you say "We need to insert '__attribute__ ((unused))' a number of times, temporarily, to keep the compiler happy..."? "Unused attributes" sounds as if we needed some attributes that were not used. Confusing to me. What we need instead is to mark some variables or functions with an attribute that says those vars/funcs are unused.
Note that 'struct nbd_block_descriptor_ext' in lib/nbd-protocol.h is exactly the same as what we want to use in C. But it is easier to stick a new public type in <libnbd.h> than to figure out how to expose just part of a header we only want to use internally.
Could be OK or fishy; I should be able to tell later :)
[1] https://zonedstorage.io/docs/linux/zbd-api
Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/API.mli | 1 + generator/API.ml | 12 +++++++++++- generator/C.ml | 24 +++++++++++++++++++++--- generator/GoLang.ml | 24 ++++++++++++++++++++++++ generator/OCaml.ml | 24 +++++++++++++++++++++--- generator/Python.ml | 21 ++++++++++++++++++++- ocaml/helpers.c | 20 ++++++++++++++++++++ ocaml/nbd-c.h | 1 + golang/handle.go | 6 ++++++ 9 files changed, 125 insertions(+), 8 deletions(-)
diff --git a/generator/API.mli b/generator/API.mli index c5bba8c2..96626c9b 100644 --- a/generator/API.mli +++ b/generator/API.mli @@ -52,6 +52,7 @@ and | BytesPersistOut of string * string | Closure of closure (** function pointer + void *opaque *) | Enum of string * enum (** enum/union type, int in C *) +| Extent64 of string (** extent descriptor, struct nbd_extent in C *) | Fd of string (** file descriptor *) | Flags of string * flags (** flags, uint32_t in C *) | Int of string (** small int *) diff --git a/generator/API.ml b/generator/API.ml index 02c1260d..1da1ec53 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -42,6 +42,7 @@ | BytesPersistOut of string * string | Closure of closure | Enum of string * enum +| Extent64 of string | Fd of string | Flags of string * flags | Int of string @@ -157,6 +158,14 @@ let extent_closure = "nr_entries"); CBMutable (Int "error") ] } +let extent64_closure = { + cbname = "extent64"; + cbargs = [ CBString "metacontext"; + CBUInt64 "offset"; + CBArrayAndLen (Extent64 "entries", + "nr_entries"); + CBMutable (Int "error") ] +} let list_closure = { cbname = "list"; cbargs = [ CBString "name"; CBString "description" ] @@ -166,7 +175,8 @@ let context_closure = cbargs = [ CBString "name" ] } let all_closures = [ chunk_closure; completion_closure; - debug_closure; extent_closure; list_closure; + debug_closure; extent_closure; extent64_closure; + list_closure; context_closure ]
(* Enums. *)
OK.
diff --git a/generator/C.ml b/generator/C.ml index f772117c..d8472153 100644 --- a/generator/C.ml +++ b/generator/C.ml @@ -94,6 +94,7 @@ let | Closure { cbname } -> [ sprintf "%s_callback" cbname; sprintf "%s_user_data" cbname ] | Enum (n, _) -> [n] + | Extent64 n -> [n] | Fd n -> [n] | Flags (n, _) -> [n] | Int n -> [n]
This extends "name_of_arg".
@@ -129,7 +130,7 @@ let (* list of strings should be marked as non-null *) | StringList n -> [ true ] (* numeric and other non-pointer types are not able to be null *) - | Bool _ | Closure _ | Enum _ | Fd _ | Flags _ + | Bool _ | Closure _ | Enum _ | Extent64 _ | Fd _ | Flags _ | Int _ | Int64 _ | SizeT _ | UInt _ | UInt32 _ | UInt64 _ | UIntPtr _ -> [ false ]
This extends "arg_attr_nonnull".
@@ -191,6 +192,9 @@ and | Enum (n, _) -> if types then pr "int "; pr "%s" n + | Extent64 n -> + if types then pr "nbd_extent "; + pr "%s" n | Flags (n, _) -> if types then pr "uint32_t "; pr "%s" n
This extends "print_arg_list".
@@ -305,6 +309,11 @@ and pr "%s, " n; if types then pr "size_t "; pr "%s" len + | CBArrayAndLen (Extent64 n, len) -> + if types then pr "nbd_extent *"; + pr "%s, " n; + if types then pr "size_t "; + pr "%s" len | CBArrayAndLen _ -> assert false | CBBytesIn (n, len) -> if types then pr "const void *";
This extends "print_cbarg_list".
@@ -477,6 +486,13 @@ let pr "extern int nbd_get_errno (void);\n"; pr "#define LIBNBD_HAVE_NBD_GET_ERRNO 1\n"; pr "\n"; + pr "/* This is used in the callback for nbd_block_status_64.\n";
Pointing a bit into the future, but fine / unavoidable.
+ pr " */\n"; + pr "typedef struct {\n"; + pr " uint64_t length;\n"; + pr " uint64_t flags;\n"; + pr "} nbd_extent;\n"; + pr "\n"; print_closure_structs (); List.iter ( fun (name, { args; optargs; ret }) ->
The "nbd_" prefix for the new type seems consistent with existent practice (e.g. "nbd_handle"). The placement of the new typedef is good as well, IMO; looking at the generated library header, this typedef appears just before the callback structures. Also in the generator's source code / context above, we can see the call to "print_closure_structs" just after the insertion. That's where our new "extent64_closure" is going to be handled too. The question is how we're going to translate "nbd_block_descriptor_ext" to this new structure, in *C* code. If we populate the members individually, together with big-endian-to-host-endian conversion, that should be OK. I figure that belongs in a later patch though.
@@ -740,7 +756,7 @@ let pr " char *%s_printable =\n" n; pr " nbd_internal_printable_string_list (%s);\n" n | BytesOut _ | BytesPersistOut _ - | Bool _ | Closure _ | Enum _ | Flags _ | Fd _ | Int _ + | Bool _ | Closure _ | Enum _ | Extent64 _ | Flags _ | Fd _ | Int _ | Int64 _ | SizeT _ | SockAddrAndLen _ | UInt _ | UInt32 _ | UInt64 _ | UIntPtr _ -> () ) args;
This extends "print_trace_enter" (near the top).
@@ -766,6 +782,7 @@ let pr " %s=\\\"%%s\\\" %s=%%zu" n count | Closure { cbname } -> pr " %s=%%s" cbname | Enum (n, _) -> pr " %s=%%d" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> pr " %s=0x%%x" n | Fd n | Int n -> pr " %s=%%d" n | Int64 n -> pr " %s=%%\"PRIi64\"" n
And this extends "print_format_string" within "print_trace_enter". Now here's why I've inserted all those "this extends ..." remarks: I think your assertion is valid here, *but* it is inconsistent with the extensions to: - name_of_arg - arg_attr_nonnull - print_arg_list - print_trace_enter We only deal with Extent64 via the "penetrating pattern match" CBArrayAndLen (Extent64 n, len), in "print_cbarg_list". (5) Therefore, for consistency, *all* other Extent64 pattern matches should assert that they are unreachable -- not only in "print_format_string", but in those four functions too that I list above. Clearly, I could be missing some code paths leading to those functions / patterns; I've not tested my proposition. The idea is that the narrowest necessary subset of patterns should be handled. Even if it is easy to "handle" a pattern with just (), it makes the wrong impression.
@@ -804,6 +821,7 @@ let pr "%s_printable ? %s_printable : \"\", %s" n n count | Closure _ -> pr "\"<fun>\"" | Enum (n, _) -> pr "%s" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> pr "%s" n | Fd n | Int n | Int64 n | SizeT n -> pr "%s" n | SockAddrAndLen (_, len) -> pr "(int) %s" len
This extends "print_args" within "print_trace_enter". The assertion is OK.
@@ -836,7 +854,7 @@ let | StringList n -> pr " free (%s_printable);\n" n | BytesOut _ | BytesPersistOut _ - | Bool _ | Closure _ | Enum _ | Flags _ | Fd _ | Int _ + | Bool _ | Closure _ | Enum _ | Extent64 _ | Flags _ | Fd _ | Int _ | Int64 _ | SizeT _ | SockAddrAndLen _ | UInt _ | UInt32 _ | UInt64 _ | UIntPtr _ -> () ) args;
(6) This is near the end of "print_trace_enter", and again, we should assert here that "Extent64 _" will never match.
diff --git a/generator/GoLang.ml b/generator/GoLang.ml index 4ab6b262..8922812b 100644 --- a/generator/GoLang.ml +++ b/generator/GoLang.ml @@ -49,6 +49,7 @@ let | BytesPersistOut (n, len) -> n | Closure { cbname } -> cbname | Enum (n, _) -> n + | Extent64 n -> n | Fd n -> n | Flags (n, _) -> n | Int n -> n @@ -71,6 +72,7 @@ let | BytesPersistOut _ -> "AioBuffer" | Closure { cbname } -> sprintf "%sCallback" (camel_case cbname) | Enum (_, { enum_prefix }) -> camel_case enum_prefix + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd _ -> "int" | Flags (_, { flag_prefix }) -> camel_case flag_prefix | Int _ -> "int" @@ -262,6 +264,7 @@ let pr " c_%s.user_data = C.alloc_cbid(C.long(%s_cbid))\n" cbname cbname | Enum (n, _) -> pr " c_%s := C.int (%s)\n" n n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd n -> pr " c_%s := C.int (%s)\n" n n | Flags (n, _) -> @@ -334,6 +337,7 @@ let | BytesPersistOut (n, len) -> pr ", c_%s, c_%s" n len | Closure { cbname } -> pr ", c_%s" cbname | Enum (n, _) -> pr ", c_%s" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd n -> pr ", c_%s" n | Flags (n, _) -> pr ", c_%s" n | Int n -> pr ", c_%s" n
(7) I care pretty little for the Go binding, but again: the hunk above that is the first for "GoLang.ml", we extend "go_name_of_arg", and I'd argue we should insert an "assert false" there.
@@ -524,6 +528,18 @@ let copy(ret, s) return ret } + +func copy_extent_array (entries *C.nbd_extent, count C.size_t) []LibnbdExtent { + unsafePtr := unsafe.Pointer(entries) + arrayPtr := (*[1 << 20]C.nbd_extent)(unsafePtr) + slice := arrayPtr[:count:count] + ret := make([]LibnbdExtent, count) + for i := 0; i < int (count); i++ { + ret[i].Length = uint64 (slice[i].length) + ret[i].Flags = uint64 (slice[i].flags) + } + return ret +} ";
List.iter (
The pre-existent copy_uint32_array() function uses the hideous trick at <https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices>, and needlessly so, IMO. - The trick is (a) hideous because it requires us to use arbitrary sizes such as "1<<30". - The trick is (b) unnecessary because we don't intend to hang on to the slice indefinitely. We only use it as a means to access the source object. But at the noted URL, the trick is "being sold" with the pitch "to create a Go slice backed by a C array (without copying the original data)" -- and we copy the original data *anyway*! So it's better to use pointer arithmetic IMO. Regarding the new copy_extent_array(), my additional complaints are: - whitespace usage before opening parens is inconsistent -- there is no space after "make" and "Pointer". - we cast "count" (a size_t in C) to a Go "int"; similarly the index variable "i" has Go type "int". (8) So how about this instead (should be split in two: the first part should update copy_uint32_array() in a separate patch, and the second part should be squashed into this patch):
diff --git a/generator/GoLang.ml b/generator/GoLang.ml index 8922812b76a4..37b2240ef5bf 100644 --- a/generator/GoLang.ml +++ b/generator/GoLang.ml @@ -521,22 +521,32 @@ let /* Closures. */
func copy_uint32_array (entries *C.uint32_t, count C.size_t) []uint32 { - ret := make([]uint32, int (count)) - // See https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices - // TODO: Use unsafe.Slice() when we require Go 1.17. - s := (*[1<<30]uint32)(unsafe.Pointer(entries))[:count:count] - copy(ret, s) + ret := make ([]uint32, count) + addr := uintptr (unsafe.Pointer (entries)) + var i uint64 = 0 + for i < uint64 (count) { + ptr := (*C.uint32_t)(unsafe.Pointer (addr)) + + ret[i] = uint32 (*ptr) + + addr += unsafe.Sizeof (*ptr) + i++ + } return ret }
func copy_extent_array (entries *C.nbd_extent, count C.size_t) []LibnbdExtent { - unsafePtr := unsafe.Pointer(entries) - arrayPtr := (*[1 << 20]C.nbd_extent)(unsafePtr) - slice := arrayPtr[:count:count] - ret := make([]LibnbdExtent, count) - for i := 0; i < int (count); i++ { - ret[i].Length = uint64 (slice[i].length) - ret[i].Flags = uint64 (slice[i].flags) + ret := make ([]LibnbdExtent, count) + addr := uintptr (unsafe.Pointer (entries)) + var i uint64 = 0 + for i < uint64 (count) { + ptr := (*C.nbd_extent)(unsafe.Pointer (addr)) + + ret[i].Length = uint64 ((*ptr).length) + ret[i].Flags = uint64 ((*ptr).flags) + + addr += unsafe.Sizeof (*ptr) + i++ } return ret }
(This passes make and make-check, when the tree is checked out at your patch#7, and this is applied on top.) Back to your patch: On 5/25/23 15:00, Eric Blake wrote:
@@ -537,6 +553,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, _) -> pr "%s []uint32" n; + | CBArrayAndLen (Extent64 n, _) -> + pr "%s []LibnbdExtent" n; | CBBytesIn (n, len) -> pr "%s []byte" n; | CBInt n -> @@ -563,6 +581,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "%s *C.uint32_t, %s C.size_t" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "%s *C.nbd_extent, %s C.size_t" n count | CBBytesIn (n, len) -> pr "%s unsafe.Pointer, %s C.size_t" n len | CBInt n -> @@ -605,6 +625,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "copy_uint32_array (%s, %s)" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "copy_extent_array (%s, %s)" n count | CBBytesIn (n, len) -> pr "C.GoBytes (%s, C.int (%s))" n len | CBInt n -> @@ -756,6 +778,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "uint32_t *%s, size_t %s" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "nbd_extent *%s, size_t %s" n count | CBBytesIn (n, len) -> pr "void *%s, size_t %s" n len | CBInt n ->
OK. Quoting the rest of the patch out of order, so I can leave the OCaml stuff for the end (this is a *terribly* large patch, when the contexts are increased suitably for reviewing, so I really wish we could split it up somehow):
diff --git a/golang/handle.go b/golang/handle.go index 5fe4ed4f..0b126005 100644 --- a/golang/handle.go +++ b/golang/handle.go @@ -58,6 +58,12 @@ func (h *Libnbd) String() string { return "&Libnbd{}" }
+/* Used for block status callback. */ +type LibnbdExtent struct { + Length uint64 // length of the extent + Flags uint64 // flags describing properties of the extent +} + /* All functions (except Close) return ([result,] LibnbdError). */ type LibnbdError struct { Op string // operation which failed
OK.
diff --git a/generator/Python.ml b/generator/Python.ml index c81878de..ccac5fed 100644 --- a/generator/Python.ml +++ b/generator/Python.ml @@ -158,6 +158,7 @@ let let print_python_closure_wrapper { cbname; cbargs } = pr "/* Wrapper for %s callback. */\n" cbname; pr "static int\n"; + pr "__attribute__ ((unused)) /* XXX temporary hack */\n"; pr "%s_wrapper " cbname; C.print_cbarg_list ~wrap:true cbargs; pr "\n";
OK, this is as announced in the commit message.
@@ -170,6 +171,7 @@ let List.iter ( function | CBArrayAndLen (UInt32 n, _) + | CBArrayAndLen (Extent64 n, _) | CBBytesIn (n, _) | CBMutable (Int n) -> pr " PyObject *py_%s = NULL;\n" n
OK, this is responsible for generating: PyObject *py_entries = NULL;
@@ -187,6 +189,16 @@ let pr " if (!py_e_%s) { PyErr_PrintEx (0); goto out; }\n" n; pr " PyList_SET_ITEM (py_%s, i_%s, py_e_%s);\n" n n n; pr " }\n" + | CBArrayAndLen (Extent64 n, len) -> + pr " py_%s = PyList_New (%s);\n" n len; + pr " size_t i_%s;\n" n;
(9) Under "CBArrayAndLen (UInt32 n, len)", we also check whether the New call succeeds; I think it should not be omitted here either: pr " if (!py_%s) { PyErr_PrintEx (0); goto out; }\n" n; producing (in the preexistent "extent_wrapper" function): if (!py_entries) { PyErr_PrintEx (0); goto out; }
+ pr " for (i_%s = 0; i_%s < %s; ++i_%s) {\n" n n len n; + pr " PyObject *py_e_%s = Py_BuildValue (\"OO\",\n" n; + pr " PyLong_FromUnsignedLong (%s[i_%s].length),\n" n n; + pr " PyLong_FromUnsignedLong (%s[i_%s].flags));\n" n n;
We have two issues here: (10) We should call PyLong_FromUnsignedLongLong(), not PyLong_FromUnsignedLong(). On an ILP32 platform, the latter will not do the right thing. (11) PyLong_FromWhatever() returns a new reference, and so it can fail due to memory exhaustion (minimally, I think). That means zero, one, or two of our PyLong_FromUnsignedLongLong() calls can fail, and return NULL. We call Py_BuildValue() with an "OO" specification. The documentation says:
Pass a Python object untouched (except for its reference count, which is incremented by one). If the object passed in is a NULL pointer, it is assumed that this was caused because the call producing the argument found an error and set an exception. Therefore, Py_BuildValue() will return NULL but won’t raise an exception. If no exception has been raised yet, SystemError is set.
Now, I don't think PyLong_FromUnsignedLongLong() sets an exception when it fails (and returns NULL), but that should be fine -- at least Py_BuildValue() will raise SystemError. The problem is that we have three possible ways to leak memory: - If we pass two NULLs to "OO" (after both our PyLong_FromUnsignedLongLong() calls fail), there's no leak. Py_BuildValue() fails in response too, and nothing is leaked. - If we pass exactly one NULL to "OO" (after one PyLong_FromUnsignedLongLong() succeeds, but the other fails), then Py_BuildValue() fails again (as expected), but we leak the PyLongObject that we *did* successfully create. - If we pass two non-NULLs to "OO" (after both PyLong_FromUnsignedLongLong() succeed), and Py_BuildValue() fails because *it* cannot allocate further memory, then we leak *both* PyLongObjects. - If we pass two non-NULLs to "OO" (after both PyLong_FromUnsignedLongLong() succeed), and Py_BuildValue() *succeds*, then we leak both PyLongObjects again: PyLong_FromUnsignedLongLong() returns a new reference, but when Py_BuildValue() succeeds, it adds a *new* reference (it does not only take ownership). The actual leak will happen later, when the object just constructed with Py_BuildValue() is torn down. So, we should implement this in a much more "staged" way. Shown as a diff on top of the code generated by your patch, covering points (9) through (11) (even (12) below):
--- python/methods.c.orig 2023-06-07 13:29:01.947636166 +0200 +++ python/methods.c 2023-06-07 13:44:23.845892043 +0200 @@ -286,15 +286,24 @@ extent64_wrapper (void *user_data, const PyObject *py_args, *py_ret; PyObject *py_entries = NULL; PyObject *py_error = NULL; + PyObject *py_e_length = NULL; + PyObject *py_e_flags = NULL;
py_entries = PyList_New (nr_entries); + if (!py_entries) { PyErr_PrintEx (0); goto out; } size_t i_entries; for (i_entries = 0; i_entries < nr_entries; ++i_entries) { - PyObject *py_e_entries = Py_BuildValue ("OO", - PyLong_FromUnsignedLong (entries[i_entries].length), - PyLong_FromUnsignedLong (entries[i_entries].flags)); + PyObject *py_e_entries; + + py_e_length = PyLong_FromUnsignedLongLong (entries[i_entries].length); + py_e_flags = PyLong_FromUnsignedLongLong (entries[i_entries].flags); + py_e_entries = Py_BuildValue ("OO", py_e_length, py_e_flags); if (!py_e_entries) { PyErr_PrintEx (0); goto out; } PyList_SET_ITEM (py_entries, i_entries, py_e_entries); + Py_XDECREF (py_e_flags); + Py_XDECREF (py_e_length); + py_e_flags = NULL; + py_e_length = NULL; } py_error = nbd_internal_py_wrap_errptr (*error); if (!py_error) { PyErr_PrintEx (0); goto out; } @@ -325,6 +334,8 @@ extent64_wrapper (void *user_data, const };
out: + Py_XDECREF (py_e_flags); + Py_XDECREF (py_e_length); Py_XDECREF (py_entries); if (py_error) { PyObject *py_error_ret = PyObject_GetAttrString (py_error, "value");
In this variant, if at least one PyLong_FromUnsignedLongLong() call fails, then Py_BuildValue() catches that, fails itself, and at the "out" label, we release precisely those PyLongObjects that we managed to construct (zero or one objects). If both PyLongObjects get allocated, but Py_BuildValue() fails, then at the "out" label, we release both PyLongObjects. If even the Py_BuildValue() call succeeds, then we reach PyList_SET_ITEM(). PyList_SET_ITEM() "steals" a reference (i.e., takes ownership of one reference), it does not add one. Therefore the just constructed "py_e_entries" object is now owned by the "py_entries" list -- and we want the new "py_e_entries" entry in that list too to own our PyLongObjects. Therefore, we release our temporary references to the PyLongObjects (see the xdecref's in the loop body). We also NULL out the variables, reflecting the ownership transfer. This is generally good practice, but it even matters, for example, when this is the last iteration of the loop, and the subsequent nbd_internal_py_wrap_errptr() call fails. Then we jump to the "out" label, and at that point, the xdecref's *there* must be no-ops. The PyLongObjects will be released via "py_entries". (In theory we could NULL out the variables right after the loop body, rather than at the end of the loop body, given that the top of the loop body immediately overwrites both variables -- but as written, the code is more understandable.) The xdecref's and the NULL'ing happen in reverse, relative to order of construction (i.e., the inverse order is intentional). Back to your patch: On 5/25/23 15:00, Eric Blake wrote:
+ pr " if (!py_e_%s) { PyErr_PrintEx (0); goto out; }\n" n; + pr " PyList_SET_ITEM (py_%s, i_%s, py_e_%s);\n" n n n; + pr " }\n" | CBBytesIn (n, len) -> pr " py_%s = nbd_internal_py_get_subview (data->view, %s, %s);\n" n n len; pr " if (!py_%s) { PyErr_PrintEx (0); goto out; }\n" n @@ -206,6 +218,7 @@ let List.map ( function | CBArrayAndLen (UInt32 n, _) -> "O", sprintf "py_%s" n + | CBArrayAndLen (Extent64 n, _) -> "O", sprintf "py_%s" n | CBBytesIn (n, _) -> "O", sprintf "py_%s" n | CBInt n -> "i", n | CBInt64 n -> "L", n
Right, this corresponds to "py_entries" being a list object.
@@ -254,7 +267,8 @@ let pr " out:\n"; List.iter ( function - | CBArrayAndLen (UInt32 n, _) -> + | CBArrayAndLen (UInt32 n, _) + | CBArrayAndLen (Extent64 n, _) -> pr " Py_XDECREF (py_%s);\n" n | CBBytesIn (n, _) -> pr " Py_XDECREF (py_%s);\n" n
(12) We do need this Py_XDECREF (py_entries); we just need the Py_XDECREF (py_e_flags); Py_XDECREF (py_e_length); calls before it, too, so I think ultimately you'll want to handle this pattern separately.
@@ -307,6 +321,7 @@ let pr ".callback = %s_wrapper, .free = free_user_data" cbname); pr " };\n" | Enum (n, _) -> pr " int %s;\n" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> pr " uint32_t %s_u32;\n" n; pr " unsigned int %s; /* really uint32_t */\n" n @@ -363,6 +378,7 @@ let "O", sprintf "&%s" n, sprintf "py_%s->buf, py_%s->len" n n | Closure { cbname } -> "O", sprintf "&py_%s_fn" cbname, cbname | Enum (n, _) -> "i", sprintf "&%s" n, n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> "I", sprintf "&%s" n, sprintf "%s_u32" n | Fd n | Int n -> "i", sprintf "&%s" n, n | Int64 n -> "L", sprintf "&%s" n, sprintf "%s_i64" n
Yes, these are an "easy sell".
@@ -454,6 +470,7 @@ let pr " if (!chunk_user_data->view) goto out;\n" ) | Enum _ -> () + | Extent64 _ -> () | Flags (n, _) -> pr " %s_u32 = %s;\n" n n | Fd _ | Int _ -> () | Int64 n -> pr " %s_i64 = %s;\n" n n @@ -552,6 +569,7 @@ let | Closure { cbname } -> pr " free_user_data (%s_user_data);\n" cbname | Enum _ -> () + | Extent64 _ -> () | Flags _ -> () | Fd _ | Int _ -> () | Int64 _ -> ()
(13) However I would turn these into assertions of "false", as well. They too are in "print_python_binding".
@@ -881,6 +899,7 @@ let | BytesPersistOut (n, _) -> n, None | Closure { cbname } -> cbname, None | Enum (n, _) -> n, None + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> n, None | Fd n | Int n -> n, None | Int64 n -> n, None
OK.
diff --git a/ocaml/nbd-c.h b/ocaml/nbd-c.h index e3abb912..adcdd15a 100644 --- a/ocaml/nbd-c.h +++ b/ocaml/nbd-c.h @@ -62,6 +62,7 @@ extern void nbd_internal_ocaml_raise_closed (const char *func) Noreturn;
extern const char **nbd_internal_ocaml_string_list (value); extern value nbd_internal_ocaml_alloc_i64_from_u32_array (uint32_t *, size_t); +extern value nbd_internal_ocaml_alloc_extent64_array (nbd_extent *, size_t); extern void nbd_internal_unix_sockaddr_to_sa (value, struct sockaddr_storage *, socklen_t *); extern void nbd_internal_ocaml_exception_in_wrapper (const char *, value);
OK.
diff --git a/generator/OCaml.ml b/generator/OCaml.ml index edb81f25..ea3b044a 100644 --- a/generator/OCaml.ml +++ b/generator/OCaml.ml @@ -44,6 +44,7 @@ and | Closure { cbargs } -> sprintf "(%s)" (ocaml_closuredecl_to_string cbargs) | Enum (_, { enum_prefix }) -> sprintf "%s.t" enum_prefix + | Extent64 _ -> "extent" | Fd _ -> "Unix.file_descr" | Flags (_, { flag_prefix }) -> sprintf "%s.t list" flag_prefix | Int _ -> "int" @@ -102,6 +103,7 @@ let | BytesPersistOut (n, len) -> n | Closure { cbname } -> cbname | Enum (n, _) -> n + | Extent64 n -> n | Fd n -> n | Flags (n, _) -> n | Int n -> n
(14) Could we use "assert false" with these too?
@@ -147,6 +149,9 @@ let
type cookie = int64
+type extent = int64 * int64 +(** Length and flags of an extent in block_status_64 callback. *) + ";
List.iter (
(15) Can we add a note here that negative values are not permitted? (See my further comments below.)
@@ -268,6 +273,7 @@ let exception Error of string * Unix.error option exception Closed of string type cookie = int64 +type extent = int64 * int64
(* Give the exceptions names so that they can be raised from the C code. *) let () = @@ -498,7 +504,8 @@ let let argnames = List.map ( function - | CBArrayAndLen (UInt32 n, _) | CBBytesIn (n, _) + | CBArrayAndLen (UInt32 n, _) | CBArrayAndLen (Extent64 n, _) + | CBBytesIn (n, _) | CBInt n | CBInt64 n | CBMutable (Int n) | CBString n | CBUInt n | CBUInt64 n -> n ^ "v"
This is inside "print_ocaml_closure_wrapper"; OK. (16) You can contract the new pattern as follows: | CBArrayAndLen ((UInt32 n | Extent64 n), _)
@@ -531,6 +538,14 @@ let pr "%s %s,\n" indent n; pr "%s %s\n" indent count; pr "%s);\n" indent + | CBArrayAndLen (Extent64 n, count) -> + pr " %sv = " n; + let fncol = output_column () in + let indent = spaces fncol in + pr "nbd_internal_ocaml_alloc_extent64_array (\n"; + pr "%s %s,\n" indent n; + pr "%s %s\n" indent count; + pr "%s);\n" indent | CBBytesIn (n, len) -> pr " %sv = caml_alloc_initialized_string (%s, %s);\n" n len n | CBInt n | CBUInt n ->
Seems OK superficially; matches the 'CBArrayAndLen (UInt32 n, count)' case, with only the helper function differing.
@@ -554,7 +569,7 @@ let
List.iter ( function - | CBArrayAndLen (UInt32 _, _) + | CBArrayAndLen (_, _) | CBBytesIn _ | CBInt _ | CBInt64 _ @@ -563,7 +578,7 @@ let | CBUInt64 _ -> () | CBMutable (Int n) -> pr " *%s = Int_val (Field (%sv, 0));\n" n n - | CBArrayAndLen _ | CBMutable _ -> assert false + | CBMutable _ -> assert false ) cbargs;
pr " if (Is_exception_result (rv)) {\n";
(17) We're still inside "print_ocaml_closure_wrapper", so I think the patterns here should follow the patterns seen in the "argnames" definition: | CBArrayAndLen (UInt32 n, _) | CBArrayAndLen (Extent64 n, _) or even better (again) | CBArrayAndLen ((UInt32 _ | Extent64 _), _) IOW I think "CBArrayAndLen (_, _)" is too generic / inconsistent with previously added code. We should handle the Extent64 constructor, and not touch the "CBArrayAndLen _" fallback.
@@ -578,6 +593,7 @@ let pr "}\n"; pr "\n"; pr "static int\n"; + pr "__attribute__ ((unused)) /* XXX temporary hack */\n"; pr "%s_wrapper " cbname; C.print_cbarg_list ~wrap:true cbargs; pr "\n";
OK, announced in the commit message.
@@ -695,6 +711,7 @@ let pr " %s_callback.free = free_user_data;\n" cbname | Enum (n, { enum_prefix }) -> pr " int %s = %s_val (%sv);\n" n enum_prefix n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd n -> pr " /* OCaml Unix.file_descr is just an int, at least on Unix. */\n"; pr " int %s = Int_val (%sv);\n" n n
OK, this is in "print_ocaml_binding".
@@ -780,6 +797,7 @@ let | BytesPersistOut _ | Closure _ | Enum _ + | Extent64 _ | Fd _ | Flags _ | Int _
This is still in "print_ocaml_binding", under comment "Any parameters which need to be freed". (18) For consistency I'd suggest "assert false" here as well.
diff --git a/ocaml/helpers.c b/ocaml/helpers.c index 3361a696..09666daf 100644 --- a/ocaml/helpers.c +++ b/ocaml/helpers.c @@ -133,6 +133,26 @@ nbd_internal_ocaml_alloc_i64_from_u32_array (uint32_t *a, size_t len) CAMLreturn (rv); }
+value +nbd_internal_ocaml_alloc_extent64_array (nbd_extent *a, size_t len) +{ + CAMLparam0 (); + CAMLlocal3 (s, v, rv); + size_t i; + + rv = caml_alloc (len, 0); + for (i = 0; i < len; ++i) { + s = caml_alloc (2, 0); + v = caml_copy_int64 (a[i].length); + Store_field (s, 0, v); + v = caml_copy_int64 (a[i].flags); + Store_field (s, 1, v); + Store_field (rv, i, s); + } + + CAMLreturn (rv); +} + /* Convert a Unix.sockaddr to a C struct sockaddr. */ void nbd_internal_unix_sockaddr_to_sa (value sockaddrv,
(19) I'd suggest the following addition:
diff --git a/ocaml/helpers.c b/ocaml/helpers.c index 09666dafa7d1..db652943141d 100644 --- a/ocaml/helpers.c +++ b/ocaml/helpers.c @@ -25,6 +25,7 @@ #include <string.h> #include <sys/socket.h> #include <assert.h> +#include <inttypes.h>
#include <caml/alloc.h> #include <caml/callback.h> @@ -140,6 +141,16 @@ nbd_internal_ocaml_alloc_extent64_array (nbd_extent *a, size_t len) CAMLlocal3 (s, v, rv); size_t i;
+ for (i = 0; i < len; ++i) + if (a[i].length > INT64_MAX || a[i].flags > INT64_MAX) { + char errmsg[256]; + + snprintf (errmsg, sizeof errmsg, + "%s: extent[%zu] = { .length = %"PRIu64", .flags = %"PRIu64"}", + __func__, i, a[i].length, a[i].flags); + caml_failwith (errmsg); + } + rv = caml_alloc (len, 0); for (i = 0; i < len; ++i) { s = caml_alloc (2, 0);
*However*, considering the nbd_internal_ocaml_alloc_extent64_array() call site, in the generated extent64_wrapper_locked() function [ocaml/nbd-c.c]:
/* Wrapper for extent64 callback. */ static int extent64_wrapper_locked (void *user_data, const char *metacontext, uint64_t offset, nbd_extent *entries, size_t nr_entries, int *error) { CAMLparam0 (); CAMLlocal4 (metacontextv, offsetv, entriesv, errorv); CAMLlocal2 (exn, rv); const struct user_data *data = user_data; int r; value args[4];
metacontextv = caml_copy_string (metacontext); offsetv = caml_copy_int64 (offset); entriesv = nbd_internal_ocaml_alloc_extent64_array ( entries, nr_entries ); errorv = caml_alloc_tuple (1); Store_field (errorv, 0, Val_int (*error)); args[0] = metacontextv; args[1] = offsetv; args[2] = entriesv; args[3] = errorv; rv = caml_callbackN_exn (data->fnv, 4, args); *error = Int_val (Field (errorv, 0)); if (Is_exception_result (rv)) { nbd_internal_ocaml_exception_in_wrapper ("extent64", rv); CAMLreturnT (int, -1); }
r = Int_val (rv); assert (r >= 0); CAMLreturnT (int, r); }
I'm not sure if raising an OCaml exception like this, in an *inner* C function, is appropriate. caml_failwith() may only be suitable for C functions *directly* called by the OCaml runtime. If that's the case, then we should propagate the error from nbd_internal_ocaml_alloc_extent64_array() differently. We might have to change the signature of nbd_internal_ocaml_alloc_extent64_array(), and then the generated call site (i.e., extent64_wrapper_locked()) must take a potential failure into account. This is really hard to reason about, as the only "prior art", namely extent_wrapper_locked() -> nbd_internal_ocaml_alloc_i64_from_u32_array(), does not face such a conundrum. There we took the easy way: we convert from uint32_t to OCaml's int64, which cannot overflow. Again, this is a huge patch; I've now spent almost a full workday on reviewing it. I suggest splitting each language to its own patch; thus, we'd get at least 5 patches (core generator changes + 4 languages). If necessary to placate the OCaml compiler, I'd try introducing bare-bones place-holder code at first in the generator for each language, in the first ("core") patch, and then elaborate on each language in a separate patch (replacing the placeholders with actual code). Laszlo
On Wed, Jun 07, 2023 at 04:23:27PM +0200, Laszlo Ersek wrote: [...]
diff --git a/ocaml/helpers.c b/ocaml/helpers.c index 3361a696..09666daf 100644 --- a/ocaml/helpers.c +++ b/ocaml/helpers.c @@ -133,6 +133,26 @@ nbd_internal_ocaml_alloc_i64_from_u32_array (uint32_t *a, size_t len) CAMLreturn (rv); }
+value +nbd_internal_ocaml_alloc_extent64_array (nbd_extent *a, size_t len) +{ + CAMLparam0 (); + CAMLlocal3 (s, v, rv); + size_t i; + + rv = caml_alloc (len, 0); + for (i = 0; i < len; ++i) { + s = caml_alloc (2, 0); + v = caml_copy_int64 (a[i].length); + Store_field (s, 0, v); + v = caml_copy_int64 (a[i].flags); + Store_field (s, 1, v); + Store_field (rv, i, s); + } + + CAMLreturn (rv); +} + /* Convert a Unix.sockaddr to a C struct sockaddr. */ void nbd_internal_unix_sockaddr_to_sa (value sockaddrv,
(19) I'd suggest the following addition:
diff --git a/ocaml/helpers.c b/ocaml/helpers.c index 09666dafa7d1..db652943141d 100644 --- a/ocaml/helpers.c +++ b/ocaml/helpers.c @@ -25,6 +25,7 @@ #include <string.h> #include <sys/socket.h> #include <assert.h> +#include <inttypes.h>
#include <caml/alloc.h> #include <caml/callback.h> @@ -140,6 +141,16 @@ nbd_internal_ocaml_alloc_extent64_array (nbd_extent *a, size_t len) CAMLlocal3 (s, v, rv); size_t i;
+ for (i = 0; i < len; ++i) + if (a[i].length > INT64_MAX || a[i].flags > INT64_MAX) { + char errmsg[256]; + + snprintf (errmsg, sizeof errmsg, + "%s: extent[%zu] = { .length = %"PRIu64", .flags = %"PRIu64"}", + __func__, i, a[i].length, a[i].flags); + caml_failwith (errmsg); + } + rv = caml_alloc (len, 0); for (i = 0; i < len; ++i) { s = caml_alloc (2, 0);
*However*, considering the nbd_internal_ocaml_alloc_extent64_array() call site, in the generated extent64_wrapper_locked() function [ocaml/nbd-c.c]:
/* Wrapper for extent64 callback. */ static int extent64_wrapper_locked (void *user_data, const char *metacontext, uint64_t offset, nbd_extent *entries, size_t nr_entries, int *error) { CAMLparam0 (); CAMLlocal4 (metacontextv, offsetv, entriesv, errorv); CAMLlocal2 (exn, rv); const struct user_data *data = user_data; int r; value args[4];
metacontextv = caml_copy_string (metacontext); offsetv = caml_copy_int64 (offset); entriesv = nbd_internal_ocaml_alloc_extent64_array ( entries, nr_entries ); errorv = caml_alloc_tuple (1); Store_field (errorv, 0, Val_int (*error)); args[0] = metacontextv; args[1] = offsetv; args[2] = entriesv; args[3] = errorv; rv = caml_callbackN_exn (data->fnv, 4, args); *error = Int_val (Field (errorv, 0)); if (Is_exception_result (rv)) { nbd_internal_ocaml_exception_in_wrapper ("extent64", rv); CAMLreturnT (int, -1); }
r = Int_val (rv); assert (r >= 0); CAMLreturnT (int, r); }
I'm not sure if raising an OCaml exception like this, in an *inner* C function, is appropriate. caml_failwith() may only be suitable for C functions *directly* called by the OCaml runtime.
caml_failwith is just a longjmp. The OCaml values on the stack are cleaned up by following a chain of stack frames which are created by the CAMLparam* macros. These macros don't need to be used in every function, although they should be used in functions which have OCaml value parameters or value local variables (but there's no harm in using them unnecessarily). They can also be omitted for functions which do not invoke the OCaml GC (don't allocate on the OCaml heap, basically). C local variables however will not be cleaned up, so if there are any arrays that have to be freed then it gets a bit more complicated. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Wed, Jun 07, 2023 at 04:23:27PM +0200, Laszlo Ersek wrote:
On 5/25/23 15:00, Eric Blake wrote:
The existing nbd_block_status() callback is permanently stuck with an array of uint32_t pairs (len/2 extents), which is both constrained on
(
so the terminology is:
- we have "len" uint32_t elements in the array,
- one extent is one uint32_t pair,
- hence we have len/2 extents in the array
)
maximum extent size (no larger than 4G) and on the status flags (must fit in 32 bits). While the "base:allocation" metacontext will never exceed 32 bits, it is not hard to envision other extension metacontexts where a 64-bit status would be useful (for example, Zoned Block Devices expressing a 64-bit offset[1]). Exposing 64-bit extents will require a new API; we now have the decision of whether to copy the existing API practice of returning a bare array containing len/2 extent pairs,
The term "extent pairs" is inconsistent with the above terminology.
(1) We should say:
- "len/2 extents", or - "len/2 uint64_t pairs".
I'll improve the wording here. I will be rebasing this on a pre-requisite patch where I introduce h->bs_count which contains the number of extents (bs_count * 2 * sizeof(uint{32,64}_t) gets the byte count conusmed over the wire for 32/64-bit replies).
or with a saner idea
(2) s/or with/or go with/ ?
of an array of structs with len extents.
This sounds OK; if the element type of the array is a struct, then one struct describes one extent, and then the "len" elements of the array describe "len" extents.
Returning an array of structs is actually easier to map to various language bindings, particularly since we know the length field will generally fit in 63-bits (fallout from the fact that NBD exports are effectively capped in size by signed off_t), but where the status field must represent a full 64 bits (assuming that the user wants to connect to a metadata extension that utilizes 64 bits, rather than existing metadata contexts that only expose 32 bits).
This is a good argument, but I needed to chew on the "but" conjunction for a good while until I understood the point. The "but" seemed out of place at first; only later did I realize it was actually the core of the argument ("easier to map").
The point is that for each extent, we have two properties that are same in size but that differ in meaning (hence in type), and *therefore* a struct is better than two adjacent, raw uint64_t elements. We're going to have two uint64_t fields, but conceptually, that's one off_t field (just serialized as a uint64_t) and a genuine uint64_t field, for status.
(3) Can you append a sentence, "That is, conceptually, the fields have different types, so a structure is best to express them"? Or something similar?
v4 will reword things based on this feedback, although probably not with the same language.
This patch introduces the struct we plan to use in the new API, along with language bindings. The bindings for Python and OCaml were relatively straightforward; the Golang bindings took a bit more effort for me to write. Temporary unused attributes are needed to keep the compiler happy until a later patch exposes a new API using the new callback type.
(4) Can you say "We need to insert '__attribute__ ((unused))' a number of times, temporarily, to keep the compiler happy..."?
"Unused attributes" sounds as if we needed some attributes that were not used. Confusing to me. What we need instead is to mark some variables or functions with an attribute that says those vars/funcs are unused.
Yes, quoting the code that is being temporarily added makes legibility nicer.
...
+++ b/generator/C.ml @@ -94,6 +94,7 @@ let | Closure { cbname } -> [ sprintf "%s_callback" cbname; sprintf "%s_user_data" cbname ] | Enum (n, _) -> [n] + | Extent64 n -> [n] | Fd n -> [n] | Flags (n, _) -> [n] | Int n -> [n]
This extends "name_of_arg". ...
+ pr " */\n"; + pr "typedef struct {\n"; + pr " uint64_t length;\n"; + pr " uint64_t flags;\n"; + pr "} nbd_extent;\n"; + pr "\n"; print_closure_structs (); List.iter ( fun (name, { args; optargs; ret }) ->
The "nbd_" prefix for the new type seems consistent with existent practice (e.g. "nbd_handle").
The placement of the new typedef is good as well, IMO; looking at the generated library header, this typedef appears just before the callback structures. Also in the generator's source code / context above, we can see the call to "print_closure_structs" just after the insertion. That's where our new "extent64_closure" is going to be handled too.
The question is how we're going to translate "nbd_block_descriptor_ext" to this new structure, in *C* code. If we populate the members individually, together with big-endian-to-host-endian conversion, that should be OK. I figure that belongs in a later patch though.
Yes, the translation between wire format and the callback array will be in a later patch. That part of my series is already undergoing some of the biggest churn (you've convinced me in 8/22 that it is better to malloc two separate arrays: one for the bytes read over the wire, one for the array to send to the callback, in order to accomodate all four pairings of server wide/narrow response vs. caller 32/64-bit API without nasty in-place overwriting). I think I've hit the point where I should post v4 of all patches prior to this one and get them settled, while I continue rebasing how I want to express block status internally according to your review comments.
@@ -740,7 +756,7 @@ let pr " char *%s_printable =\n" n; pr " nbd_internal_printable_string_list (%s);\n" n | BytesOut _ | BytesPersistOut _ - | Bool _ | Closure _ | Enum _ | Flags _ | Fd _ | Int _ + | Bool _ | Closure _ | Enum _ | Extent64 _ | Flags _ | Fd _ | Int _ | Int64 _ | SizeT _ | SockAddrAndLen _ | UInt _ | UInt32 _ | UInt64 _ | UIntPtr _ -> () ) args;
This extends "print_trace_enter" (near the top).
@@ -766,6 +782,7 @@ let pr " %s=\\\"%%s\\\" %s=%%zu" n count | Closure { cbname } -> pr " %s=%%s" cbname | Enum (n, _) -> pr " %s=%%d" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Flags (n, _) -> pr " %s=0x%%x" n | Fd n | Int n -> pr " %s=%%d" n | Int64 n -> pr " %s=%%\"PRIi64\"" n
And this extends "print_format_string" within "print_trace_enter".
Now here's why I've inserted all those "this extends ..." remarks:
I think your assertion is valid here, *but* it is inconsistent with the extensions to:
- name_of_arg - arg_attr_nonnull - print_arg_list - print_trace_enter
We only deal with Extent64 via the "penetrating pattern match" CBArrayAndLen (Extent64 n, len), in "print_cbarg_list".
(5) Therefore, for consistency, *all* other Extent64 pattern matches should assert that they are unreachable -- not only in "print_format_string", but in those four functions too that I list above.
Clearly, I could be missing some code paths leading to those functions / patterns; I've not tested my proposition. The idea is that the narrowest necessary subset of patterns should be handled. Even if it is easy to "handle" a pattern with just (), it makes the wrong impression.
I will double check which ones can be turned into unreachable assertions.
+++ b/generator/GoLang.ml ... @@ -334,6 +337,7 @@ let | BytesPersistOut (n, len) -> pr ", c_%s, c_%s" n len | Closure { cbname } -> pr ", c_%s" cbname | Enum (n, _) -> pr ", c_%s" n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd n -> pr ", c_%s" n | Flags (n, _) -> pr ", c_%s" n | Int n -> pr ", c_%s" n
(7) I care pretty little for the Go binding, but again: the hunk above that is the first for "GoLang.ml", we extend "go_name_of_arg", and I'd argue we should insert an "assert false" there.
Indeed, it is more than just C.ml that may get this treatment of using unreachable assertions more broadly. Reading ahead[1], I see you mention the idea of splitting this patch across languages. If I do that, then the first patch will add unreachable assertions EVERYWHERE, then later per-language patches will enable what it needs. The end result will be the same, but splitting the work across more patches may be easier to review, so I'm game to try it.
@@ -524,6 +528,18 @@ let copy(ret, s) return ret } + +func copy_extent_array (entries *C.nbd_extent, count C.size_t) []LibnbdExtent { + unsafePtr := unsafe.Pointer(entries) + arrayPtr := (*[1 << 20]C.nbd_extent)(unsafePtr) + slice := arrayPtr[:count:count] + ret := make([]LibnbdExtent, count) + for i := 0; i < int (count); i++ { + ret[i].Length = uint64 (slice[i].length) + ret[i].Flags = uint64 (slice[i].flags) + } + return ret +} ";
List.iter (
The pre-existent copy_uint32_array() function uses the hideous trick at <https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices>, and needlessly so, IMO.
- The trick is (a) hideous because it requires us to use arbitrary sizes such as "1<<30".
- The trick is (b) unnecessary because we don't intend to hang on to the slice indefinitely. We only use it as a means to access the source object. But at the noted URL, the trick is "being sold" with the pitch "to create a Go slice backed by a C array (without copying the original data)" -- and we copy the original data *anyway*! So it's better to use pointer arithmetic IMO.
Separate patch, but reasonable idea.
Regarding the new copy_extent_array(), my additional complaints are:
- whitespace usage before opening parens is inconsistent -- there is no space after "make" and "Pointer".
I'm not very proficient in Go to know whether it has a specific style that I'm violating, or whether we value consistency with our C style, or anything else. But I have no qualms adjusting it as appropriate.
- we cast "count" (a size_t in C) to a Go "int"; similarly the index variable "i" has Go type "int".
(8) So how about this instead (should be split in two: the first part should update copy_uint32_array() in a separate patch, and the second part should be squashed into this patch):
diff --git a/generator/GoLang.ml b/generator/GoLang.ml index 8922812b76a4..37b2240ef5bf 100644 --- a/generator/GoLang.ml +++ b/generator/GoLang.ml @@ -521,22 +521,32 @@ let /* Closures. */
func copy_uint32_array (entries *C.uint32_t, count C.size_t) []uint32 { - ret := make([]uint32, int (count)) - // See https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices - // TODO: Use unsafe.Slice() when we require Go 1.17. - s := (*[1<<30]uint32)(unsafe.Pointer(entries))[:count:count] - copy(ret, s) + ret := make ([]uint32, count) + addr := uintptr (unsafe.Pointer (entries)) + var i uint64 = 0 + for i < uint64 (count) { + ptr := (*C.uint32_t)(unsafe.Pointer (addr)) + + ret[i] = uint32 (*ptr) + + addr += unsafe.Sizeof (*ptr) + i++ + } return ret }
func copy_extent_array (entries *C.nbd_extent, count C.size_t) []LibnbdExtent { - unsafePtr := unsafe.Pointer(entries) - arrayPtr := (*[1 << 20]C.nbd_extent)(unsafePtr) - slice := arrayPtr[:count:count] - ret := make([]LibnbdExtent, count) - for i := 0; i < int (count); i++ { - ret[i].Length = uint64 (slice[i].length) - ret[i].Flags = uint64 (slice[i].flags) + ret := make ([]LibnbdExtent, count) + addr := uintptr (unsafe.Pointer (entries)) + var i uint64 = 0 + for i < uint64 (count) { + ptr := (*C.nbd_extent)(unsafe.Pointer (addr)) + + ret[i].Length = uint64 ((*ptr).length) + ret[i].Flags = uint64 ((*ptr).flags) + + addr += unsafe.Sizeof (*ptr) + i++ } return ret }
(This passes make and make-check, when the tree is checked out at your patch#7, and this is applied on top.)
Seems reasonable.
Back to your patch:
On 5/25/23 15:00, Eric Blake wrote:
@@ -537,6 +553,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, _) -> pr "%s []uint32" n; + | CBArrayAndLen (Extent64 n, _) -> + pr "%s []LibnbdExtent" n; | CBBytesIn (n, len) -> pr "%s []byte" n; | CBInt n -> @@ -563,6 +581,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "%s *C.uint32_t, %s C.size_t" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "%s *C.nbd_extent, %s C.size_t" n count | CBBytesIn (n, len) -> pr "%s unsafe.Pointer, %s C.size_t" n len | CBInt n -> @@ -605,6 +625,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "copy_uint32_array (%s, %s)" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "copy_extent_array (%s, %s)" n count | CBBytesIn (n, len) -> pr "C.GoBytes (%s, C.int (%s))" n len | CBInt n -> @@ -756,6 +778,8 @@ let match cbarg with | CBArrayAndLen (UInt32 n, count) -> pr "uint32_t *%s, size_t %s" n count + | CBArrayAndLen (Extent64 n, count) -> + pr "nbd_extent *%s, size_t %s" n count | CBBytesIn (n, len) -> pr "void *%s, size_t %s" n len | CBInt n ->
OK.
Quoting the rest of the patch out of order, so I can leave the OCaml stuff for the end (this is a *terribly* large patch, when the contexts are increased suitably for reviewing, so I really wish we could split it up somehow):
[1] Yes, using unreachable notations everywhere (regardless of context), then doing per-language patching to enable the code that makes sense, should work.
diff --git a/golang/handle.go b/golang/handle.go index 5fe4ed4f..0b126005 100644 --- a/golang/handle.go +++ b/golang/handle.go @@ -58,6 +58,12 @@ func (h *Libnbd) String() string { return "&Libnbd{}" }
+/* Used for block status callback. */ +type LibnbdExtent struct { + Length uint64 // length of the extent + Flags uint64 // flags describing properties of the extent +} + /* All functions (except Close) return ([result,] LibnbdError). */ type LibnbdError struct { Op string // operation which failed
OK.
diff --git a/generator/Python.ml b/generator/Python.ml index c81878de..ccac5fed 100644 --- a/generator/Python.ml +++ b/generator/Python.ml @@ -158,6 +158,7 @@ let let print_python_closure_wrapper { cbname; cbargs } = pr "/* Wrapper for %s callback. */\n" cbname; pr "static int\n"; + pr "__attribute__ ((unused)) /* XXX temporary hack */\n"; pr "%s_wrapper " cbname; C.print_cbarg_list ~wrap:true cbargs; pr "\n";
OK, this is as announced in the commit message.
@@ -170,6 +171,7 @@ let List.iter ( function | CBArrayAndLen (UInt32 n, _) + | CBArrayAndLen (Extent64 n, _) | CBBytesIn (n, _) | CBMutable (Int n) -> pr " PyObject *py_%s = NULL;\n" n
OK, this is responsible for generating:
PyObject *py_entries = NULL;
@@ -187,6 +189,16 @@ let pr " if (!py_e_%s) { PyErr_PrintEx (0); goto out; }\n" n; pr " PyList_SET_ITEM (py_%s, i_%s, py_e_%s);\n" n n n; pr " }\n" + | CBArrayAndLen (Extent64 n, len) -> + pr " py_%s = PyList_New (%s);\n" n len; + pr " size_t i_%s;\n" n;
(9) Under "CBArrayAndLen (UInt32 n, len)", we also check whether the New call succeeds; I think it should not be omitted here either:
pr " if (!py_%s) { PyErr_PrintEx (0); goto out; }\n" n;
Good catch.
producing (in the preexistent "extent_wrapper" function):
if (!py_entries) { PyErr_PrintEx (0); goto out; }
+ pr " for (i_%s = 0; i_%s < %s; ++i_%s) {\n" n n len n; + pr " PyObject *py_e_%s = Py_BuildValue (\"OO\",\n" n; + pr " PyLong_FromUnsignedLong (%s[i_%s].length),\n" n n; + pr " PyLong_FromUnsignedLong (%s[i_%s].flags));\n" n n;
We have two issues here:
(10) We should call PyLong_FromUnsignedLongLong(), not PyLong_FromUnsignedLong(). On an ILP32 platform, the latter will not do the right thing.
Indeed. Will fix.
(11) PyLong_FromWhatever() returns a new reference, and so it can fail due to memory exhaustion (minimally, I think). That means zero, one, or two of our PyLong_FromUnsignedLongLong() calls can fail, and return NULL.
We call Py_BuildValue() with an "OO" specification. The documentation says:
Pass a Python object untouched (except for its reference count, which is incremented by one). If the object passed in is a NULL pointer, it is assumed that this was caused because the call producing the argument found an error and set an exception. Therefore, Py_BuildValue() will return NULL but won’t raise an exception. If no exception has been raised yet, SystemError is set.
Now, I don't think PyLong_FromUnsignedLongLong() sets an exception when it fails (and returns NULL), but that should be fine -- at least Py_BuildValue() will raise SystemError.
If PyLong_FromUnsignedLongLong() returns NULL, it set an error.
The problem is that we have three possible ways to leak memory:
- If we pass two NULLs to "OO" (after both our PyLong_FromUnsignedLongLong() calls fail), there's no leak. Py_BuildValue() fails in response too, and nothing is leaked.
- If we pass exactly one NULL to "OO" (after one PyLong_FromUnsignedLongLong() succeeds, but the other fails), then Py_BuildValue() fails again (as expected), but we leak the PyLongObject that we *did* successfully create.
- If we pass two non-NULLs to "OO" (after both PyLong_FromUnsignedLongLong() succeed), and Py_BuildValue() fails because *it* cannot allocate further memory, then we leak *both* PyLongObjects.
- If we pass two non-NULLs to "OO" (after both PyLong_FromUnsignedLongLong() succeed), and Py_BuildValue() *succeds*, then we leak both PyLongObjects again: PyLong_FromUnsignedLongLong() returns a new reference, but when Py_BuildValue() succeeds, it adds a *new* reference (it does not only take ownership). The actual leak will happen later, when the object just constructed with Py_BuildValue() is torn down.
So, we should implement this in a much more "staged" way. Shown as a diff on top of the code generated by your patch, covering points (9) through (11) (even (12) below):
I'll double-check the Py_BuildValue() docs; I seem to recall that I could use something other than "OO" as the format string in order to pass in C values directly, rather than having to pre-construct intermediate python objects myself, and doing that would be even better at avoiding leak issues. (Trimming the rest of your Python observations until I can focus on that aspect later, as they may be rendered moot if I can do the conversion smarter in the first place)
Back to your patch: ...
+++ b/ocaml/nbd-c.h @@ -62,6 +62,7 @@ extern void nbd_internal_ocaml_raise_closed (const char *func) Noreturn;
extern const char **nbd_internal_ocaml_string_list (value); extern value nbd_internal_ocaml_alloc_i64_from_u32_array (uint32_t *, size_t); +extern value nbd_internal_ocaml_alloc_extent64_array (nbd_extent *, size_t); extern void nbd_internal_unix_sockaddr_to_sa (value, struct sockaddr_storage *, socklen_t *); extern void nbd_internal_ocaml_exception_in_wrapper (const char *, value);
OK.
diff --git a/generator/OCaml.ml b/generator/OCaml.ml index edb81f25..ea3b044a 100644 --- a/generator/OCaml.ml +++ b/generator/OCaml.ml @@ -44,6 +44,7 @@ and | Closure { cbargs } -> sprintf "(%s)" (ocaml_closuredecl_to_string cbargs) | Enum (_, { enum_prefix }) -> sprintf "%s.t" enum_prefix + | Extent64 _ -> "extent" | Fd _ -> "Unix.file_descr" | Flags (_, { flag_prefix }) -> sprintf "%s.t list" flag_prefix | Int _ -> "int" @@ -102,6 +103,7 @@ let | BytesPersistOut (n, len) -> n | Closure { cbname } -> cbname | Enum (n, _) -> n + | Extent64 n -> n | Fd n -> n | Flags (n, _) -> n | Int n -> n
(14) Could we use "assert false" with these too?
Same as before. I'm sold on the idea of the intro patch using assert false everywhere, then the per-language followup patches adding code only for what is needed.
@@ -147,6 +149,9 @@ let
type cookie = int64
+type extent = int64 * int64 +(** Length and flags of an extent in block_status_64 callback. *) + ";
List.iter (
(15) Can we add a note here that negative values are not permitted? (See my further comments below.)
Yes. Hmm, right now, I think we're implicitly relying on a server never exceeding off_t in its advertised size. I have intentions to actually formalize that in the NBD spec, but have not done so yet; so I guess there is the theoretical case of a server that advertises an unsigned 64-bit size and makes libnbd blow up where we don't expect that. I should probably prepare a pre-requisite patch where the state machine transfers to DEAD if the server ever advertises a too-large size, so that this code can then assert (rather than just comment) that we aren't dealing with negative values.
@@ -268,6 +273,7 @@ let exception Error of string * Unix.error option exception Closed of string type cookie = int64 +type extent = int64 * int64
(* Give the exceptions names so that they can be raised from the C code. *) let () = @@ -498,7 +504,8 @@ let let argnames = List.map ( function - | CBArrayAndLen (UInt32 n, _) | CBBytesIn (n, _) + | CBArrayAndLen (UInt32 n, _) | CBArrayAndLen (Extent64 n, _) + | CBBytesIn (n, _) | CBInt n | CBInt64 n | CBMutable (Int n) | CBString n | CBUInt n | CBUInt64 n -> n ^ "v"
This is inside "print_ocaml_closure_wrapper"; OK.
(16) You can contract the new pattern as follows:
| CBArrayAndLen ((UInt32 n | Extent64 n), _)
Cool; I'm still polishing my OCaml skills.
@@ -531,6 +538,14 @@ let pr "%s %s,\n" indent n; pr "%s %s\n" indent count; pr "%s);\n" indent + | CBArrayAndLen (Extent64 n, count) -> + pr " %sv = " n; + let fncol = output_column () in + let indent = spaces fncol in + pr "nbd_internal_ocaml_alloc_extent64_array (\n"; + pr "%s %s,\n" indent n; + pr "%s %s\n" indent count; + pr "%s);\n" indent | CBBytesIn (n, len) -> pr " %sv = caml_alloc_initialized_string (%s, %s);\n" n len n | CBInt n | CBUInt n ->
Seems OK superficially; matches the 'CBArrayAndLen (UInt32 n, count)' case, with only the helper function differing.
@@ -554,7 +569,7 @@ let
List.iter ( function - | CBArrayAndLen (UInt32 _, _) + | CBArrayAndLen (_, _) | CBBytesIn _ | CBInt _ | CBInt64 _ @@ -563,7 +578,7 @@ let | CBUInt64 _ -> () | CBMutable (Int n) -> pr " *%s = Int_val (Field (%sv, 0));\n" n n - | CBArrayAndLen _ | CBMutable _ -> assert false + | CBMutable _ -> assert false ) cbargs;
pr " if (Is_exception_result (rv)) {\n";
(17) We're still inside "print_ocaml_closure_wrapper", so I think the patterns here should follow the patterns seen in the "argnames" definition:
| CBArrayAndLen (UInt32 n, _) | CBArrayAndLen (Extent64 n, _)
or even better (again)
| CBArrayAndLen ((UInt32 _ | Extent64 _), _)
IOW I think "CBArrayAndLen (_, _)" is too generic / inconsistent with previously added code. We should handle the Extent64 constructor, and not touch the "CBArrayAndLen _" fallback.
Fair enough. I was relying on the compiler telling me that it didn't encounter any uncovered branches, but using a consistent style (even if it means longer source code) doesn't hurt.
@@ -578,6 +593,7 @@ let pr "}\n"; pr "\n"; pr "static int\n"; + pr "__attribute__ ((unused)) /* XXX temporary hack */\n"; pr "%s_wrapper " cbname; C.print_cbarg_list ~wrap:true cbargs; pr "\n";
OK, announced in the commit message.
@@ -695,6 +711,7 @@ let pr " %s_callback.free = free_user_data;\n" cbname | Enum (n, { enum_prefix }) -> pr " int %s = %s_val (%sv);\n" n enum_prefix n + | Extent64 _ -> assert false (* only used in extent64_closure *) | Fd n -> pr " /* OCaml Unix.file_descr is just an int, at least on Unix. */\n"; pr " int %s = Int_val (%sv);\n" n n
OK, this is in "print_ocaml_binding".
@@ -780,6 +797,7 @@ let | BytesPersistOut _ | Closure _ | Enum _ + | Extent64 _ | Fd _ | Flags _ | Int _
This is still in "print_ocaml_binding", under comment "Any parameters which need to be freed".
(18) For consistency I'd suggest "assert false" here as well.
diff --git a/ocaml/helpers.c b/ocaml/helpers.c index 3361a696..09666daf 100644 --- a/ocaml/helpers.c +++ b/ocaml/helpers.c @@ -133,6 +133,26 @@ nbd_internal_ocaml_alloc_i64_from_u32_array (uint32_t *a, size_t len) CAMLreturn (rv); }
+value +nbd_internal_ocaml_alloc_extent64_array (nbd_extent *a, size_t len) +{ + CAMLparam0 (); + CAMLlocal3 (s, v, rv); + size_t i; + + rv = caml_alloc (len, 0); + for (i = 0; i < len; ++i) { + s = caml_alloc (2, 0); + v = caml_copy_int64 (a[i].length); + Store_field (s, 0, v); + v = caml_copy_int64 (a[i].flags); + Store_field (s, 1, v); + Store_field (rv, i, s); + } + + CAMLreturn (rv); +} + /* Convert a Unix.sockaddr to a C struct sockaddr. */ void nbd_internal_unix_sockaddr_to_sa (value sockaddrv,
(19) I'd suggest the following addition:
diff --git a/ocaml/helpers.c b/ocaml/helpers.c index 09666dafa7d1..db652943141d 100644 --- a/ocaml/helpers.c +++ b/ocaml/helpers.c @@ -25,6 +25,7 @@ #include <string.h> #include <sys/socket.h> #include <assert.h> +#include <inttypes.h>
#include <caml/alloc.h> #include <caml/callback.h> @@ -140,6 +141,16 @@ nbd_internal_ocaml_alloc_extent64_array (nbd_extent *a, size_t len) CAMLlocal3 (s, v, rv); size_t i;
+ for (i = 0; i < len; ++i) + if (a[i].length > INT64_MAX || a[i].flags > INT64_MAX) { + char errmsg[256]; + + snprintf (errmsg, sizeof errmsg, + "%s: extent[%zu] = { .length = %"PRIu64", .flags = %"PRIu64"}", + __func__, i, a[i].length, a[i].flags); + caml_failwith (errmsg); + }
For a[i].length, I think I can just use an assertion, instead of caml_failwith(), if I can guarantee with a pre-requisite patch that we will never proceed with a server that sends larger than a 63-bit size. For a[i].flags, things are trickier (as the client really CAN send a 64-bit value). Maybe this is a question for Rich on whether I've picked the best OCaml types.
+ rv = caml_alloc (len, 0); for (i = 0; i < len; ++i) { s = caml_alloc (2, 0);
*However*, considering the nbd_internal_ocaml_alloc_extent64_array() call site, in the generated extent64_wrapper_locked() function [ocaml/nbd-c.c]:
/* Wrapper for extent64 callback. */ static int extent64_wrapper_locked (void *user_data, const char *metacontext, uint64_t offset, nbd_extent *entries, size_t nr_entries, int *error) { CAMLparam0 (); CAMLlocal4 (metacontextv, offsetv, entriesv, errorv); CAMLlocal2 (exn, rv); const struct user_data *data = user_data; int r; value args[4];
metacontextv = caml_copy_string (metacontext); offsetv = caml_copy_int64 (offset); entriesv = nbd_internal_ocaml_alloc_extent64_array ( entries, nr_entries ); errorv = caml_alloc_tuple (1); Store_field (errorv, 0, Val_int (*error)); args[0] = metacontextv; args[1] = offsetv; args[2] = entriesv; args[3] = errorv; rv = caml_callbackN_exn (data->fnv, 4, args); *error = Int_val (Field (errorv, 0)); if (Is_exception_result (rv)) { nbd_internal_ocaml_exception_in_wrapper ("extent64", rv); CAMLreturnT (int, -1); }
r = Int_val (rv); assert (r >= 0); CAMLreturnT (int, r); }
I'm not sure if raising an OCaml exception like this, in an *inner* C function, is appropriate. caml_failwith() may only be suitable for C functions *directly* called by the OCaml runtime.
If that's the case, then we should propagate the error from nbd_internal_ocaml_alloc_extent64_array() differently. We might have to change the signature of nbd_internal_ocaml_alloc_extent64_array(), and then the generated call site (i.e., extent64_wrapper_locked()) must take a potential failure into account.
This is really hard to reason about, as the only "prior art", namely extent_wrapper_locked() -> nbd_internal_ocaml_alloc_i64_from_u32_array(), does not face such a conundrum. There we took the easy way: we convert from uint32_t to OCaml's int64, which cannot overflow.
My intent (although we may still need to get the OCaml-specific split patch to match that) is that length will always fit in OCaml's int64 (because it is a 63-bit value); while flags needs all 64 bits represented (whether signed or unsigned may depend on how easy or hard it is to do bit-twiddling on a 64-bit value in OCaml).
Again, this is a huge patch; I've now spent almost a full workday on reviewing it. I suggest splitting each language to its own patch; thus, we'd get at least 5 patches (core generator changes + 4 languages). If necessary to placate the OCaml compiler, I'd try introducing bare-bones place-holder code at first in the generator for each language, in the first ("core") patch, and then elaborate on each language in a separate patch (replacing the placeholders with actual code).
Laszlo
-- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
On Wed, Jun 07, 2023 at 04:23:27PM +0200, Laszlo Ersek wrote:
+++ b/generator/GoLang.ml @@ -524,6 +528,18 @@ let copy(ret, s) return ret } + +func copy_extent_array (entries *C.nbd_extent, count C.size_t) []LibnbdExtent { + unsafePtr := unsafe.Pointer(entries) + arrayPtr := (*[1 << 20]C.nbd_extent)(unsafePtr) + slice := arrayPtr[:count:count] + ret := make([]LibnbdExtent, count) + for i := 0; i < int (count); i++ { + ret[i].Length = uint64 (slice[i].length) + ret[i].Flags = uint64 (slice[i].flags) + } + return ret +} ";
List.iter (
The pre-existent copy_uint32_array() function uses the hideous trick at <https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices>, and needlessly so, IMO.
- The trick is (a) hideous because it requires us to use arbitrary sizes such as "1<<30".
- The trick is (b) unnecessary because we don't intend to hang on to the slice indefinitely. We only use it as a means to access the source object. But at the noted URL, the trick is "being sold" with the pitch "to create a Go slice backed by a C array (without copying the original data)" -- and we copy the original data *anyway*! So it's better to use pointer arithmetic IMO.
Regarding the new copy_extent_array(), my additional complaints are:
- whitespace usage before opening parens is inconsistent -- there is no space after "make" and "Pointer".
- we cast "count" (a size_t in C) to a Go "int"; similarly the index variable "i" has Go type "int".
(8) So how about this instead (should be split in two: the first part should update copy_uint32_array() in a separate patch, and the second part should be squashed into this patch):
diff --git a/generator/GoLang.ml b/generator/GoLang.ml index 8922812b76a4..37b2240ef5bf 100644 --- a/generator/GoLang.ml +++ b/generator/GoLang.ml @@ -521,22 +521,32 @@ let /* Closures. */
func copy_uint32_array (entries *C.uint32_t, count C.size_t) []uint32 { - ret := make([]uint32, int (count)) - // See https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices - // TODO: Use unsafe.Slice() when we require Go 1.17. - s := (*[1<<30]uint32)(unsafe.Pointer(entries))[:count:count] - copy(ret, s) + ret := make ([]uint32, count) + addr := uintptr (unsafe.Pointer (entries)) + var i uint64 = 0 + for i < uint64 (count) { + ptr := (*C.uint32_t)(unsafe.Pointer (addr)) + + ret[i] = uint32 (*ptr) + + addr += unsafe.Sizeof (*ptr) + i++ + }
Pointer arithmetic is straightforward, but not necessarily the best use of hardware. For all I know, the copy() routine is vectorized, and therefore can achieve better performance by copying multiple bytes at once using better hardware primitives. So there may still be an advantage to using the hideous hack for the sake of performance. But as I have not bothered to benchmark that claim, I'm happy to change back to linear copying.
return ret }
func copy_extent_array (entries *C.nbd_extent, count C.size_t) []LibnbdExtent { - unsafePtr := unsafe.Pointer(entries) - arrayPtr := (*[1 << 20]C.nbd_extent)(unsafePtr) - slice := arrayPtr[:count:count] - ret := make([]LibnbdExtent, count) - for i := 0; i < int (count); i++ { - ret[i].Length = uint64 (slice[i].length) - ret[i].Flags = uint64 (slice[i].flags) + ret := make ([]LibnbdExtent, count) + addr := uintptr (unsafe.Pointer (entries)) + var i uint64 = 0 + for i < uint64 (count) { + ptr := (*C.nbd_extent)(unsafe.Pointer (addr)) + + ret[i].Length = uint64 ((*ptr).length) + ret[i].Flags = uint64 ((*ptr).flags) + + addr += unsafe.Sizeof (*ptr) + i++
That sentiment is further strengthened by the fact that neither my original proposal nor your replacement can use copy() for the struct purposes. Even if copy() can speed up []uint32 copies, I'd rather let the Go compiler worry about vectorizing the loop if we can't find an obvious library function that already takes advantage of hardware vectors. -- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
On Mon, Jul 24, 2023 at 07:04:15PM -0500, Eric Blake wrote:
On Wed, Jun 07, 2023 at 04:23:27PM +0200, Laszlo Ersek wrote:
+++ b/generator/GoLang.ml @@ -524,6 +528,18 @@ let copy(ret, s) return ret } + +func copy_extent_array (entries *C.nbd_extent, count C.size_t) []LibnbdExtent { + unsafePtr := unsafe.Pointer(entries) + arrayPtr := (*[1 << 20]C.nbd_extent)(unsafePtr) + slice := arrayPtr[:count:count] + ret := make([]LibnbdExtent, count) + for i := 0; i < int (count); i++ { + ret[i].Length = uint64 (slice[i].length) + ret[i].Flags = uint64 (slice[i].flags) + } + return ret +} ";
List.iter (
The pre-existent copy_uint32_array() function uses the hideous trick at <https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices>, and needlessly so, IMO.
- The trick is (a) hideous because it requires us to use arbitrary sizes such as "1<<30".
- The trick is (b) unnecessary because we don't intend to hang on to the slice indefinitely. We only use it as a means to access the source object. But at the noted URL, the trick is "being sold" with the pitch "to create a Go slice backed by a C array (without copying the original data)" -- and we copy the original data *anyway*! So it's better to use pointer arithmetic IMO.
Regarding the new copy_extent_array(), my additional complaints are:
- whitespace usage before opening parens is inconsistent -- there is no space after "make" and "Pointer".
- we cast "count" (a size_t in C) to a Go "int"; similarly the index variable "i" has Go type "int".
(8) So how about this instead (should be split in two: the first part should update copy_uint32_array() in a separate patch, and the second part should be squashed into this patch):
diff --git a/generator/GoLang.ml b/generator/GoLang.ml index 8922812b76a4..37b2240ef5bf 100644 --- a/generator/GoLang.ml +++ b/generator/GoLang.ml @@ -521,22 +521,32 @@ let /* Closures. */
func copy_uint32_array (entries *C.uint32_t, count C.size_t) []uint32 { - ret := make([]uint32, int (count)) - // See https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices - // TODO: Use unsafe.Slice() when we require Go 1.17. - s := (*[1<<30]uint32)(unsafe.Pointer(entries))[:count:count] - copy(ret, s) + ret := make ([]uint32, count) + addr := uintptr (unsafe.Pointer (entries)) + var i uint64 = 0 + for i < uint64 (count) { + ptr := (*C.uint32_t)(unsafe.Pointer (addr)) + + ret[i] = uint32 (*ptr) + + addr += unsafe.Sizeof (*ptr) + i++ + }
Pointer arithmetic is straightforward, but not necessarily the best use of hardware. For all I know, the copy() routine is vectorized, and therefore can achieve better performance by copying multiple bytes at once using better hardware primitives. So there may still be an advantage to using the hideous hack for the sake of performance. But as I have not bothered to benchmark that claim, I'm happy to change back to linear copying.
On the other hand, that would be reverting Nir's patch 6725fa0e, which introduced that notation in the first place, but also without benchmarking. Nir, any thoughts before we touch this code? -- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
When extended headers are in use, the server is required to use a response that can include 64-bit extent lengths and flags (even if it chooses to keep the extent length under 4G, and even for metacontexts like "base:allocation" where the flags never need more than 32 bits). For maximum flexibility, we are better off storing 64-bit data internally, with a goal of letting the client's 32-bit interface work as much as possible, and for a future API addition of a 64-bit interface to work even with older servers that only give 32-bit results. For backwards compatibility, a client that never negotiates a 64-bit status context can be handled without errors by truncating any 64-bit lengths down to just under 4G; so the old 32-bit interface will continue to work in most cases. A server should not advertise a metacontext that requires flags larger than 32 bits unless the client negotiated extended headers; but if such a client still tries to use the 32-bit interface, the code now reports EOVERFLOW for a server response with a flags value that cannot be represented without using the future API for 64-bit extent queries. Note that the existing 32-bit nbd_block_status() API is now slightly slower, particularly when talking with a server that lacks extended headers: we are doing two size conversions (32 to 64 internally, 64 back to 32 to the user). But this speed penalty is likely in the noise compared to the network delays, and ideally clients will switch over to the new 64-bit interfaces as more and more servers start supporting extended headers. One of the trickier aspects of this patch is auditing that both the user's extent and our malloc'd shim get cleaned up once on all possible paths, so that there is neither a leak nor a double free. Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 8 +++- generator/states-reply-structured.c | 30 ++++++++----- lib/handle.c | 2 +- lib/rw.c | 70 ++++++++++++++++++++++++++++- 4 files changed, 95 insertions(+), 15 deletions(-) diff --git a/lib/internal.h b/lib/internal.h index e4e21a4d..11186e24 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -80,7 +80,7 @@ struct export { struct command_cb { union { - nbd_extent_callback extent; + nbd_extent64_callback extent; nbd_chunk_callback chunk; nbd_list_callback list; nbd_context_callback context; @@ -303,7 +303,11 @@ struct nbd_handle { size_t querynum; /* When receiving block status, this is used. */ - uint32_t *bs_entries; + union { + char *storage; /* malloc's view */ + nbd_extent *normal; /* Our 64-bit preferred internal form; n slots */ + uint32_t *narrow; /* 32-bit NBD_REPLY_TYPE_BLOCK_STATUS form; n*2 slots */ + } bs_entries; /* Commands which are waiting to be issued [meaning the request * packet is sent to the server]. This is used as a simple linked diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index c53aed7b..784adb78 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -436,6 +436,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_HOLE: REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: struct command *cmd = h->reply_cmd; uint32_t length; + uint32_t count; switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; @@ -450,15 +451,19 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (length >= 12); length -= sizeof h->sbuf.reply.payload.bs_hdr; + count = length / (2 * sizeof (uint32_t)); - free (h->bs_entries); - h->bs_entries = malloc (length); - if (h->bs_entries == NULL) { + /* Read raw data into a subset of h->bs_entries, then expand it + * into place later during byte-swapping. + */ + free (h->bs_entries.storage); + h->bs_entries.storage = malloc (count * sizeof *h->bs_entries.normal); + if (h->bs_entries.storage == NULL) { SET_NEXT_STATE (%.DEAD); set_error (errno, "malloc"); return 0; } - h->rbuf = h->bs_entries; + h->rbuf = h->bs_entries.narrow; h->rlen = length; SET_NEXT_STATE (%RECV_BS_ENTRIES); } @@ -470,6 +475,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: uint32_t count; size_t i; uint32_t context_id; + uint32_t *raw; switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; @@ -483,17 +489,21 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); - assert (h->bs_entries); + assert (h->bs_entries.normal); assert (length >= 12); assert (h->meta_valid); count = (length - sizeof h->sbuf.reply.payload.bs_hdr) / - sizeof *h->bs_entries; + (2 * sizeof (uint32_t)); /* Need to byte-swap the entries returned, but apart from that we - * don't validate them. + * don't validate them. Reverse order is essential, since we are + * expanding in-place from narrow to wider type. */ - for (i = 0; i < count; ++i) - h->bs_entries[i] = be32toh (h->bs_entries[i]); + raw = h->bs_entries.narrow; + for (i = count; i-- > 0; ) { + h->bs_entries.normal[i].flags = be32toh (raw[i * 2 + 1]); + h->bs_entries.normal[i].length = be32toh (raw[i * 2]); + } /* Look up the context ID. */ context_id = be32toh (h->sbuf.reply.payload.bs_hdr.context_id); @@ -507,7 +517,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: if (CALL_CALLBACK (cmd->cb.fn.extent, h->meta_contexts.ptr[i].name, cmd->offset, - h->bs_entries, count, + h->bs_entries.normal, count, &error) == -1) if (cmd->error == 0) cmd->error = error ? error : EPROTO; diff --git a/lib/handle.c b/lib/handle.c index 0f11bee5..fba6d1c4 100644 --- a/lib/handle.c +++ b/lib/handle.c @@ -130,7 +130,7 @@ nbd_close (struct nbd_handle *h) nbd_unlocked_clear_debug_callback (h); string_vector_empty (&h->querylist); - free (h->bs_entries); + free (h->bs_entries.storage); nbd_internal_reset_size_and_flags (h); for (i = 0; i < h->meta_contexts.len; ++i) free (h->meta_contexts.ptr[i].name); diff --git a/lib/rw.c b/lib/rw.c index 8b2bd4cc..58b05a4b 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -42,6 +42,61 @@ wait_for_command (struct nbd_handle *h, int64_t cookie) return r == -1 ? -1 : 0; } +/* Convert from 64-bit to 32-bit extent callback. */ +static int +nbd_convert_extent (void *data, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t nr_entries, int *error) +{ + nbd_extent_callback *cb = data; + uint32_t *array = malloc (nr_entries * 2 * sizeof *array); + size_t i; + int ret; + bool fail = false; + + if (array == NULL) { + set_error (*error = errno, "malloc"); + return -1; + } + + for (i = 0; i < nr_entries; i++) { + array[i * 2] = entries[i].length; + array[i * 2 + 1] = entries[i].flags; + /* If an extent is larger than 32 bits, silently truncate the rest + * of the server's response; the client can then make progress + * instead of needing to see failure. Rather than track the + * connection's alignment, just clamp the large extent to 4G-64M. + * However, if flags doesn't fit in 32 bits, it's better to inform + * the caller of an EOVERFLOW failure. + * + * Technically, a server with 64-bit answers is non-compliant if + * the client did not negotiate extended headers - contexts that + * include 64-bit flags should not have been negotiated in that + * case. + */ + if (entries[i].length > UINT32_MAX) { + array[i++ * 2] = -MAX_REQUEST_SIZE; + break; + } + if (entries[i].flags > UINT32_MAX) { + *error = EOVERFLOW; + fail = true; + break; + } + } + + ret = CALL_CALLBACK (*cb, metacontext, offset, array, i * 2, error); + free (array); + return fail ? -1 : ret; +} + +static void +nbd_convert_extent_free (void *data) +{ + nbd_extent_callback *cb = data; + FREE_CALLBACK (*cb); + free (cb); +} + /* Issue a read command and wait for the reply. */ int nbd_unlocked_pread (struct nbd_handle *h, void *buf, @@ -487,12 +542,23 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, nbd_completion_callback *completion, uint32_t flags) { - struct command_cb cb = { .fn.extent = *extent, + nbd_extent_callback *shim = malloc (sizeof *shim); + struct command_cb cb = { .fn.extent.callback = nbd_convert_extent, + .fn.extent.user_data = shim, + .fn.extent.free = nbd_convert_extent_free, .completion = *completion }; + if (shim == NULL) { + set_error (errno, "malloc"); + return -1; + } + *shim = *extent; + SET_CALLBACK_TO_NULL (*extent); + if (h->strict & LIBNBD_STRICT_COMMANDS) { if (!h->structured_replies) { set_error (ENOTSUP, "server does not support structured replies"); + FREE_CALLBACK (cb.fn.extent); return -1; } @@ -500,11 +566,11 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, set_error (ENOTSUP, "did not negotiate any metadata contexts, " "either you did not call nbd_add_meta_context before " "connecting or the server does not support it"); + FREE_CALLBACK (cb.fn.extent); return -1; } } - SET_CALLBACK_TO_NULL (*extent); SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb); -- 2.40.1
Fair warning: this patch has made me both very excited and very uncomfortable; that's the reason my style might come across as needlessly "combative". I don't mean to be combative. It's just how my thoughts have come out, and (unfortunately) I need to go pick up my son now, and don't have time for further rounds of polishing my style. I apologize in advance. Again, it's a consequence of me caring a bit too much perhaps, and getting worked up. On 5/25/23 15:00, Eric Blake wrote:
When extended headers are in use, the server is required to use a response that can include 64-bit extent lengths and flags (even if it chooses to keep the extent length under 4G, and even for metacontexts like "base:allocation" where the flags never need more than 32 bits).
Yes. Per spec, negotiating NBD_OPT_EXTENDED_HEADERS is equivalent to the server sending NBD_REPLY_TYPE_BLOCK_STATUS_EXT, as opposed to NBD_REPLY_TYPE_BLOCK_STATUS.
For maximum flexibility, we are better off storing 64-bit data internally, with a goal of letting the client's 32-bit interface work as much as possible, and for a future API addition of a 64-bit interface to work even with older servers that only give 32-bit results.
I agree that a 64-bit-only internal representation makes sense. Once corresponding code part (quoting out of order for discussion's sake) is:
diff --git a/lib/internal.h b/lib/internal.h index e4e21a4d..11186e24 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -80,7 +80,7 @@ struct export {
struct command_cb { union { - nbd_extent_callback extent; + nbd_extent64_callback extent; nbd_chunk_callback chunk; nbd_list_callback list; nbd_context_callback context;
and that looks good to me.
For backwards compatibility, a client that never negotiates a 64-bit status context can be handled without errors by truncating any 64-bit lengths down to just under 4G; so the old 32-bit interface will continue to work in most cases.
I absolutely don't follow this. The spec is clear. NBD_OPT_EXTENDED_HEADERS corresponds to NBD_REPLY_TYPE_BLOCK_STATUS_EXT, and (!NBD_OPT_EXTENDED_HEADERS) corresponds to NBD_REPLY_TYPE_BLOCK_STATUS. If NBD_OPT_EXTENDED_HEADERS is negotiated (which means both client and server agree to it), then the client interface is 64-bit clean, so there's no need to truncate anything. If NBD_OPT_EXTENDED_HEADERS is *not* negotiated, then our internal 64-bit clean representation will never be filled with information that doesn't fit into 32-bits (due to the reply type being restricted to NBD_REPLY_TYPE_BLOCK_STATUS), so there is again no need to (observably) truncate anything (technically there would be a uint64_t to uint32_t conversion, but it would be value-preserving). The spec also says that, if a client wants to query a particular metadata context that is known to use 64-bit values, then the client must first negotiate NBD_OPT_EXTENDED_HEADERS.
A server should not advertise a metacontext that requires flags larger than 32 bits unless the client negotiated extended headers; but if such a client still tries to use the 32-bit interface, the code now reports EOVERFLOW for a server response with a flags value that cannot be represented without using the future API for 64-bit extent queries.
Again, I don't follow. Whenever the client issues an NBD_CMD_BLOCK_STATUS command (which can only occur after negotiation): - the client must have set at least one metacontext with NBD_OPT_SET_META_CONTEXT during negotiation, - the client may or may not have agreed upon with the server on NBD_OPT_EXTENDED_HEADERS. Therefore, when the server receives the NBD_CMD_BLOCK_STATUS command, it can immediately decide whether there is *any* configured metacontext for which *lack* of extended headers could be too restrictive. If only 32-bit metacontexts have been requested, this is not a problem. If ext-hdrs have been negotiated, also not a problem. If there is at least one 64-bit metacontext (regarding *either* status flags *or* lengths) *and* ext-hdrs have not been negotiated, the server should immediately fail (reject) the request. Now, peeking a bit forward, I can see patch#10: "api: Add [aio_]nbd_block_status_64". That means we can reject a 32-bit-only nbd_block_status() API call on the client side immediately, if the client has negotiated ext-hdrs. That is, if the client is smart enough to ask for extended headers, it can be smart enough to use the 64-bit API, and then it can accept everything the server throws at it. If the client never even asks for extended headers, then it is permitted to stick with the 32-bit-only API, but then the server must never respond with NBD_REPLY_TYPE_BLOCK_STATUS_EXT [*], so actually 64-bit values are never going to be present in our internal representation. [*] If it still does, then the server is buggy, and we should realize that as soon as we parse the generic reply magic, or the specific reply type code. I simply fail to see the *possibility* truncating the internal 64-bit representation to 32-bit such that it would make a difference for the caller. Speaking in terms of code:
diff --git a/lib/rw.c b/lib/rw.c index 8b2bd4cc..58b05a4b 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -42,6 +42,61 @@ wait_for_command (struct nbd_handle *h, int64_t cookie) return r == -1 ? -1 : 0; }
+/* Convert from 64-bit to 32-bit extent callback. */ +static int +nbd_convert_extent (void *data, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t nr_entries, int *error) +{ + nbd_extent_callback *cb = data; + uint32_t *array = malloc (nr_entries * 2 * sizeof *array);
Side question: what guarantees that the multiplication won't overflow? If we have a guarantee, we should document it, and (preferably) assert it.
+ size_t i; + int ret; + bool fail = false; + + if (array == NULL) { + set_error (*error = errno, "malloc"); + return -1; + } + + for (i = 0; i < nr_entries; i++) { + array[i * 2] = entries[i].length; + array[i * 2 + 1] = entries[i].flags; + /* If an extent is larger than 32 bits, silently truncate the rest + * of the server's response; the client can then make progress + * instead of needing to see failure. Rather than track the + * connection's alignment, just clamp the large extent to 4G-64M. + * However, if flags doesn't fit in 32 bits, it's better to inform + * the caller of an EOVERFLOW failure. + * + * Technically, a server with 64-bit answers is non-compliant if + * the client did not negotiate extended headers - contexts that + * include 64-bit flags should not have been negotiated in that + * case.
Yes, and we should absolutely enforce that. Don't we break off the connection (or at least abort the request) when we get the wrong reply magic from the server, or else if we do get the right reply magic, but an NBD_REPLY_TYPE_BLOCK_STATUS_EXT reply type code inside, without having negotiated extended headers? This compatibility goo creates a big maintenance and debugging burden.
+ */ + if (entries[i].length > UINT32_MAX) { + array[i++ * 2] = -MAX_REQUEST_SIZE; + break; + } + if (entries[i].flags > UINT32_MAX) { + *error = EOVERFLOW; + fail = true; + break; + } + } + + ret = CALL_CALLBACK (*cb, metacontext, offset, array, i * 2, error); + free (array); + return fail ? -1 : ret; +}
Back to the commit message:
Note that the existing 32-bit nbd_block_status() API is now slightly slower, particularly when talking with a server that lacks extended headers: we are doing two size conversions (32 to 64 internally, 64 back to 32 to the user). But this speed penalty is likely in the noise compared to the network delays, and ideally clients will switch over to the new 64-bit interfaces as more and more servers start supporting extended headers.
I'm not worried about the speed penalty, but at this point (and looking at the rest of the patch) I'm actually changing my mind, and questioning whether we are better off with the unified, 64-bit only internal representation. It's not just about converting back and forth (and hence code complexity): we're introducing further malloc()s, and those are all new failure points. Even if we do handle memory allocation failures everywhere correctly, they still introduce pontential *late* breaking points for the client.
One of the trickier aspects of this patch is auditing that both the user's extent and our malloc'd shim get cleaned up once on all possible paths, so that there is neither a leak nor a double free.
Right, and I happen to disagree with your implementation; more on that below.
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 8 +++- generator/states-reply-structured.c | 30 ++++++++----- lib/handle.c | 2 +- lib/rw.c | 70 ++++++++++++++++++++++++++++- 4 files changed, 95 insertions(+), 15 deletions(-)
diff --git a/lib/internal.h b/lib/internal.h index e4e21a4d..11186e24 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -303,7 +303,11 @@ struct nbd_handle { size_t querynum;
/* When receiving block status, this is used. */ - uint32_t *bs_entries; + union { + char *storage; /* malloc's view */ + nbd_extent *normal; /* Our 64-bit preferred internal form; n slots */ + uint32_t *narrow; /* 32-bit NBD_REPLY_TYPE_BLOCK_STATUS form; n*2 slots */ + } bs_entries;
/* Commands which are waiting to be issued [meaning the request * packet is sent to the server]. This is used as a simple linked
Now this could be a good approach in my opinion; however, I disagree with how it's being used (the in-place conversion below, which IMO arguably runs afoul of the effective type rules in C).
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index c53aed7b..784adb78 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -436,6 +436,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_HOLE: REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: struct command *cmd = h->reply_cmd; uint32_t length; + uint32_t count;
switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; @@ -450,15 +451,19 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (length >= 12); length -= sizeof h->sbuf.reply.payload.bs_hdr; + count = length / (2 * sizeof (uint32_t));
This division will work as expected, but it's hard to see why. After my request under patch#1, for introducing bs_reply_length_ok(), and asserting it here, it will be easier to understand. I do dislike the open-coded "sizeof (uint32_t)" though, we should say "sizeof *h->bs_entries.narrow".
- free (h->bs_entries); - h->bs_entries = malloc (length); - if (h->bs_entries == NULL) { + /* Read raw data into a subset of h->bs_entries, then expand it + * into place later during byte-swapping. + */ + free (h->bs_entries.storage); + h->bs_entries.storage = malloc (count * sizeof *h->bs_entries.normal); + if (h->bs_entries.storage == NULL) { SET_NEXT_STATE (%.DEAD); set_error (errno, "malloc"); return 0; } - h->rbuf = h->bs_entries; + h->rbuf = h->bs_entries.narrow; h->rlen = length; SET_NEXT_STATE (%RECV_BS_ENTRIES); } @@ -470,6 +475,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: uint32_t count; size_t i; uint32_t context_id; + uint32_t *raw;
switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; @@ -483,17 +489,21 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); - assert (h->bs_entries); + assert (h->bs_entries.normal); assert (length >= 12); assert (h->meta_valid); count = (length - sizeof h->sbuf.reply.payload.bs_hdr) / - sizeof *h->bs_entries; + (2 * sizeof (uint32_t));
/* Need to byte-swap the entries returned, but apart from that we - * don't validate them. + * don't validate them. Reverse order is essential, since we are + * expanding in-place from narrow to wider type. */ - for (i = 0; i < count; ++i) - h->bs_entries[i] = be32toh (h->bs_entries[i]); + raw = h->bs_entries.narrow; + for (i = count; i-- > 0; ) { + h->bs_entries.normal[i].flags = be32toh (raw[i * 2 + 1]); + h->bs_entries.normal[i].length = be32toh (raw[i * 2]); + }
/* Look up the context ID. */ context_id = be32toh (h->sbuf.reply.payload.bs_hdr.context_id);
So I find this way too clever; worse, it almost certainly breaks the effective type rules in C. First, I would be OK with storing pointers in the bs_entries union, if our only purpose with the union were to only handle one of those pointers at any given time. However, we're type punning the pointers back and forth, and relying on their representation being identical. (Pointer to structure vs. pointer to char vs. pointer to uint32_t.) Not catastrophic, but not nice (and IMO not even necessary, see below). Second, after reading the effective type rules very carefully, the code might break them minimally (I think) when "i" equals zero, and the "length" field is assigned. In that instance, you are reading the same memory location as a uint32_t and writing it as a uint64_t, and not going through a union. I'd prefer one of the two following options: - bs_entries should indeed be the union that you recommend, but used differently. First, "storage" should be allocated, and filled. When the reading completes, a new uint32_t array should be allocated, populated via byte-swapping, "storage" should be released, and the new (now byte-order-correct) array should be recorded in "narrow". Same approach for the future "normal" population. The advantages of this approach are: (a) not reading a pointer-to-type-X value as a pointer-to-type-Y value (reinterpreting the bit pattern) via the union, as we should always know what reply type we're dealing with, (b) any allocated array would only be accessed with a singular element type at any point in time. A kind of "double buffering". - Better yet: drop the "storage" field from the union, and only have "narrow" and "normal" in it. Treat them entirely separately. For the 32-bit case, simply rename the field references (from bs_normal to bs_normal.narrow), no need to change the existent logic. For the 64-bit case, duplicate and widen the 32-bit case (the byte swapping etc), using bs_entries.normal.
@@ -507,7 +517,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES:
if (CALL_CALLBACK (cmd->cb.fn.extent, h->meta_contexts.ptr[i].name, cmd->offset, - h->bs_entries, count, + h->bs_entries.normal, count, &error) == -1) if (cmd->error == 0) cmd->error = error ? error : EPROTO; diff --git a/lib/handle.c b/lib/handle.c index 0f11bee5..fba6d1c4 100644 --- a/lib/handle.c +++ b/lib/handle.c @@ -130,7 +130,7 @@ nbd_close (struct nbd_handle *h) nbd_unlocked_clear_debug_callback (h);
string_vector_empty (&h->querylist); - free (h->bs_entries); + free (h->bs_entries.storage); nbd_internal_reset_size_and_flags (h); for (i = 0; i < h->meta_contexts.len; ++i) free (h->meta_contexts.ptr[i].name);
[empty line here due to earlier out-of-order quoting]
diff --git a/lib/rw.c b/lib/rw.c index 8b2bd4cc..58b05a4b 100644 --- a/lib/rw.c +++ b/lib/rw.c +static void +nbd_convert_extent_free (void *data) +{ + nbd_extent_callback *cb = data; + FREE_CALLBACK (*cb); + free (cb); +} +
For better readability, "data" should be "user_data" IMO, and "cb" should be "shim".
/* Issue a read command and wait for the reply. */ int nbd_unlocked_pread (struct nbd_handle *h, void *buf, @@ -487,12 +542,23 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, nbd_completion_callback *completion, uint32_t flags) { - struct command_cb cb = { .fn.extent = *extent, + nbd_extent_callback *shim = malloc (sizeof *shim); + struct command_cb cb = { .fn.extent.callback = nbd_convert_extent, + .fn.extent.user_data = shim, + .fn.extent.free = nbd_convert_extent_free, .completion = *completion };
+ if (shim == NULL) { + set_error (errno, "malloc"); + return -1; + } + *shim = *extent; + SET_CALLBACK_TO_NULL (*extent); + if (h->strict & LIBNBD_STRICT_COMMANDS) { if (!h->structured_replies) { set_error (ENOTSUP, "server does not support structured replies"); + FREE_CALLBACK (cb.fn.extent); return -1; }
@@ -500,11 +566,11 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, set_error (ENOTSUP, "did not negotiate any metadata contexts, " "either you did not call nbd_add_meta_context before " "connecting or the server does not support it"); + FREE_CALLBACK (cb.fn.extent); return -1; } }
- SET_CALLBACK_TO_NULL (*extent); SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb);
I disagree with this order of operations. (Note, I'm not saying there are leaks or double free's.) Consider the nbd_unlocked_aio_block_status() call site in nbd_aio_block_status():
int64_t nbd_aio_block_status ( struct nbd_handle *h, uint64_t count, uint64_t offset, nbd_extent_callback extent_callback, nbd_completion_callback completion_callback, uint32_t flags ) { /* ... */ ret = nbd_unlocked_aio_block_status (h, count, offset, &extent_callback, &completion_callback, flags); /* ... */ FREE_CALLBACK (extent_callback); FREE_CALLBACK (completion_callback); /* ... */ }
If that call succeeds (returns a value different from (-1)), then we expect nbd_unlocked_aio_block_status() to have taken ownership of both "extent_callback" and "completion_callback", by emptying them. That way, the FREE_CALLBACK()s at the end of the function decay to NOPs. If the call fails OTOH, we expect the call to have left "extent_callback" and "completion_callback" untouched, and the FREE_CALLBACK()s at the end to have "fangs" -- perform actual freeing. Pre-patch, nbd_unlocked_aio_block_status() conforms to this interface contract quite well. It first goes through all possible local error branches, and if all of those pass, only then does it zero the callback structures (i.e., takes ownership of them). This patch changes that; we take ownershop of "extent_callback" (by copying it to "shim" and nulling it) before we check our local error conditions, such as structured_replies and metadata context validity. That breaks the interface contract, and so we cannot rely on nbd_aio_block_status() for freeing "extent_callback" when we fail in one of those local error conditions -- this is why we have to go through FREE_CALLBACK (cb.fn.extent) on each of those error paths, for freeing *both* "shim" and the "extent_callback" internals. And (most importantly to me) then we return to the caller, namely nbd_aio_block_status(), in such a way that (a) we fail, yet (b) we have already destroyed "extent_callback" (but not "completion_callback"). The caller copes with this (due to how FREE_CALLBACK() works), but it's unclean. If nbd_unlocked_aio_block_status() fails, it shouldn't have messed with "extent_callback". (I notice a *slight* inconsistency with this requirement of mine in the pre-patch code as well, BTW: pre-patch, we null both callback structures *before* calling nbd_internal_command_common(); if the latter fails, we get into the same situation. So I actually tend to disagree with that too.) Anyway, I would suggest (on top of your patch):
diff --git a/lib/rw.c b/lib/rw.c index 58b05a4b5529..1913cbb11813 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -552,26 +552,28 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, set_error (errno, "malloc"); return -1; } - *shim = *extent; - SET_CALLBACK_TO_NULL (*extent);
if (h->strict & LIBNBD_STRICT_COMMANDS) { if (!h->structured_replies) { set_error (ENOTSUP, "server does not support structured replies"); - FREE_CALLBACK (cb.fn.extent); - return -1; + goto free_shim; }
if (!h->meta_valid || h->meta_contexts.len == 0) { set_error (ENOTSUP, "did not negotiate any metadata contexts, " "either you did not call nbd_add_meta_context before " "connecting or the server does not support it"); - FREE_CALLBACK (cb.fn.extent); - return -1; + goto free_shim; } }
+ *shim = *extent; + SET_CALLBACK_TO_NULL (*extent); SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb); + +free_shim: + free (shim); + return -1; }
Which, after squashing, would result in a patch like:
@@ -487,25 +542,38 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, nbd_completion_callback *completion, uint32_t flags) { - struct command_cb cb = { .fn.extent = *extent, + nbd_extent_callback *shim = malloc (sizeof *shim); + struct command_cb cb = { .fn.extent.callback = nbd_convert_extent, + .fn.extent.user_data = shim, + .fn.extent.free = nbd_convert_extent_free, .completion = *completion };
+ if (shim == NULL) { + set_error (errno, "malloc"); + return -1; + } + if (h->strict & LIBNBD_STRICT_COMMANDS) { if (!h->structured_replies) { set_error (ENOTSUP, "server does not support structured replies"); - return -1; + goto free_shim; }
if (!h->meta_valid || h->meta_contexts.len == 0) { set_error (ENOTSUP, "did not negotiate any metadata contexts, " "either you did not call nbd_add_meta_context before " "connecting or the server does not support it"); - return -1; + goto free_shim; } }
+ *shim = *extent; SET_CALLBACK_TO_NULL (*extent); SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb); + +free_shim: + free (shim); + return -1; }
But, to repeat myself -- I'm not convinced any more we benefit from the 64-bit-only internal representation. We should always be able to tell precisely which one of the 32-bit and the 64-bit internal representations we need. We should handle each separately, and then we don't need the conversions + shim. Laszlo
On Fri, Jun 09, 2023 at 11:39:27AM +0200, Laszlo Ersek wrote:
Fair warning: this patch has made me both very excited and very uncomfortable; that's the reason my style might come across as needlessly "combative". I don't mean to be combative. It's just how my thoughts have come out, and (unfortunately) I need to go pick up my son now, and don't have time for further rounds of polishing my style. I apologize in advance. Again, it's a consequence of me caring a bit too much perhaps, and getting worked up.
The disclaimer is much appreciated. And I likewise hope my reply is not seen as combative, although I readily admit it is long-winded.
On 5/25/23 15:00, Eric Blake wrote:
When extended headers are in use, the server is required to use a response that can include 64-bit extent lengths and flags (even if it chooses to keep the extent length under 4G, and even for metacontexts like "base:allocation" where the flags never need more than 32 bits).
Yes. Per spec, negotiating NBD_OPT_EXTENDED_HEADERS is equivalent to the server sending NBD_REPLY_TYPE_BLOCK_STATUS_EXT, as opposed to NBD_REPLY_TYPE_BLOCK_STATUS.
For maximum flexibility, we are better off storing 64-bit data internally, with a goal of letting the client's 32-bit interface work as much as possible, and for a future API addition of a 64-bit interface to work even with older servers that only give 32-bit results.
I agree that a 64-bit-only internal representation makes sense. Once corresponding code part (quoting out of order for discussion's sake) is:
diff --git a/lib/internal.h b/lib/internal.h index e4e21a4d..11186e24 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -80,7 +80,7 @@ struct export {
struct command_cb { union { - nbd_extent_callback extent; + nbd_extent64_callback extent; nbd_chunk_callback chunk; nbd_list_callback list; nbd_context_callback context;
and that looks good to me.
For backwards compatibility, a client that never negotiates a 64-bit status context can be handled without errors by truncating any 64-bit lengths down to just under 4G; so the old 32-bit interface will continue to work in most cases.
I absolutely don't follow this. The spec is clear. NBD_OPT_EXTENDED_HEADERS corresponds to NBD_REPLY_TYPE_BLOCK_STATUS_EXT, and (!NBD_OPT_EXTENDED_HEADERS) corresponds to NBD_REPLY_TYPE_BLOCK_STATUS. If NBD_OPT_EXTENDED_HEADERS is negotiated (which means both client and server agree to it), then the client interface is 64-bit clean, so there's no need to truncate anything. If NBD_OPT_EXTENDED_HEADERS is *not* negotiated, then our internal 64-bit clean representation will never be filled with information that doesn't fit into 32-bits (due to the reply type being restricted to NBD_REPLY_TYPE_BLOCK_STATUS), so there is again no need to (observably) truncate anything (technically there would be a uint64_t to uint32_t conversion, but it would be value-preserving).
The spec also says that, if a client wants to query a particular metadata context that is known to use 64-bit values, then the client must first negotiate NBD_OPT_EXTENDED_HEADERS.
Maybe I can improve my wording in that part of the commit message. But first, let's step back and see the big picture. For the sake of argument, let's suppose there are two NBD servers in play; server A that does not know extended headers but supports "base:allocation" (a good set of servers today), and server B that has implemented extended headers and has added a new metacontext (let's call it "x-new:big") that returns 64-bit status flags, but is willing to also accept older clients. (I hope that someday I can get nbdkit to be such a server, but doing that requires adding a v3 protocol for nbdkit plugins; the current v2 plugin protocol can ONLY provide "base:allocation", but being able to let a plugin provide its own metacontexts will add some nice expressive power to nbdkit). Clients compiled against libnbd 1.16 do NOT have access to nbd_block_status_64() (the API did not exist yet); they expect that nbd_block_status() (which returns responses constrained to 32-bit lengths, by its API contract) will work. When run with the .so for libnbd 1.16 (currently named libnbd.so.0.0.0, because we have never broken ABI compatibility across version updates - everything has been additions-only), the client does not request extended headers (that was not part of the code base), so we can reasonably state that if such a client requests "x-new:big" during negotiation with server B, a compliant server will reply with NBD_REP_ERR_EXT_HEADER_REQD; libnbd 1.16 doesn't recognize that constant, but does recognize it as an error reply, and will gracefully proceed by telling the client it cannot access "x-new:big", so there are never any 64-bit flag values to worry about. You are also right that if that same app requests "base:allocation", the server will accept the negotiation, and reply with NBD_REPLY_TYPE_BLOCK_STATUS where lengths are inherently capped at 32 bits for a status flag value that never exceeds 32 bits. But now what happens when you upgrade your system to libnbd 1.18? I see at least three plausible designs: 1. We keep the .so version unchanged and state that libnbd blindly defaults to older behavior, where extended headers require explicit opt-in (for example, where you used to use 'h = nbd_create(); nbd_connect_uri(h, uri);', you now have to edit your code to add 'nbd_set_request_extended_headers(h, true);' in between). In this scenario, backwards-compatibility is easy: the app compiled against 1.16 maintains the same behavior, and newer apps can write conditional code that guards the call to nbd_set_request_extended_headers() by #ifdef LIBNBD_HAVE_NBD_SET_REQUEST_EXTENDED_HEADERS to compile against old and new libnbd. However, over time, as more and more servers support extended headers, we've actively made it harder for newer applications to quickly connect to the full power of such servers: the boilerplate for connection with maximal features has to be repeated in every client, which does not scale. 2. We keep the .so version unchanged (or maybe bump the micro to denote new APIs were added - but we haven't been doing that - I'm not sure if we are correctly following libtool-recommended practice on how to number our .so), and state that libnbd blindly defaults to requesting extended headers but you can now explicitly opt-out. The existence of nbd_set_request_extended_headers(h, false) is the escape valve that new code can use to guarantee behavior that 1.16 provided, and #ifdef LIBNBD_HAVE_NBD_SET_REQUEST_EXTENDED_HEADERS lets you write code compatible to either set of headers. The common case of connecting to a server with as many features as possible remains as short as possible when writing a client. 3. We take a hybrid approach, bump the .so version, and figure out some way to make the behavior determined by what version of the headers you compiled/linked against. For example, maybe have some linker magic where you export both nbd_create@@1_16 which starts a handle with request_eh=false, and nbd_create@@1_18 which starts a handle with request_eh=true; apps compiled against 1.16 will have nbd_create() resolve to nbd_create@@1_16 and never see a change in behavior, while apps compiled against 1.18 will have nbd_create() resolve to nbd_create@@1_18. This adds a lot of complexity to our build scripts (glibc does this sort of backwards-compatibility, but it can be very tricky to get right, and very few other libraries do it). This series went with option 2: default to new behavior, while bending over backwards to avoid scenarios in older clients that already worked. Now, what are the implications of that change? Back to the example I started earlier: If I write a new client against libnbd 1.18, and request "x-new:big" from server B, I don't have to do anything special for the negotiation to work: libnbd negotiated extended headers automatically, so the server allows my request. But when I want to get block status, I have two interfaces exposed: nbd_block_status() and nbd_block_status_64(). Presumably, because I just wrote the client against libnbd 1.18, the documentation hints are good enough that I should use nbd_block_status_64(), and all is well in the world (the server can send 64-bit lengths and 64-bit flags, and I can see them, with no munging required). What's more, even when I ask for "base:allocation", even though I never need 64-bit flags, I automatically benefit from 64-bit lengths when I use nbd_block_status_64() (patches 12-17 in this series show what is required to make an app use the new interfaces - it's boilerplate that has to be copied to every client written against older libnbd, but it is not too bad, and scaling is less of a concern because there are only a finite number of older apps). If that same client now talks to server A, the request for extended headers will fail, and the request for "x-new:big" will fail (the server doesn't support it); but the request for "base:allocation" will succeed. But we've already written the new client to call nbd_block_status_64() which reports 64-bit lengths; this works because under the hood, even though server A is sending us NBD_REPLY_TYPE_BLOCK_STATUS with 32-bit lengths, it is easy to widen that into our 64-bit internal representation, which we then hand to the client app. The app could also choose to explicitly check nbd_get_extended_headers_negotiated() in order to make a runtime decision of whether to call nbd_block_status() or nbd_block_status_64() based on what the server supports, but that's a lot of boilerplate unlikely to be used in most new clients. Thus, I prefer my decision to normalize to a 64-bit internal representation as soon as possible (at the risk of having to convert it back to 32-bits if the client made a conscious choice to use the old interface), rather than bending over backwards to store the server's response as-is as long as possible (and only convert it to the other width if the client called the API opposite from what the server negotiated). But back to my old client compiled against 1.16 and now dynamically loading the 1.18 .so. When talking to server A, the new .so defaults result in the old client now requesting (but not getting) get extended headers, and then calls to nbd_block_status() (the only API available at the time the app was compiled) will see server A's NBD_REPLY_TYPE_BLOCK_STATUS with 32-bit answers, temporarly widened into our 64-bit internal representation, and then the shim narrows it back to 32 bits for presentation to the client application, with no loss of information. Because of the double conversion, this may be slightly slower than it was with the 1.16 .so, but it is not an obvious behavior change. But the final combination is where the fun enters in. The old client talking to server B using the 1.18 .so defaults to requesting extended headers, and the server honors that request. However, the old client does not have access to the nbd_block_status_64() API (it was not around at the time of compilation), so it WILL be using the 32-bit interface nbd_block_status(). Whereas with the 1.16 .so an attempt to request "x-new:big" was rejected during negotiation, that request is now honored, and so we really do risk a flags value that we have to represent to the client as an EOVERFLOW error (an early failure is now a late failure; fortunately, this behavior change is not common in practice - other than clients like nbdsh that really do allow you to decide at runtime which metacontext(s) you want to request, most applications know in advance what they want from the server, and won't be asking for "x-new:big"). But more importantly, our request for "base:allocation" is STILL granted, and the server is now replying with NBD_REPLY_TYPE_BLOCK_STATUS_EXT. Even though the flags values will never exceed 32 bits, the server _can_ now give us lengths larger than 4G for an extent. We do _not_ want to fail the request with EOVERFLOW just because the server was now permitted to answer us with a larger extent than before, but we can take advantage of the NBD spec, which says that as long as the reply to NBD_CMD_BLOCK_STATUS makes progress, it does NOT have to go as far as possible - even though the server gave us a length larger than 4G over the wire, we are still compliant to the spec from the client's point of view if we truncate that response to being less than 4G - it still made progress, and just as with server A, the client can then ask for block status at the next byte after where the current answer ended. The tl;dr summary of the above discourse: There are two orthogonal communications going on: libnbd <-> server choice of NBD_REPLY_TYPE_BLOCK_STATUS{,_EXT} app <-> libnbd choice of nbd_block_status() or nbd_block_status_64() and all four combinations are easy to encounter when loading the .so for libnbd 1.18: 32 x 32 (app compiled against libnbd 1.16 to server A) 64 x 32 (app compiled against libnbd 1.16 to server B) 32 x 64 (app compiled against libnbd 1.18 to server A) 64 x 64 (app compiled against libnbd 1.18 to server B) and we want all four combinations to work insofar as possible. 32x32 and 64x64 obviously work, as does 32x64 (widening the server's responses never fails); for 32x64 (using the 32-bit nbd_block_status() API to access a server's response through 64-bit NBD_REPLY_TYPE_BLOCK_STATUS), accessing a metacontext with large flags changes from fail early to fail late; and accessing a metacontext with 32-bit flags but now potential 64-bit lengths obeys overall NBD semantics that a block status response can be truncated as long as it makes progress (the app shouldn't care whether it was the server or libnbd that did the truncation).
A server should not advertise a metacontext that requires flags larger than 32 bits unless the client negotiated extended headers; but if such a client still tries to use the 32-bit interface, the code now reports EOVERFLOW for a server response with a flags value that cannot be represented without using the future API for 64-bit extent queries.
Again, I don't follow. Whenever the client issues an NBD_CMD_BLOCK_STATUS command (which can only occur after negotiation):
- the client must have set at least one metacontext with NBD_OPT_SET_META_CONTEXT during negotiation,
- the client may or may not have agreed upon with the server on NBD_OPT_EXTENDED_HEADERS.
Therefore, when the server receives the NBD_CMD_BLOCK_STATUS command, it can immediately decide whether there is *any* configured metacontext for which *lack* of extended headers could be too restrictive. If only 32-bit metacontexts have been requested, this is not a problem. If ext-hdrs have been negotiated, also not a problem. If there is at least one 64-bit metacontext (regarding *either* status flags *or* lengths) *and* ext-hdrs have not been negotiated, the server should immediately fail (reject) the request.
I think I see where you missed something. The decision of whether a metacontext can have 64-bit flag content is dependent on the metacontext, and therefore something that can be fully decided at the time of NBD_OPT_SET_META_CONTEXT and/or NBD_OPT_GO (it is not a decision that needs to wait until NBD_CMD_BLOCK_STATUS). Either the metacontext has the potential for 64-bit flags (and the server will have failed with NBD_REP_ERR_EXT_HEADER_REQD if the client did not request extended headers), or it does not. But the decision of whether a metacontext can have 64-bit lengths is NOT dependent on the metacontext, but rather on the image size and our ability to make large queries. If we have a 5G image that is known to be all data, we HAVE to truncate our reply to less than 4G for a client that did not request extended headers, and then the client asks for a second block status at the point where it left off. But once extended headers are negotiated, we can give a single reply stating that the entire image is a single all-data extent. The server does not know at the time of NBD_CMD_BLOCK_STATUS whether the client is using 32-bit nbd_block_status or 64-bit nbd_block_status_64(). It is this distinction between length and flags that explains why I want nbd_block_status_64() to return a struct, rather than an array of uint64_t. Even though the wire representation of the extent is a pair of uint64_t values, the interpretation of those values is vastly different: length should never exceed 63 bits (because of off_t or other limitations) but can easily exceed 4G; while flags could be a full 64 bits, or in the case of "base:allocation" will never be more than 2 bits. The definition of flags is context-dependent, but the definition of length is not.
Now, peeking a bit forward, I can see patch#10: "api: Add [aio_]nbd_block_status_64". That means we can reject a 32-bit-only nbd_block_status() API call on the client side immediately, if the client has negotiated ext-hdrs.
We can, but that breaks backwards compatibility. We do not want an app compiled against libnbd 1.16 to stop working merely because it is now loading the 1.18 .so.
That is, if the client is smart enough to ask for extended headers, it can be smart enough to use the 64-bit API, and then it can accept everything the server throws at it. If the client never even asks for extended headers, then it is permitted to stick with the 32-bit-only API, but then the server must never respond with NBD_REPLY_TYPE_BLOCK_STATUS_EXT [*], so actually 64-bit values are never going to be present in our internal representation.
[*] If it still does, then the server is buggy, and we should realize that as soon as we parse the generic reply magic, or the specific reply type code.
It is indeed a bug if a server replies with NBD_REPLY_TYPE_BLOCK_STATUS_EXT for a client that did not request extended headers. But it is also a bug if a server replise with NBD_REPLY_TYPE_BLOCK_STATUS for a client that did request extended headers, even if the reply does not need more than 32 bits for either the length or the flags.
I simply fail to see the *possibility* truncating the internal 64-bit representation to 32-bit such that it would make a difference for the caller.
Speaking in terms of code:
diff --git a/lib/rw.c b/lib/rw.c index 8b2bd4cc..58b05a4b 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -42,6 +42,61 @@ wait_for_command (struct nbd_handle *h, int64_t cookie) return r == -1 ? -1 : 0; }
+/* Convert from 64-bit to 32-bit extent callback. */ +static int +nbd_convert_extent (void *data, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t nr_entries, int *error) +{ + nbd_extent_callback *cb = data; + uint32_t *array = malloc (nr_entries * 2 * sizeof *array);
Side question: what guarantees that the multiplication won't overflow?
The fact that we cap all incoming server replies to at most 64M (see pre-series REPLY.STRUCTURED_REPLY.CHECK which rejects payloads larger than MAX_REQUEST_SIZE). Even though the NBD spec only recommends (rather than requires) that a server should not send more than 2M extents in a single block status reply, we will have entered the DEAD state if the server sent more extents than fit in MAX_REQUEST_SIZE.
If we have a guarantee, we should document it, and (preferably) assert it.
Yes, I can add 'assert (nr_entries < SIZE_MAX / 2 / sizeof *array);' or similar, along with a comment pointing to the MAX_REQUEST_SIZE check.
+ size_t i; + int ret; + bool fail = false; + + if (array == NULL) { + set_error (*error = errno, "malloc"); + return -1; + } + + for (i = 0; i < nr_entries; i++) { + array[i * 2] = entries[i].length; + array[i * 2 + 1] = entries[i].flags; + /* If an extent is larger than 32 bits, silently truncate the rest + * of the server's response; the client can then make progress + * instead of needing to see failure. Rather than track the + * connection's alignment, just clamp the large extent to 4G-64M. + * However, if flags doesn't fit in 32 bits, it's better to inform + * the caller of an EOVERFLOW failure. + * + * Technically, a server with 64-bit answers is non-compliant if + * the client did not negotiate extended headers - contexts that + * include 64-bit flags should not have been negotiated in that + * case.
Yes, and we should absolutely enforce that.
Don't we break off the connection (or at least abort the request) when we get the wrong reply magic from the server, or else if we do get the right reply magic, but an NBD_REPLY_TYPE_BLOCK_STATUS_EXT reply type code inside, without having negotiated extended headers?
Yes, the code is set up to gracefully ignore server replies sent with the wrong reply type (the code does try hard to keep the connection alive when possible, by using payload lengths to skip to the next viable reply header; but moves to the DEAD state when it was not viable - either way, nbd_block_status() cannot succeed if the server uses the wrong reply type).
This compatibility goo creates a big maintenance and debugging burden.
Yes, backwards compatibility is always something that requires careful design.
+ */ + if (entries[i].length > UINT32_MAX) { + array[i++ * 2] = -MAX_REQUEST_SIZE; + break; + } + if (entries[i].flags > UINT32_MAX) { + *error = EOVERFLOW; + fail = true; + break; + } + } + + ret = CALL_CALLBACK (*cb, metacontext, offset, array, i * 2, error); + free (array); + return fail ? -1 : ret; +}
Back to the commit message:
Note that the existing 32-bit nbd_block_status() API is now slightly slower, particularly when talking with a server that lacks extended headers: we are doing two size conversions (32 to 64 internally, 64 back to 32 to the user). But this speed penalty is likely in the noise compared to the network delays, and ideally clients will switch over to the new 64-bit interfaces as more and more servers start supporting extended headers.
I'm not worried about the speed penalty, but at this point (and looking at the rest of the patch) I'm actually changing my mind, and questioning whether we are better off with the unified, 64-bit only internal representation. It's not just about converting back and forth (and hence code complexity): we're introducing further malloc()s, and those are all new failure points. Even if we do handle memory allocation failures everywhere correctly, they still introduce pontential *late* breaking points for the client.
Okay, I see what you are getting at. We do one malloc() in response to whether the server sent NBD_REPLY_TYPE_BLOCK_STATUS or NBD_REPLY_TYPE_BLOCK_STATUS_EXT, and then another malloc() if the client called nbd_block_status(). You are worried that having two malloc()s doubles the chance of hitting a failure at an odd point, making it harder to give correct responses to the user's callbacks. Here's a strawman idea for a different internal representation: what if we store BOTH 32- and 64-bit data in a single malloc()? That is, in highly-abbreviated code sketch (doing it without abusing type aliases will obviously take more lines of code): REPLY.STRUCTURED_REPLY.CHECK: ... case NBD_REPLY_TYPE_BLOCK_STATUS: compute nr_entries; // number of uint32_t pairs char *bs_entries = malloc (6 * nr_entries); // relies on malloc being 8-byte aligned rbuf = bs_entries + nr_entries * 4 * sizeof(uint32_t); break; case NBD_REPLY_TYPE_BLOCK_STATUS_EXT: compute nr_entries; // number of uint64_t pairs char *bs_entries = malloc (nr_entries * 3); rbuf = bs_entries; break; ... REPLY.STRUCTURED_REPLY.BS_ENTRIES: ... if (type == NBD_REPLY_TYPE_BLOCK_STATUS) { // widen 32-bit entries in last third into 64-bit entries in first two-thirds for (i = 0; i < nr_entries * 2; i++) *(uint64_t*)bs_entries[i*sizeof(uint64_t)] = *(uint32_t*)bs_entries[(i+nr_entries*4)*sizeof(uint32_t)] } else /* type == NBD_REPLY_TYPE_BLOCK_STATUS_EXT) { // narrow 64-bit entries in first two-thirds into 32-bit entries in last third for (i = 0; i < nr_entries; i++) { // check for overflow or truncation as needed } } if (/* user called nbd_block_status */) CALL_CALLBACK (cmd->cb.fn.extent32 (..., nr_entries*2, (uint32_t*)&bs_entries[nr_entries * 4 * sizeof(uint32_t)] ...) else /* user called nbd_block_status_64 */ CALL_CALLBACK (cmd->cb.fn.extent64 (..., nr_entries, (nbd_extent*)&bs_entries[0] ...) In this approach, no shims are needed (both 32- and 64-bit nbd_block_status* API calls natively call into the applications callback with the correct typing), and we have fewer malloc() calls, but need more memory reserved per the duration of a callback, and waste time on conversion for every API call. Or add even more complicated code to only do the conversion when we compare the user's API call to the server's response type. Just because it is a possible approach doesn't mean it is more appealing. Then again, you are right that having a shim makes life harder on verifying that we are doing correct memory management on all code paths, and coding things so that no shim is required might make the code easier to reason about despite its complexity in managing more memory.
One of the trickier aspects of this patch is auditing that both the user's extent and our malloc'd shim get cleaned up once on all possible paths, so that there is neither a leak nor a double free.
Right, and I happen to disagree with your implementation; more on that below.
Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 8 +++- generator/states-reply-structured.c | 30 ++++++++----- lib/handle.c | 2 +- lib/rw.c | 70 ++++++++++++++++++++++++++++- 4 files changed, 95 insertions(+), 15 deletions(-)
diff --git a/lib/internal.h b/lib/internal.h index e4e21a4d..11186e24 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -303,7 +303,11 @@ struct nbd_handle { size_t querynum;
/* When receiving block status, this is used. */ - uint32_t *bs_entries; + union { + char *storage; /* malloc's view */ + nbd_extent *normal; /* Our 64-bit preferred internal form; n slots */ + uint32_t *narrow; /* 32-bit NBD_REPLY_TYPE_BLOCK_STATUS form; n*2 slots */ + } bs_entries;
/* Commands which are waiting to be issued [meaning the request * packet is sent to the server]. This is used as a simple linked
Now this could be a good approach in my opinion; however, I disagree with how it's being used (the in-place conversion below, which IMO arguably runs afoul of the effective type rules in C).
Vladimir also didn't like my attempts to do in-place conversions in the qemu counterpart code (see https://listman.redhat.com/archives/libguestfs/2023-May/031685.html), in part because you MUST be careful about index directions (incrementing when narrowing, decrementing when widening). So yet another approach is to read the server's memory as-is, then copy into the form passed to the user (rather than doing in-place conversions), regardless of whether that copy widens or narrows. That's back to two malloc's per reply (one to get the server's bytes, another for what to hand to the callback), but may be easier to reason about.
diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index c53aed7b..784adb78 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -436,6 +436,7 @@ REPLY.STRUCTURED_REPLY.RECV_OFFSET_HOLE: REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: struct command *cmd = h->reply_cmd; uint32_t length; + uint32_t count;
switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; @@ -450,15 +451,19 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (length >= 12); length -= sizeof h->sbuf.reply.payload.bs_hdr; + count = length / (2 * sizeof (uint32_t));
This division will work as expected, but it's hard to see why. After my request under patch#1, for introducing bs_reply_length_ok(), and asserting it here, it will be easier to understand.
I do dislike the open-coded "sizeof (uint32_t)" though, we should say "sizeof *h->bs_entries.narrow".
Yes, I'm still working to rebase on your earlier comments (the helper function bs_reply_length_ok() idea made sense), and naming the actual variable rather than having to check that the correct type is doable (even though it makes for longer lines and possibly more line breaks).
- free (h->bs_entries); - h->bs_entries = malloc (length); - if (h->bs_entries == NULL) { + /* Read raw data into a subset of h->bs_entries, then expand it + * into place later during byte-swapping. + */ + free (h->bs_entries.storage); + h->bs_entries.storage = malloc (count * sizeof *h->bs_entries.normal); + if (h->bs_entries.storage == NULL) { SET_NEXT_STATE (%.DEAD); set_error (errno, "malloc"); return 0; } - h->rbuf = h->bs_entries; + h->rbuf = h->bs_entries.narrow; h->rlen = length; SET_NEXT_STATE (%RECV_BS_ENTRIES); } @@ -470,6 +475,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: uint32_t count; size_t i; uint32_t context_id; + uint32_t *raw;
switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; @@ -483,17 +489,21 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); - assert (h->bs_entries); + assert (h->bs_entries.normal); assert (length >= 12); assert (h->meta_valid); count = (length - sizeof h->sbuf.reply.payload.bs_hdr) / - sizeof *h->bs_entries; + (2 * sizeof (uint32_t));
/* Need to byte-swap the entries returned, but apart from that we - * don't validate them. + * don't validate them. Reverse order is essential, since we are + * expanding in-place from narrow to wider type. */ - for (i = 0; i < count; ++i) - h->bs_entries[i] = be32toh (h->bs_entries[i]); + raw = h->bs_entries.narrow; + for (i = count; i-- > 0; ) { + h->bs_entries.normal[i].flags = be32toh (raw[i * 2 + 1]); + h->bs_entries.normal[i].length = be32toh (raw[i * 2]); + }
/* Look up the context ID. */ context_id = be32toh (h->sbuf.reply.payload.bs_hdr.context_id);
So I find this way too clever; worse, it almost certainly breaks the effective type rules in C.
First, I would be OK with storing pointers in the bs_entries union, if our only purpose with the union were to only handle one of those pointers at any given time. However, we're type punning the pointers back and forth, and relying on their representation being identical. (Pointer to structure vs. pointer to char vs. pointer to uint32_t.) Not catastrophic, but not nice (and IMO not even necessary, see below).
Second, after reading the effective type rules very carefully, the code might break them minimally (I think) when "i" equals zero, and the "length" field is assigned. In that instance, you are reading the same memory location as a uint32_t and writing it as a uint64_t, and not going through a union.
I'd prefer one of the two following options:
- bs_entries should indeed be the union that you recommend, but used differently. First, "storage" should be allocated, and filled. When the reading completes, a new uint32_t array should be allocated, populated via byte-swapping, "storage" should be released, and the new (now byte-order-correct) array should be recorded in "narrow". Same approach for the future "normal" population. The advantages of this approach are: (a) not reading a pointer-to-type-X value as a pointer-to-type-Y value (reinterpreting the bit pattern) via the union, as we should always know what reply type we're dealing with, (b) any allocated array would only be accessed with a singular element type at any point in time. A kind of "double buffering".
Aha - now that I've read this far, you are steering towards the same solution I started reaching above. Use two malloc()s, but be more specific about which types are represented. Your idea of releasing 'storage' and reassigning with 'wide' or 'narrow' is clever, but definitely nicer type-wise than in-place reinterpretation of a single malloc() across different types where (during the conversion) there is no single type that properly describes the overall storage.
- Better yet: drop the "storage" field from the union, and only have "narrow" and "normal" in it. Treat them entirely separately. For the 32-bit case, simply rename the field references (from bs_normal to bs_normal.narrow), no need to change the existent logic. For the 64-bit case, duplicate and widen the 32-bit case (the byte swapping etc), using bs_entries.normal.
@@ -507,7 +517,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES:
if (CALL_CALLBACK (cmd->cb.fn.extent, h->meta_contexts.ptr[i].name, cmd->offset, - h->bs_entries, count, + h->bs_entries.normal, count, &error) == -1) if (cmd->error == 0) cmd->error = error ? error : EPROTO; diff --git a/lib/handle.c b/lib/handle.c index 0f11bee5..fba6d1c4 100644 --- a/lib/handle.c +++ b/lib/handle.c @@ -130,7 +130,7 @@ nbd_close (struct nbd_handle *h) nbd_unlocked_clear_debug_callback (h);
string_vector_empty (&h->querylist); - free (h->bs_entries); + free (h->bs_entries.storage); nbd_internal_reset_size_and_flags (h); for (i = 0; i < h->meta_contexts.len; ++i) free (h->meta_contexts.ptr[i].name);
[empty line here due to earlier out-of-order quoting]
diff --git a/lib/rw.c b/lib/rw.c index 8b2bd4cc..58b05a4b 100644 --- a/lib/rw.c +++ b/lib/rw.c +static void +nbd_convert_extent_free (void *data) +{ + nbd_extent_callback *cb = data; + FREE_CALLBACK (*cb); + free (cb); +} +
For better readability, "data" should be "user_data" IMO, and "cb" should be "shim".
/* Issue a read command and wait for the reply. */ int nbd_unlocked_pread (struct nbd_handle *h, void *buf, @@ -487,12 +542,23 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, nbd_completion_callback *completion, uint32_t flags) { - struct command_cb cb = { .fn.extent = *extent, + nbd_extent_callback *shim = malloc (sizeof *shim); + struct command_cb cb = { .fn.extent.callback = nbd_convert_extent, + .fn.extent.user_data = shim, + .fn.extent.free = nbd_convert_extent_free, .completion = *completion };
+ if (shim == NULL) { + set_error (errno, "malloc"); + return -1; + } + *shim = *extent; + SET_CALLBACK_TO_NULL (*extent); + if (h->strict & LIBNBD_STRICT_COMMANDS) { if (!h->structured_replies) { set_error (ENOTSUP, "server does not support structured replies"); + FREE_CALLBACK (cb.fn.extent); return -1; }
@@ -500,11 +566,11 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, set_error (ENOTSUP, "did not negotiate any metadata contexts, " "either you did not call nbd_add_meta_context before " "connecting or the server does not support it"); + FREE_CALLBACK (cb.fn.extent); return -1; } }
- SET_CALLBACK_TO_NULL (*extent); SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb);
I disagree with this order of operations. (Note, I'm not saying there are leaks or double free's.)
Consider the nbd_unlocked_aio_block_status() call site in nbd_aio_block_status():
int64_t nbd_aio_block_status ( struct nbd_handle *h, uint64_t count, uint64_t offset, nbd_extent_callback extent_callback, nbd_completion_callback completion_callback, uint32_t flags ) { /* ... */ ret = nbd_unlocked_aio_block_status (h, count, offset, &extent_callback, &completion_callback, flags); /* ... */ FREE_CALLBACK (extent_callback); FREE_CALLBACK (completion_callback); /* ... */ }
If that call succeeds (returns a value different from (-1)), then we expect nbd_unlocked_aio_block_status() to have taken ownership of both "extent_callback" and "completion_callback", by emptying them. That way, the FREE_CALLBACK()s at the end of the function decay to NOPs. If the call fails OTOH, we expect the call to have left "extent_callback" and "completion_callback" untouched, and the FREE_CALLBACK()s at the end to have "fangs" -- perform actual freeing.
Pre-patch, nbd_unlocked_aio_block_status() conforms to this interface contract quite well. It first goes through all possible local error branches, and if all of those pass, only then does it zero the callback structures (i.e., takes ownership of them).
This patch changes that; we take ownershop of "extent_callback" (by copying it to "shim" and nulling it) before we check our local error conditions, such as structured_replies and metadata context validity. That breaks the interface contract, and so we cannot rely on nbd_aio_block_status() for freeing "extent_callback" when we fail in one of those local error conditions -- this is why we have to go through FREE_CALLBACK (cb.fn.extent) on each of those error paths, for freeing *both* "shim" and the "extent_callback" internals. And (most importantly to me) then we return to the caller, namely nbd_aio_block_status(), in such a way that (a) we fail, yet (b) we have already destroyed "extent_callback" (but not "completion_callback"). The caller copes with this (due to how FREE_CALLBACK() works), but it's unclean. If nbd_unlocked_aio_block_status() fails, it shouldn't have messed with "extent_callback".
(I notice a *slight* inconsistency with this requirement of mine in the pre-patch code as well, BTW: pre-patch, we null both callback structures *before* calling nbd_internal_command_common(); if the latter fails, we get into the same situation. So I actually tend to disagree with that too.)
Anyway, I would suggest (on top of your patch):
diff --git a/lib/rw.c b/lib/rw.c index 58b05a4b5529..1913cbb11813 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -552,26 +552,28 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, set_error (errno, "malloc"); return -1; } - *shim = *extent; - SET_CALLBACK_TO_NULL (*extent);
if (h->strict & LIBNBD_STRICT_COMMANDS) { if (!h->structured_replies) { set_error (ENOTSUP, "server does not support structured replies"); - FREE_CALLBACK (cb.fn.extent); - return -1; + goto free_shim; }
if (!h->meta_valid || h->meta_contexts.len == 0) { set_error (ENOTSUP, "did not negotiate any metadata contexts, " "either you did not call nbd_add_meta_context before " "connecting or the server does not support it"); - FREE_CALLBACK (cb.fn.extent); - return -1; + goto free_shim; } }
+ *shim = *extent; + SET_CALLBACK_TO_NULL (*extent); SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb); + +free_shim: + free (shim); + return -1; }
Which, after squashing, would result in a patch like:
@@ -487,25 +542,38 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, nbd_completion_callback *completion, uint32_t flags) { - struct command_cb cb = { .fn.extent = *extent, + nbd_extent_callback *shim = malloc (sizeof *shim); + struct command_cb cb = { .fn.extent.callback = nbd_convert_extent, + .fn.extent.user_data = shim, + .fn.extent.free = nbd_convert_extent_free, .completion = *completion };
+ if (shim == NULL) { + set_error (errno, "malloc"); + return -1; + } + if (h->strict & LIBNBD_STRICT_COMMANDS) { if (!h->structured_replies) { set_error (ENOTSUP, "server does not support structured replies"); - return -1; + goto free_shim; }
if (!h->meta_valid || h->meta_contexts.len == 0) { set_error (ENOTSUP, "did not negotiate any metadata contexts, " "either you did not call nbd_add_meta_context before " "connecting or the server does not support it"); - return -1; + goto free_shim; } }
+ *shim = *extent; SET_CALLBACK_TO_NULL (*extent); SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb); + +free_shim: + free (shim); + return -1; }
But, to repeat myself -- I'm not convinced any more we benefit from the 64-bit-only internal representation. We should always be able to tell precisely which one of the 32-bit and the 64-bit internal representations we need. We should handle each separately, and then we don't need the conversions + shim.
We always know what width the server send, and what width the caller requested. But since the two are orthonal, and we must support all four combinations, two of those pairing need a conversion. You are right that we can do conversions only when needed, and even set it up so that we do not need a shim. I'll play with ideas mentioned in this review (probably leaning towards the double-buffering idea: we have to read in the server's bytes, and have to do byte swapping - but how we do the byte-swapping can either be in-place when no widening or narrowing is needed, or double-buffered if we also have to adjust to match the caller's size choice). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
[fixing some of my editing errors] On Fri, Jun 09, 2023 at 10:42:25AM -0500, Eric Blake wrote: [...]
The tl;dr summary of the above discourse: There are two orthogonal communications going on:
libnbd <-> server choice of NBD_REPLY_TYPE_BLOCK_STATUS{,_EXT} app <-> libnbd choice of nbd_block_status() or nbd_block_status_64()
and all four combinations are easy to encounter when loading the .so for libnbd 1.18:
32 x 32 (app compiled against libnbd 1.16 to server A) 64 x 32 (app compiled against libnbd 1.16 to server B) 32 x 64 (app compiled against libnbd 1.18 to server A) 64 x 64 (app compiled against libnbd 1.18 to server B)
and we want all four combinations to work insofar as possible. 32x32 and 64x64 obviously work, as does 32x64 (widening the server's responses never fails); for 32x64 (using the 32-bit nbd_block_status() API to access a server's response through 64-bit NBD_REPLY_TYPE_BLOCK_STATUS), accessing a metacontext with large flags
for 64x32 (using the 32-bit nbd_block_status() to access a server's response through 64-bit NBD_REPLY_TYPE_BLOCK_STATUS_EXT),
changes from fail early to fail late; and accessing a metacontext with 32-bit flags but now potential 64-bit lengths obeys overall NBD semantics that a block status response can be truncated as long as it makes progress (the app shouldn't care whether it was the server or libnbd that did the truncation).
[...]
It is indeed a bug if a server replies with NBD_REPLY_TYPE_BLOCK_STATUS_EXT for a client that did not request extended headers. But it is also a bug if a server replise with
replies
NBD_REPLY_TYPE_BLOCK_STATUS for a client that did request extended headers, even if the reply does not need more than 32 bits for either the length or the flags.
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Support a server giving us a 64-bit extent. Note that the protocol says a server should not give a 64-bit answer when extended headers are not negotiated; we can handle that by reporting EPROTO but otherwise accepting the information. Conversely, when extended headers are in effect, the server must respond with the 64-bit extent type, even if the extent length and flags both fit in 32 bits. Since we already store 64-bit extents internally, the user's 32-bit callback doesn't have to care which reply chunk the server uses (the shim takes care of that, and an upcoming patch adds new APIs to let the client use a 64-bit callback). Of course, until a later patch enables extended headers negotiation, no compliant server will trigger the new code here. Implementation-wise, 'normal' and 'wide' are two different types but have the same underlying size; keeping the two names makes it easier to reason about when values are still in network byte order from the server or native endian for local processing. Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 3 + generator/states-reply-structured.c | 94 +++++++++++++++++++++++------ 2 files changed, 78 insertions(+), 19 deletions(-) diff --git a/lib/internal.h b/lib/internal.h index 11186e24..e5423245 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -249,6 +249,7 @@ struct nbd_handle { struct nbd_structured_reply_offset_data offset_data; struct nbd_structured_reply_offset_hole offset_hole; struct nbd_structured_reply_block_status_hdr bs_hdr; + struct nbd_structured_reply_block_status_ext_hdr bs_ext_hdr; struct { struct nbd_structured_reply_error error; char msg[NBD_MAX_STRING]; /* Common to all error types */ @@ -307,6 +308,8 @@ struct nbd_handle { char *storage; /* malloc's view */ nbd_extent *normal; /* Our 64-bit preferred internal form; n slots */ uint32_t *narrow; /* 32-bit NBD_REPLY_TYPE_BLOCK_STATUS form; n*2 slots */ + struct nbd_block_descriptor_ext *wide; + /* 64-bit NBD_REPLY_TYPE_BLOCK_STATUS_EXT; n slots */ } bs_entries; /* Commands which are waiting to be issued [meaning the request diff --git a/generator/states-reply-structured.c b/generator/states-reply-structured.c index 784adb78..55d6bb2d 100644 --- a/generator/states-reply-structured.c +++ b/generator/states-reply-structured.c @@ -22,6 +22,8 @@ #include <stdint.h> #include <inttypes.h> +#include "minmax.h" + /* Structured reply must be completely inside the bounds of the * requesting command. */ @@ -144,9 +146,34 @@ REPLY.STRUCTURED_REPLY.CHECK: length < 12 || ((length-4) & 7) != 0) goto resync; assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); + if (h->extended_headers) { + debug (h, "unexpected 32-bit block status with extended headers, " + "this is probably a server bug"); + if (cmd->error == 0) + cmd->error = EPROTO; + } /* Start by reading the context ID. */ h->rbuf = &h->sbuf.reply.payload.bs_hdr; h->rlen = sizeof h->sbuf.reply.payload.bs_hdr; + h->sbuf.reply.payload.bs_ext_hdr.count = 0; + SET_NEXT_STATE (%RECV_BS_HEADER); + break; + + case NBD_REPLY_TYPE_BLOCK_STATUS_EXT: + if (cmd->type != NBD_CMD_BLOCK_STATUS || + length < 24 || + (length-8) % sizeof (struct nbd_block_descriptor_ext)) + goto resync; + assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); + if (!h->extended_headers) { + debug (h, "unexpected 64-bit block status without extended headers, " + "this is probably a server bug"); + if (cmd->error == 0) + cmd->error = EPROTO; + } + /* Start by reading the context ID. */ + h->rbuf = &h->sbuf.reply.payload.bs_ext_hdr; + h->rlen = sizeof h->sbuf.reply.payload.bs_ext_hdr; SET_NEXT_STATE (%RECV_BS_HEADER); break; @@ -437,6 +464,7 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: struct command *cmd = h->reply_cmd; uint32_t length; uint32_t count; + uint16_t type; switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; @@ -446,24 +474,43 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: return 0; case 0: length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ + type = be16toh (h->sbuf.reply.hdr.structured.type); assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); - assert (length >= 12); - length -= sizeof h->sbuf.reply.payload.bs_hdr; - count = length / (2 * sizeof (uint32_t)); - /* Read raw data into a subset of h->bs_entries, then expand it + if (type == NBD_REPLY_TYPE_BLOCK_STATUS) { + assert (length >= 12); + length -= sizeof h->sbuf.reply.payload.bs_hdr; + count = length / (2 * sizeof (uint32_t)); + } + else { + assert (type == NBD_REPLY_TYPE_BLOCK_STATUS_EXT); + assert (length >= 24); + length -= sizeof h->sbuf.reply.payload.bs_ext_hdr; + count = length / sizeof (struct nbd_block_descriptor_ext); + if (count != be32toh (h->sbuf.reply.payload.bs_ext_hdr.count)) { + h->rbuf = NULL; + h->rlen = length; + SET_NEXT_STATE (%RESYNC); + return 0; + } + } + /* Normalize count for later use. */ + h->sbuf.reply.payload.bs_ext_hdr.count = count; + + /* Read raw data into an overlap of h->bs_entries, then move it * into place later during byte-swapping. */ free (h->bs_entries.storage); - h->bs_entries.storage = malloc (count * sizeof *h->bs_entries.normal); + h->bs_entries.storage = malloc (MAX (count * sizeof *h->bs_entries.normal, + length)); if (h->bs_entries.storage == NULL) { SET_NEXT_STATE (%.DEAD); set_error (errno, "malloc"); return 0; } - h->rbuf = h->bs_entries.narrow; + h->rbuf = h->bs_entries.storage; h->rlen = length; SET_NEXT_STATE (%RECV_BS_ENTRIES); } @@ -471,11 +518,10 @@ REPLY.STRUCTURED_REPLY.RECV_BS_HEADER: REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: struct command *cmd = h->reply_cmd; - uint32_t length; uint32_t count; size_t i; uint32_t context_id; - uint32_t *raw; + uint16_t type; switch (recv_into_rbuf (h)) { case -1: SET_NEXT_STATE (%.DEAD); return 0; @@ -484,25 +530,35 @@ REPLY.STRUCTURED_REPLY.RECV_BS_ENTRIES: SET_NEXT_STATE (%.READY); return 0; case 0: - length = h->sbuf.reply.hdr.structured.length; /* normalized in CHECK */ + type = be16toh (h->sbuf.reply.hdr.structured.type); + count = h->sbuf.reply.payload.bs_ext_hdr.count; /* normalized in BS_HEADER */ assert (cmd); /* guaranteed by CHECK */ assert (cmd->type == NBD_CMD_BLOCK_STATUS); assert (CALLBACK_IS_NOT_NULL (cmd->cb.fn.extent)); - assert (h->bs_entries.normal); - assert (length >= 12); + assert (h->bs_entries.normal && count); assert (h->meta_valid); - count = (length - sizeof h->sbuf.reply.payload.bs_hdr) / - (2 * sizeof (uint32_t)); /* Need to byte-swap the entries returned, but apart from that we - * don't validate them. Reverse order is essential, since we are - * expanding in-place from narrow to wider type. + * don't validate them. */ - raw = h->bs_entries.narrow; - for (i = count; i-- > 0; ) { - h->bs_entries.normal[i].flags = be32toh (raw[i * 2 + 1]); - h->bs_entries.normal[i].length = be32toh (raw[i * 2]); + if (type == NBD_REPLY_TYPE_BLOCK_STATUS) { + uint32_t *raw = h->bs_entries.narrow; + + /* Expanding in-place from narrow to wide, must use reverse order. */ + for (i = count; i-- > 0; ) { + h->bs_entries.normal[i].flags = be32toh (raw[i * 2 + 1]); + h->bs_entries.normal[i].length = be32toh (raw[i * 2]); + } + } + else { + struct nbd_block_descriptor_ext *wide = h->bs_entries.wide; + + assert (type == NBD_REPLY_TYPE_BLOCK_STATUS_EXT); + for (i = 0; i < count; i++) { + h->bs_entries.normal[i].length = be64toh (wide[i].length); + h->bs_entries.normal[i].flags = be64toh (wide[i].status_flags); + } } /* Look up the context ID. */ -- 2.40.1
Overcome the inherent 32-bit limitation of our existing nbd_block_status command by adding a 64-bit variant. The command sent to the server does not change, but the user's callback is now handed 64-bit information regardless of whether the server replies with 32- or 64-bit extents. Unit tests prove that the new API works in each of C, Python, OCaml, and Go bindings. We can also get rid of the temporary hack added to appease the compiler in an earlier patch. Signed-off-by: Eric Blake <eblake@redhat.com> --- docs/libnbd.pod | 18 +-- sh/nbdsh.pod | 2 +- generator/API.ml | 146 +++++++++++++++++++--- generator/OCaml.ml | 1 - generator/Python.ml | 1 - lib/rw.c | 48 +++++-- python/t/465-block-status-64.py | 56 +++++++++ ocaml/tests/Makefile.am | 1 + ocaml/tests/test_465_block_status_64.ml | 58 +++++++++ tests/meta-base-allocation.c | 104 +++++++++++++-- fuzzing/libnbd-fuzz-wrapper.c | 20 ++- golang/Makefile.am | 1 + golang/libnbd_465_block_status_64_test.go | 119 ++++++++++++++++++ 13 files changed, 527 insertions(+), 48 deletions(-) create mode 100644 python/t/465-block-status-64.py create mode 100644 ocaml/tests/test_465_block_status_64.ml create mode 100644 golang/libnbd_465_block_status_64_test.go diff --git a/docs/libnbd.pod b/docs/libnbd.pod index 72f74053..ddbb09f3 100644 --- a/docs/libnbd.pod +++ b/docs/libnbd.pod @@ -661,14 +661,14 @@ In order to utilize block status, the client must call L<nbd_add_meta_context(3)> prior to connecting, for each meta context in which it is interested, then check L<nbd_can_meta_context(3)> after connection to see which contexts the server actually supports. If a -context is supported, the client can then use L<nbd_block_status(3)> -with a callback function that will receive an array of 32-bit integer -pairs describing consecutive extents within a context. In each pair, -the first integer is the length of the extent, the second is a bitmask -description of that extent (for the "base:allocation" context, the -bitmask may include C<LIBNBD_STATE_HOLE> for unallocated portions of -the file, and/or C<LIBNBD_STATE_ZERO> for portions of the file known -to read as zero). +context is supported, the client can then use +L<nbd_block_status_64(3)> with a callback function that will receive +an array of structs describing consecutive extents within a context. +Each struct gives the length of the extent, then a bitmask description +of that extent (for the "base:allocation" context, the bitmask may +include C<LIBNBD_STATE_HOLE> for unallocated portions of the file, +and/or C<LIBNBD_STATE_ZERO> for portions of the file known to read as +zero). There is a full example of requesting meta context and using block status available at @@ -917,7 +917,7 @@ will tie up resources until L<nbd_close(3)> is eventually reached). =head2 Callbacks with C<int *error> parameter Some of the high-level commands (L<nbd_pread_structured(3)>, -L<nbd_block_status(3)>) involve the use of a callback function invoked +L<nbd_block_status_64(3)>) involve the use of a callback function invoked by the state machine at appropriate points in the server's reply before the overall command is complete. These callback functions, along with all of the completion callbacks, include a parameter diff --git a/sh/nbdsh.pod b/sh/nbdsh.pod index d50b738a..d222c0ff 100644 --- a/sh/nbdsh.pod +++ b/sh/nbdsh.pod @@ -59,7 +59,7 @@ Display brief command line help and exit. =item B<--base-allocation> Request the use of the "base:allocation" meta context, which is the -most common context used with L<nbd_block_status(3)>. This is +most common context used with L<nbd_block_status_64(3)>. This is equivalent to calling S<C<h.set_meta_context(nbd.CONTEXT_BASE_ALLOCATION)>> in the shell prior to connecting, and works even when combined with C<--uri> (while diff --git a/generator/API.ml b/generator/API.ml index 1da1ec53..f04f4b79 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -1565,7 +1565,7 @@ "add_meta_context", { During connection libnbd can negotiate zero or more metadata contexts with the server. Metadata contexts are features (such as C<\"base:allocation\">) which describe information returned -by the L<nbd_block_status(3)> command (for C<\"base:allocation\"> +by the L<nbd_block_status_64(3)> command (for C<\"base:allocation\"> this is whether blocks of data are allocated, zero or sparse). This call adds one metadata context to the list to be negotiated. @@ -1592,7 +1592,7 @@ "add_meta_context", { Other metadata contexts are server-specific, but include C<\"qemu:dirty-bitmap:...\"> and C<\"qemu:allocation-depth\"> for qemu-nbd (see qemu-nbd I<-B> and I<-A> options)."; - see_also = [Link "block_status"; Link "can_meta_context"; + see_also = [Link "block_status_64"; Link "can_meta_context"; Link "get_nr_meta_contexts"; Link "get_meta_context"; Link "clear_meta_contexts"]; }; @@ -1605,14 +1605,14 @@ "get_nr_meta_contexts", { During connection libnbd can negotiate zero or more metadata contexts with the server. Metadata contexts are features (such as C<\"base:allocation\">) which describe information returned -by the L<nbd_block_status(3)> command (for C<\"base:allocation\"> +by the L<nbd_block_status_64(3)> command (for C<\"base:allocation\"> this is whether blocks of data are allocated, zero or sparse). This command returns how many meta contexts have been added to the list to request from the server via L<nbd_add_meta_context(3)>. The server is not obligated to honor all of the requests; to see what it actually supports, see L<nbd_can_meta_context(3)>."; - see_also = [Link "block_status"; Link "can_meta_context"; + see_also = [Link "block_status_64"; Link "can_meta_context"; Link "add_meta_context"; Link "get_meta_context"; Link "clear_meta_contexts"]; }; @@ -1625,13 +1625,13 @@ "get_meta_context", { During connection libnbd can negotiate zero or more metadata contexts with the server. Metadata contexts are features (such as C<\"base:allocation\">) which describe information returned -by the L<nbd_block_status(3)> command (for C<\"base:allocation\"> +by the L<nbd_block_status_64(3)> command (for C<\"base:allocation\"> this is whether blocks of data are allocated, zero or sparse). This command returns the i'th meta context request, as added by L<nbd_add_meta_context(3)>, and bounded by L<nbd_get_nr_meta_contexts(3)>."; - see_also = [Link "block_status"; Link "can_meta_context"; + see_also = [Link "block_status_64"; Link "can_meta_context"; Link "add_meta_context"; Link "get_nr_meta_contexts"; Link "clear_meta_contexts"]; }; @@ -1645,7 +1645,7 @@ "clear_meta_contexts", { During connection libnbd can negotiate zero or more metadata contexts with the server. Metadata contexts are features (such as C<\"base:allocation\">) which describe information returned -by the L<nbd_block_status(3)> command (for C<\"base:allocation\"> +by the L<nbd_block_status_64(3)> command (for C<\"base:allocation\"> this is whether blocks of data are allocated, zero or sparse). This command resets the list of meta contexts to request back to @@ -1654,7 +1654,7 @@ "clear_meta_contexts", { negotiation mode is selected (see L<nbd_set_opt_mode(3)>), for altering the list of attempted contexts between subsequent export queries."; - see_also = [Link "block_status"; Link "can_meta_context"; + see_also = [Link "block_status_64"; Link "can_meta_context"; Link "add_meta_context"; Link "get_nr_meta_contexts"; Link "get_meta_context"; Link "set_opt_mode"]; }; @@ -2281,7 +2281,7 @@ "can_meta_context", { ^ non_blocking_test_call_description; see_also = [SectionLink "Flag calls"; Link "opt_info"; Link "add_meta_context"; - Link "block_status"; Link "aio_block_status"; + Link "block_status_64"; Link "aio_block_status_64"; Link "set_request_meta_context"; Link "opt_set_meta_context"]; }; @@ -2712,7 +2712,7 @@ "block_status", { optargs = [ OFlags ("flags", cmd_flags, Some ["REQ_ONE"]) ]; ret = RErr; permitted_states = [ Connected ]; - shortdesc = "send block status command to the NBD server"; + shortdesc = "send block status command, with 32-bit callback"; longdesc = "\ Issue the block status command to the NBD server. If supported by the server, this causes metadata context @@ -2727,7 +2727,15 @@ "block_status", { The NBD protocol does not yet have a way for a client to learn if the server will enforce an even smaller maximum block status size, although a future extension may add a constraint visible in -L<nbd_get_block_size(3)>. +L<nbd_get_block_size(3)>. Furthermore, this function is inherently +limited to 32-bit values. If the server replies with a larger +extent, the length of that extent will be truncated to just +below 32 bits and any further extents from the server will be +ignored. If the server replies with a status value larger than +32 bits (only possible when extended headers are in use), the +callback function will be passed an C<EOVERFLOW> error. To get +the full extent information from a server that supports 64-bit +extents, you must use L<nbd_block_status_64(3)>. Depending on which metadata contexts were enabled before connecting (see L<nbd_add_meta_context(3)>) and which are @@ -2747,7 +2755,7 @@ "block_status", { array is an array of pairs of integers with the first entry in each pair being the length (in bytes) of the block and the second entry being a status/flags field which is specific to the metadata context. -(The number of pairs passed to the function is C<nr_entries/2>.) The +The number of pairs passed to the function is C<nr_entries/2>. The NBD protocol document in the section about C<NBD_REPLY_TYPE_BLOCK_STATUS> describes the meaning of this array; for contexts known to libnbd, B<E<lt>libnbd.hE<gt>> contains constants @@ -2771,10 +2779,79 @@ "block_status", { does not exceed C<count> bytes; however, libnbd does not validate that the server obeyed the flag." ^ strict_call_description; - see_also = [Link "add_meta_context"; Link "can_meta_context"; + see_also = [Link "block_status_64"; + Link "add_meta_context"; Link "can_meta_context"; Link "aio_block_status"; Link "set_strict_mode"]; }; + "block_status_64", { + default_call with + args = [ UInt64 "count"; UInt64 "offset"; Closure extent64_closure ]; + optargs = [ OFlags ("flags", cmd_flags, Some ["REQ_ONE"]) ]; + ret = RErr; + permitted_states = [ Connected ]; + shortdesc = "send block status command, with 64-bit callback"; + longdesc = "\ +Issue the block status command to the NBD server. If +supported by the server, this causes metadata context +information about blocks beginning from the specified +offset to be returned. The C<count> parameter is a hint: the +server may choose to return less status, or the final block +may extend beyond the requested range. If multiple contexts +are supported, the number of blocks and cumulative length +of those blocks need not be identical between contexts. + +Note that not all servers can support a C<count> of 4GiB or larger. +The NBD protocol does not yet have a way for a client to learn if +the server will enforce an even smaller maximum block status size, +although a future extension may add a constraint visible in +L<nbd_get_block_size(3)>. + +Depending on which metadata contexts were enabled before +connecting (see L<nbd_add_meta_context(3)>) and which are +supported by the server (see L<nbd_can_meta_context(3)>) this call +returns information about extents by calling back to the +C<extent64> function. The callback cannot call C<nbd_*> APIs on the +same handle since it holds the handle lock and will +cause a deadlock. If the callback returns C<-1>, and no earlier +error has been detected, then the overall block status command +will fail with any non-zero value stored into the callback's +C<error> parameter (with a default of C<EPROTO>); but any further +contexts will still invoke the callback. + +The C<extent64> function is called once per type of metadata available, +with the C<user_data> passed to this function. The C<metacontext> +parameter is a string such as C<\"base:allocation\">. The C<entries> +array is an array of B<nbd_extent> structs, containing length (in bytes) +of the block and a status/flags field which is specific to the metadata +context. The number of array entries passed to the function is +C<nr_entries>. The NBD protocol document in the section about +C<NBD_REPLY_TYPE_BLOCK_STATUS> describes the meaning of this array; +for contexts known to libnbd, B<E<lt>libnbd.hE<gt>> contains constants +beginning with C<LIBNBD_STATE_> that may help decipher the values. +On entry to the callback, the C<error> parameter contains the errno +value of any previously detected error. + +It is possible for the extent function to be called +more times than you expect (if the server is buggy), +so always check the C<metacontext> field to ensure you +are receiving the data you expect. It is also possible +that the extent function is not called at all, even for +metadata contexts that you requested. This indicates +either that the server doesn't support the context +or for some other reason cannot return the data. + +The C<flags> parameter may be C<0> for no flags, or may contain +C<LIBNBD_CMD_FLAG_REQ_ONE> meaning that the server should +return only one extent per metadata context where that extent +does not exceed C<count> bytes; however, libnbd does not +validate that the server obeyed the flag." +^ strict_call_description; + see_also = [Link "block_status"; + Link "add_meta_context"; Link "can_meta_context"; + Link "aio_block_status_64"; Link "set_strict_mode"]; + }; + "poll", { default_call with args = [ Int "timeout" ]; ret = RInt; @@ -3369,7 +3446,7 @@ "aio_block_status", { OFlags ("flags", cmd_flags, Some ["REQ_ONE"]) ]; ret = RCookie; permitted_states = [ Connected ]; - shortdesc = "send block status command to the NBD server"; + shortdesc = "send block status command, with 32-bit callback"; longdesc = "\ Send the block status command to the NBD server. @@ -3377,13 +3454,48 @@ "aio_block_status", { Or supply the optional C<completion_callback> which will be invoked as described in L<libnbd(3)/Completion callbacks>. -Other parameters behave as documented in L<nbd_block_status(3)>." +Other parameters behave as documented in L<nbd_block_status(3)>. + +This function is inherently limited to 32-bit values. If the +server replies with a larger extent, the length of that extent +will be truncated to just below 32 bits and any further extents +from the server will be ignored. If the server replies with a +status value larger than 32 bits (only possible when extended +headers are in use), the callback function will be passed an +C<EOVERFLOW> error. To get the full extent information from a +server that supports 64-bit extents, you must use +L<nbd_aio_block_status_64(3)>. +" ^ strict_call_description; see_also = [SectionLink "Issuing asynchronous commands"; + Link "aio_block_status_64"; Link "can_meta_context"; Link "block_status"; Link "set_strict_mode"]; }; + "aio_block_status_64", { + default_call with + args = [ UInt64 "count"; UInt64 "offset"; Closure extent64_closure ]; + optargs = [ OClosure completion_closure; + OFlags ("flags", cmd_flags, Some ["REQ_ONE"]) ]; + ret = RCookie; + permitted_states = [ Connected ]; + shortdesc = "send block status command, with 64-bit callback"; + longdesc = "\ +Send the block status command to the NBD server. + +To check if the command completed, call L<nbd_aio_command_completed(3)>. +Or supply the optional C<completion_callback> which will be invoked +as described in L<libnbd(3)/Completion callbacks>. + +Other parameters behave as documented in L<nbd_block_status_64(3)>." +^ strict_call_description; + see_also = [SectionLink "Issuing asynchronous commands"; + Link "aio_block_status"; + Link "can_meta_context"; Link "block_status_64"; + Link "set_strict_mode"]; + }; + "aio_get_fd", { default_call with args = []; ret = RFd; @@ -3909,6 +4021,10 @@ let first_version = "set_socket_activation_name", (1, 16); "get_socket_activation_name", (1, 16); + (* Added in 1.17.x development cycle, will be stable and supported in 1.18. *) + "block_status_64", (1, 18); + "aio_block_status_64", (1, 18); + (* These calls are proposed for a future version of libnbd, but * have not been added to any released version so far. "get_tls_certificates", (1, ??); diff --git a/generator/OCaml.ml b/generator/OCaml.ml index ea3b044a..b05cd1d2 100644 --- a/generator/OCaml.ml +++ b/generator/OCaml.ml @@ -593,7 +593,6 @@ let pr "}\n"; pr "\n"; pr "static int\n"; - pr "__attribute__ ((unused)) /* XXX temporary hack */\n"; pr "%s_wrapper " cbname; C.print_cbarg_list ~wrap:true cbargs; pr "\n"; diff --git a/generator/Python.ml b/generator/Python.ml index ccac5fed..6f87b0cf 100644 --- a/generator/Python.ml +++ b/generator/Python.ml @@ -158,7 +158,6 @@ let let print_python_closure_wrapper { cbname; cbargs } = pr "/* Wrapper for %s callback. */\n" cbname; pr "static int\n"; - pr "__attribute__ ((unused)) /* XXX temporary hack */\n"; pr "%s_wrapper " cbname; C.print_cbarg_list ~wrap:true cbargs; pr "\n"; diff --git a/lib/rw.c b/lib/rw.c index 58b05a4b..bea55fa1 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -205,7 +205,7 @@ nbd_unlocked_zero (struct nbd_handle *h, return wait_for_command (h, cookie); } -/* Issue a block status command and wait for the reply. */ +/* Issue a block status command and wait for the reply, 32-bit callback. */ int nbd_unlocked_block_status (struct nbd_handle *h, uint64_t count, uint64_t offset, @@ -223,6 +223,25 @@ nbd_unlocked_block_status (struct nbd_handle *h, return wait_for_command (h, cookie); } +/* Issue a block status command and wait for the reply, 64-bit callback. */ +int +nbd_unlocked_block_status_64 (struct nbd_handle *h, + uint64_t count, uint64_t offset, + nbd_extent64_callback *extent64, + uint32_t flags) +{ + int64_t cookie; + nbd_completion_callback c = NBD_NULL_COMPLETION; + + cookie = nbd_unlocked_aio_block_status_64 (h, count, offset, extent64, &c, + flags); + if (cookie == -1) + return -1; + + assert (CALLBACK_IS_NULL (*extent64)); + return wait_for_command (h, cookie); +} + /* count_err represents the errno to return if bounds check fail */ int64_t nbd_internal_command_common (struct nbd_handle *h, @@ -543,10 +562,10 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, uint32_t flags) { nbd_extent_callback *shim = malloc (sizeof *shim); - struct command_cb cb = { .fn.extent.callback = nbd_convert_extent, - .fn.extent.user_data = shim, - .fn.extent.free = nbd_convert_extent_free, - .completion = *completion }; + nbd_extent64_callback wrapper = { .callback = nbd_convert_extent, + .user_data = shim, + .free = nbd_convert_extent_free, }; + int ret; if (shim == NULL) { set_error (errno, "malloc"); @@ -555,10 +574,25 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, *shim = *extent; SET_CALLBACK_TO_NULL (*extent); + ret = nbd_unlocked_aio_block_status_64 (h, count, offset, &wrapper, + completion, flags); + FREE_CALLBACK (wrapper); + return ret; +} + +int64_t +nbd_unlocked_aio_block_status_64 (struct nbd_handle *h, + uint64_t count, uint64_t offset, + nbd_extent64_callback *extent64, + nbd_completion_callback *completion, + uint32_t flags) +{ + struct command_cb cb = { .fn.extent = *extent64, + .completion = *completion }; + if (h->strict & LIBNBD_STRICT_COMMANDS) { if (!h->structured_replies) { set_error (ENOTSUP, "server does not support structured replies"); - FREE_CALLBACK (cb.fn.extent); return -1; } @@ -566,11 +600,11 @@ nbd_unlocked_aio_block_status (struct nbd_handle *h, set_error (ENOTSUP, "did not negotiate any metadata contexts, " "either you did not call nbd_add_meta_context before " "connecting or the server does not support it"); - FREE_CALLBACK (cb.fn.extent); return -1; } } + SET_CALLBACK_TO_NULL (*extent64); SET_CALLBACK_TO_NULL (*completion); return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb); diff --git a/python/t/465-block-status-64.py b/python/t/465-block-status-64.py new file mode 100644 index 00000000..a8a8eaea --- /dev/null +++ b/python/t/465-block-status-64.py @@ -0,0 +1,56 @@ +# libnbd Python bindings +# Copyright Red Hat +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +import os + +import nbd + +script = "%s/../tests/meta-base-allocation.sh" % os.getenv("srcdir", ".") + +h = nbd.NBD() +h.add_meta_context("base:allocation") +h.connect_command(["nbdkit", "-s", "--exit-with-parent", "-v", "sh", script]) + +entries = [] + + +def f(user_data, metacontext, offset, e, err): + global entries + assert user_data == 42 + assert err.value == 0 + if metacontext != "base:allocation": + return + entries = e + + +h.block_status_64(65536, 0, lambda *args: f(42, *args)) +print("entries = %r" % entries) +assert entries == [(8192, 0), + (8192, 1), + (16384, 3), + (16384, 2), + (16384, 0)] + +h.block_status_64(1024, 32256, lambda *args: f(42, *args)) +print("entries = %r" % entries) +assert entries == [(512, 3), + (16384, 2)] + +h.block_status_64(1024, 32256, lambda *args: f(42, *args), + nbd.CMD_FLAG_REQ_ONE) +print("entries = %r" % entries) +assert entries == [(512, 3)] diff --git a/ocaml/tests/Makefile.am b/ocaml/tests/Makefile.am index 20546350..4ddd0f9f 100644 --- a/ocaml/tests/Makefile.am +++ b/ocaml/tests/Makefile.am @@ -40,6 +40,7 @@ ML_TESTS = \ test_405_pread_structured.ml \ test_410_pwrite.ml \ test_460_block_status.ml \ + test_465_block_status_64.ml \ test_500_aio_pread.ml \ test_505_aio_pread_structured_callback.ml \ test_510_aio_pwrite.ml \ diff --git a/ocaml/tests/test_465_block_status_64.ml b/ocaml/tests/test_465_block_status_64.ml new file mode 100644 index 00000000..0634a025 --- /dev/null +++ b/ocaml/tests/test_465_block_status_64.ml @@ -0,0 +1,58 @@ +(* hey emacs, this is OCaml code: -*- tuareg -*- *) +(* libnbd OCaml test case + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + *) + +open Printf + +let script = + try + let srcdir = Sys.getenv "srcdir" in + sprintf "%s/../../tests/meta-base-allocation.sh" srcdir + with + Not_found -> failwith "error: srcdir is not defined" + +let entries = ref [||] +let f user_data metacontext offset e err = + assert (user_data = 42); + assert (!err = 0); + if metacontext = "base:allocation" then + entries := e; + 0 + +let () = + let nbd = NBD.create () in + NBD.add_meta_context nbd "base:allocation"; + NBD.connect_command nbd ["nbdkit"; "-s"; "--exit-with-parent"; "-v"; + "sh"; script]; + + NBD.block_status_64 nbd 65536_L 0_L (f 42); + assert (!entries = [| 8192_L, 0_L; + 8192_L, 1_L; + 16384_L, 3_L; + 16384_L, 2_L; + 16384_L, 0_L |]); + + NBD.block_status_64 nbd 1024_L 32256_L (f 42); + assert (!entries = [| 512_L, 3_L; + 16384_L, 2_L |]); + + let flags = let open NBD.CMD_FLAG in [REQ_ONE] in + NBD.block_status_64 nbd 1024_L 32256_L (f 42) ~flags; + assert (!entries = [| 512_L, 3_L |]) + +let () = Gc.compact () diff --git a/tests/meta-base-allocation.c b/tests/meta-base-allocation.c index a7b3af09..7697b5da 100644 --- a/tests/meta-base-allocation.c +++ b/tests/meta-base-allocation.c @@ -32,10 +32,13 @@ #define BOGUS_CONTEXT "x-libnbd:nosuch" -static int check_extent (void *data, - const char *metacontext, - uint64_t offset, - uint32_t *entries, size_t nr_entries, int *error); +static int check_extent32 (void *data, const char *metacontext, + uint64_t offset, + uint32_t *entries, size_t nr_entries, int *error); + +static int check_extent64 (void *data, const char *metacontext, + uint64_t offset, + nbd_extent *entries, size_t nr_entries, int *error); int main (int argc, char *argv[]) @@ -43,8 +46,10 @@ main (int argc, char *argv[]) struct nbd_handle *nbd; char plugin_path[256]; int id; - nbd_extent_callback extent_callback = { .callback = check_extent, - .user_data = &id }; + nbd_extent_callback extent32_callback = { .callback = check_extent32, + .user_data = &id }; + nbd_extent64_callback extent64_callback = { .callback = check_extent64, + .user_data = &id }; int r; const char *s; char *tmp; @@ -150,23 +155,36 @@ main (int argc, char *argv[]) /* Read the block status. */ id = 1; - if (nbd_block_status (nbd, 65536, 0, extent_callback, 0) == -1) { + if (nbd_block_status (nbd, 65536, 0, extent32_callback, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (nbd_block_status_64 (nbd, 65536, 0, extent64_callback, 0) == -1) { fprintf (stderr, "%s\n", nbd_get_error ()); exit (EXIT_FAILURE); } id = 2; - if (nbd_block_status (nbd, 1024, 32768-512, extent_callback, 0) == -1) { + if (nbd_block_status (nbd, 1024, 32768-512, extent32_callback, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (nbd_block_status_64 (nbd, 1024, 32768-512, extent64_callback, 0) == -1) { fprintf (stderr, "%s\n", nbd_get_error ()); exit (EXIT_FAILURE); } id = 3; - if (nbd_block_status (nbd, 1024, 32768-512, extent_callback, + if (nbd_block_status (nbd, 1024, 32768-512, extent32_callback, LIBNBD_CMD_FLAG_REQ_ONE) == -1) { fprintf (stderr, "%s\n", nbd_get_error ()); exit (EXIT_FAILURE); } + if (nbd_block_status_64 (nbd, 1024, 32768-512, extent64_callback, + LIBNBD_CMD_FLAG_REQ_ONE) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } if (nbd_shutdown (nbd, 0) == -1) { fprintf (stderr, "%s\n", nbd_get_error ()); @@ -178,10 +196,8 @@ main (int argc, char *argv[]) } static int -check_extent (void *data, - const char *metacontext, - uint64_t offset, - uint32_t *entries, size_t nr_entries, int *error) +check_extent32 (void *data, const char *metacontext, uint64_t offset, + uint32_t *entries, size_t nr_entries, int *error) { size_t i; int id; @@ -235,3 +251,65 @@ check_extent (void *data, return 0; } + +static int +check_extent64 (void *data, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t nr_entries, int *error) +{ + size_t i; + int id; + + id = * (int *)data; + + printf ("extent: id=%d, metacontext=%s, offset=%" PRIu64 ", " + "nr_entries=%zu, error=%d\n", + id, metacontext, offset, nr_entries, *error); + + assert (*error == 0); + if (strcmp (metacontext, LIBNBD_CONTEXT_BASE_ALLOCATION) == 0) { + for (i = 0; i < nr_entries; i++) { + printf ("\t%zu\tlength=%" PRIu64 ", status=%" PRIu64 "\n", + i, entries[i].length, entries[i].flags); + } + fflush (stdout); + + switch (id) { + case 1: + assert (nr_entries == 5); + assert (entries[0].length == 8192); + assert (entries[0].flags == 0); + assert (entries[1].length == 8192); + assert (entries[1].flags == LIBNBD_STATE_HOLE); + assert (entries[2].length == 16384); + assert (entries[2].flags == (LIBNBD_STATE_HOLE|LIBNBD_STATE_ZERO)); + assert (entries[3].length == 16384); + assert (entries[3].flags == LIBNBD_STATE_ZERO); + assert (entries[4].length == 16384); + assert (entries[4].flags == 0); + break; + + case 2: + assert (nr_entries == 2); + assert (entries[0].length == 512); + assert (entries[0].flags == (LIBNBD_STATE_HOLE|LIBNBD_STATE_ZERO)); + assert (entries[1].length == 16384); + assert (entries[1].flags == LIBNBD_STATE_ZERO); + break; + + case 3: + assert (nr_entries == 1); + assert (entries[0].length == 512); + assert (entries[0].flags == (LIBNBD_STATE_HOLE|LIBNBD_STATE_ZERO)); + break; + + default: + abort (); + } + + } + else + fprintf (stderr, "warning: ignored unexpected meta context %s\n", + metacontext); + + return 0; +} diff --git a/fuzzing/libnbd-fuzz-wrapper.c b/fuzzing/libnbd-fuzz-wrapper.c index e111866f..cbd55380 100644 --- a/fuzzing/libnbd-fuzz-wrapper.c +++ b/fuzzing/libnbd-fuzz-wrapper.c @@ -165,6 +165,17 @@ extent_callback (void *user_data, return 0; } +/* Block status (extents64) callback, does nothing. */ +static int +extent64_callback (void *user_data, + const char *metacontext, + uint64_t offset, nbd_extent *entries, + size_t nr_entries, int *error) +{ + //fprintf (stderr, "extent64 called, nr_entries = %zu\n", nr_entries); + return 0; +} + /* This is the client (parent process) running libnbd. */ static char buf[512]; static char prbuf[65536]; @@ -213,7 +224,7 @@ client (int sock) }, 0); - /* Test block status. */ + /* Test both sizes of block status. */ nbd_block_status (nbd, length, 0, (nbd_extent_callback) { .callback = extent_callback, @@ -221,6 +232,13 @@ client (int sock) .free = NULL }, 0); + nbd_block_status_64 (nbd, length, 0, + (nbd_extent64_callback) { + .callback = extent64_callback, + .user_data = NULL, + .free = NULL + }, + 0); nbd_shutdown (nbd, 0); } diff --git a/golang/Makefile.am b/golang/Makefile.am index fac65248..b4151250 100644 --- a/golang/Makefile.am +++ b/golang/Makefile.am @@ -43,6 +43,7 @@ source_files = \ libnbd_405_pread_structured_test.go \ libnbd_410_pwrite_test.go \ libnbd_460_block_status_test.go \ + libnbd_465_block_status_64_test.go \ libnbd_500_aio_pread_test.go \ libnbd_510_aio_pwrite_test.go \ libnbd_590_aio_copy_test.go \ diff --git a/golang/libnbd_465_block_status_64_test.go b/golang/libnbd_465_block_status_64_test.go new file mode 100644 index 00000000..7659a21d --- /dev/null +++ b/golang/libnbd_465_block_status_64_test.go @@ -0,0 +1,119 @@ +/* libnbd golang tests + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +package libnbd + +import ( + "fmt" + "os" + "strings" + "testing" +) + +var entries64 []LibnbdExtent + +func mcf64(metacontext string, offset uint64, e []LibnbdExtent, error *int) int { + if *error != 0 { + panic("expected *error == 0") + } + if metacontext == "base:allocation" { + entries64 = e + } + return 0 +} + +// Seriously WTF? +func mc64_compare(a1 []LibnbdExtent, a2 []LibnbdExtent) bool { + if len(a1) != len(a2) { + return false + } + for i := 0; i < len(a1); i++ { + if a1[i] != a2[i] { + return false + } + } + return true +} + +func mc64_to_string(a []LibnbdExtent) string { + ss := make([]string, len(a)) + for i := 0; i < len(a); i++ { + ss[i] = fmt.Sprintf("%#v", a[i]) + } + return strings.Join(ss, ", ") +} + +func Test465BlockStatus64(t *testing.T) { + srcdir := os.Getenv("abs_top_srcdir") + script := srcdir + "/tests/meta-base-allocation.sh" + + h, err := Create() + if err != nil { + t.Fatalf("could not create handle: %s", err) + } + defer h.Close() + + err = h.AddMetaContext("base:allocation") + if err != nil { + t.Fatalf("%s", err) + } + err = h.ConnectCommand([]string{ + "nbdkit", "-s", "--exit-with-parent", "-v", + "sh", script, + }) + if err != nil { + t.Fatalf("%s", err) + } + + err = h.BlockStatus64(65536, 0, mcf64, nil) + if err != nil { + t.Fatalf("%s", err) + } + if !mc64_compare(entries64, []LibnbdExtent{ + {8192, 0}, + {8192, 1}, + {16384, 3}, + {16384, 2}, + {16384, 0}, + }) { + t.Fatalf("unexpected entries (1): %s", mc64_to_string(entries64)) + } + + err = h.BlockStatus64(1024, 32256, mcf64, nil) + if err != nil { + t.Fatalf("%s", err) + } + if !mc64_compare(entries64, []LibnbdExtent{ + {512, 3}, + {16384, 2}, + }) { + t.Fatalf("unexpected entries (2): %s", mc64_to_string(entries64)) + } + + var optargs BlockStatus64Optargs + optargs.FlagsSet = true + optargs.Flags = CMD_FLAG_REQ_ONE + err = h.BlockStatus64(1024, 32256, mcf64, &optargs) + if err != nil { + t.Fatalf("%s", err) + } + if !mc64_compare(entries64, []LibnbdExtent{{512, 3}}) { + t.Fatalf("unexpected entries (3): %s", mc64_to_string(entries64)) + } + +} -- 2.40.1
The new NBD_OPT_EXTENDED_HEADERS feature is worth using by default, but there may be cases where the user explicitly wants to stick with the older 32-bit headers. nbd_set_request_extended_headers() will let the client override the default, nbd_get_request_extended_headers() determines the current state of the request, and nbd_get_extended_headers_negotiated() determines what the client and server actually settled on. These use nbd_set_request_structured_replies() and friends as a template. Note that this patch just adds the API for controlling the defaults, but ignores the state variable; a later one will then tweak the state machine to actually request structured headers when the state variable is set, as well as add nbd_opt_extended_headers() for manual control. The intent is that because extended headers takes priority over structured replies, the functions will interact as follows during the automatic handshaking that takes place prior to nbd_opt_set_mode() relinquishing control to the client in negotiation mode: 1. client default: request_eh=1, request_sr=1 => try EXTENDED_HEADERS - server supports it: nothing further to do, use extended headers, but also report structured replies as active - server lacks it: behave like case 2 2. request_eh=0 (explicit client downgrade), request_sr=1 (left at default) => try STRUCTURED_REPLY - server supports it: expect structured replies - server lacks it: expect simple replies 3. request_sr=0 (explicit client downgrade), request_eh ignored => don't try either handshake - expect simple replies Client code that wants to manually force simple replies only has to disable structured replies; and only code that wants to disable extended headers but still use structured replies has to use the new nbd_set_request_extended_headers() API. Until a later patch adds an explicit API nbd_opt_extended_headers(), there is no way to request extended headers after structured replies are already negotiated. Signed-off-by: Eric Blake <eblake@redhat.com> --- lib/internal.h | 1 + generator/API.ml | 115 ++++++++++++++++-- generator/states-newstyle-opt-starttls.c | 1 + .../states-newstyle-opt-structured-reply.c | 3 +- lib/handle.c | 23 ++++ python/t/110-defaults.py | 1 + python/t/120-set-non-defaults.py | 2 + ocaml/tests/test_110_defaults.ml | 2 + ocaml/tests/test_120_set_non_defaults.ml | 3 + golang/libnbd_110_defaults_test.go | 8 ++ golang/libnbd_120_set_non_defaults_test.go | 12 ++ 11 files changed, 160 insertions(+), 11 deletions(-) diff --git a/lib/internal.h b/lib/internal.h index e5423245..2948b77b 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -111,6 +111,7 @@ struct nbd_handle { char *tls_psk_file; /* PSK filename, NULL = no PSK */ /* Extended headers. */ + bool request_eh; /* Whether to request extended headers */ bool extended_headers; /* If we negotiated NBD_OPT_EXTENDED_HEADERS */ /* Desired metadata contexts. */ diff --git a/generator/API.ml b/generator/API.ml index f04f4b79..7616990a 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -808,6 +808,77 @@ "get_tls_psk_file", { }; *) + "set_request_extended_headers", { + default_call with + args = [Bool "request"]; ret = RErr; + permitted_states = [ Created ]; + shortdesc = "control use of extended headers"; + longdesc = "\ +By default, libnbd tries to negotiate extended headers with the +server, as this protocol extension permits the use of 64-bit +zero, trim, and block status actions. However, +for integration testing, it can be useful to clear this flag +rather than find a way to alter the server to fail the negotiation +request. + +For backwards compatibility, the setting of this knob is ignored +if L<nbd_set_request_structured_replies(3)> is also set to false, +since the use of extended headers implies structured replies."; + see_also = [Link "get_request_extended_headers"; + Link "set_handshake_flags"; Link "set_strict_mode"; + Link "get_extended_headers_negotiated"; + Link "zero"; Link "trim"; Link "cache"; + Link "block_status_64"; + Link "set_request_structured_replies"]; + }; + + "get_request_extended_headers", { + default_call with + args = []; ret = RBool; + may_set_error = false; + shortdesc = "see if extended headers are attempted"; + longdesc = "\ +Return the state of the request extended headers flag on this +handle. + +B<Note:> If you want to find out if extended headers were actually +negotiated on a particular connection use +L<nbd_get_extended_headers_negotiated(3)> instead."; + see_also = [Link "set_request_extended_headers"; + Link "get_extended_headers_negotiated"; + Link "get_request_extended_headers"]; + }; + + "get_extended_headers_negotiated", { + default_call with + args = []; ret = RBool; + permitted_states = [ Negotiating; Connected; Closed ]; + shortdesc = "see if extended headers are in use"; + longdesc = "\ +After connecting you may call this to find out if the connection is +using extended headers. + +When extended headers are not in use, commands are limited to a +32-bit length, even when the libnbd API uses a 64-bit parameter +to express the length. But even when extended headers are +supported, the server may enforce other limits, visible through +L<nbd_get_block_size(3)>. + +Note that when extended headers are negotiated, you should +prefer the use of L<nbd_block_status_64(3)> instead of +L<nbd_block_status(3)> if any of the meta contexts you requested +via L<nbd_add_meta_context(3)> might return 64-bit status +values; however, all of the well-known meta contexts covered +by current C<LIBNBD_CONTEXT_*> constants only return 32-bit +status."; + see_also = [Link "set_request_extended_headers"; + Link "get_request_extended_headers"; + Link "zero"; Link "trim"; Link "cache"; + Link "block_status_64"; Link "get_block_size"; + Link "get_protocol"; + Link "get_structured_replies_negotiated"]; + }; + "set_request_structured_replies", { default_call with args = [Bool "request"]; ret = RErr; @@ -821,12 +892,16 @@ "set_request_structured_replies", { rather than find a way to alter the server to fail the negotiation request. It is also useful to set this to false prior to using L<nbd_set_opt_mode(3)> if it is desired to control when to send -L<nbd_opt_structured_reply(3)> during negotiation."; +L<nbd_opt_structured_reply(3)> during negotiation. + +Note that setting this knob to false also disables any automatic +request for extended headers."; see_also = [Link "get_request_structured_replies"; Link "set_handshake_flags"; Link "set_strict_mode"; Link "opt_structured_reply"; Link "get_structured_replies_negotiated"; - Link "can_meta_context"; Link "can_df"]; + Link "can_meta_context"; Link "can_df"; + Link "set_request_extended_headers"]; }; "get_request_structured_replies", { @@ -842,7 +917,8 @@ "get_request_structured_replies", { negotiated on a particular connection use L<nbd_get_structured_replies_negotiated(3)> instead."; see_also = [Link "set_request_structured_replies"; - Link "get_structured_replies_negotiated"]; + Link "get_structured_replies_negotiated"; + Link "get_request_extended_headers"]; }; "get_structured_replies_negotiated", { @@ -854,11 +930,17 @@ "get_structured_replies_negotiated", { After connecting you may call this to find out if the connection is using structured replies. Note that this setting is sticky; this can return true even after a second L<nbd_opt_structured_reply(3)> -returns false because the server detected a duplicate request."; +returns false because the server detected a duplicate request. + +Note that if the connection negotiates extended headers, this +function returns true (as extended headers imply structured +replies) even if no explicit request for structured replies was +attempted."; see_also = [Link "set_request_structured_replies"; Link "get_request_structured_replies"; Link "opt_structured_reply"; - Link "get_protocol"]; + Link "get_protocol"; + Link "get_extended_headers_negotiated"]; }; "set_request_meta_context", { @@ -2623,7 +2705,9 @@ "trim", { or there is an error. Note this will generally return an error if L<nbd_can_trim(3)> is false or L<nbd_is_read_only(3)> is true. -Note that not all servers can support a C<count> of 4GiB or larger. +Note that not all servers can support a C<count> of 4GiB or larger; +L<nbd_get_extended_headers_negotiated(3)> indicates which servers +will parse a request larger than 32 bits. The NBD protocol does not yet have a way for a client to learn if the server will enforce an even smaller maximum trim size, although a future extension may add a constraint visible in @@ -2654,7 +2738,9 @@ "cache", { this command. Note this will generally return an error if L<nbd_can_cache(3)> is false. -Note that not all servers can support a C<count> of 4GiB or larger. +Note that not all servers can support a C<count> of 4GiB or larger; +L<nbd_get_extended_headers_negotiated(3)> indicates which servers +will parse a request larger than 32 bits. The NBD protocol does not yet have a way for a client to learn if the server will enforce an even smaller maximum cache size, although a future extension may add a constraint visible in @@ -2683,7 +2769,9 @@ "zero", { or there is an error. Note this will generally return an error if L<nbd_can_zero(3)> is false or L<nbd_is_read_only(3)> is true. -Note that not all servers can support a C<count> of 4GiB or larger. +Note that not all servers can support a C<count> of 4GiB or larger; +L<nbd_get_extended_headers_negotiated(3)> indicates which servers +will parse a request larger than 32 bits. The NBD protocol does not yet have a way for a client to learn if the server will enforce an even smaller maximum zero size, although a future extension may add a constraint visible in @@ -2723,7 +2811,9 @@ "block_status", { are supported, the number of blocks and cumulative length of those blocks need not be identical between contexts. -Note that not all servers can support a C<count> of 4GiB or larger. +Note that not all servers can support a C<count> of 4GiB or larger; +L<nbd_get_extended_headers_negotiated(3)> indicates which servers +will parse a request larger than 32 bits. The NBD protocol does not yet have a way for a client to learn if the server will enforce an even smaller maximum block status size, although a future extension may add a constraint visible in @@ -2801,7 +2891,9 @@ "block_status_64", { are supported, the number of blocks and cumulative length of those blocks need not be identical between contexts. -Note that not all servers can support a C<count> of 4GiB or larger. +Note that not all servers can support a C<count> of 4GiB or larger; +L<nbd_get_extended_headers_negotiated(3)> indicates which servers +will parse a request larger than 32 bits. The NBD protocol does not yet have a way for a client to learn if the server will enforce an even smaller maximum block status size, although a future extension may add a constraint visible in @@ -4024,6 +4116,9 @@ let first_version = (* Added in 1.17.x development cycle, will be stable and supported in 1.18. *) "block_status_64", (1, 18); "aio_block_status_64", (1, 18); + "set_request_extended_headers", (1, 18); + "get_request_extended_headers", (1, 18); + "get_extended_headers_negotiated", (1, 18); (* These calls are proposed for a future version of libnbd, but * have not been added to any released version so far. diff --git a/generator/states-newstyle-opt-starttls.c b/generator/states-newstyle-opt-starttls.c index 700604dd..e497548c 100644 --- a/generator/states-newstyle-opt-starttls.c +++ b/generator/states-newstyle-opt-starttls.c @@ -86,6 +86,7 @@ NEWSTYLE.OPT_STARTTLS.CHECK_REPLY: } nbd_internal_reset_size_and_flags (h); h->structured_replies = false; + h->extended_headers = false; h->meta_valid = false; new_sock = nbd_internal_crypto_create_session (h, h->sock); if (new_sock == NULL) { diff --git a/generator/states-newstyle-opt-structured-reply.c b/generator/states-newstyle-opt-structured-reply.c index 18245054..bb94b70a 100644 --- a/generator/states-newstyle-opt-structured-reply.c +++ b/generator/states-newstyle-opt-structured-reply.c @@ -25,7 +25,7 @@ NEWSTYLE.OPT_STRUCTURED_REPLY.START: assert (h->opt_mode); else { assert (CALLBACK_IS_NULL (h->opt_cb.completion)); - if (!h->request_sr) { + if (!h->request_sr || h->structured_replies) { if (h->opt_mode) SET_NEXT_STATE (%.NEGOTIATING); else @@ -84,6 +84,7 @@ NEWSTYLE.OPT_STRUCTURED_REPLY.CHECK_REPLY: err = 0; break; case NBD_REP_ERR_INVALID: + case NBD_REP_ERR_EXT_HEADER_REQD: err = EINVAL; /* fallthrough */ default: diff --git a/lib/handle.c b/lib/handle.c index fba6d1c4..71034835 100644 --- a/lib/handle.c +++ b/lib/handle.c @@ -64,6 +64,7 @@ nbd_create (void) h->unique = 1; h->tls_verify_peer = true; + h->request_eh = true; h->request_sr = true; h->request_meta = true; h->request_block_size = true; @@ -436,6 +437,28 @@ nbd_unlocked_clear_meta_contexts (struct nbd_handle *h) return 0; } + +int +nbd_unlocked_set_request_extended_headers (struct nbd_handle *h, + bool request) +{ + h->request_eh = request; + return 0; +} + +/* NB: may_set_error = false. */ +int +nbd_unlocked_get_request_extended_headers (struct nbd_handle *h) +{ + return h->request_eh; +} + +int +nbd_unlocked_get_extended_headers_negotiated (struct nbd_handle *h) +{ + return h->extended_headers; +} + int nbd_unlocked_set_request_structured_replies (struct nbd_handle *h, bool request) diff --git a/python/t/110-defaults.py b/python/t/110-defaults.py index ed60b786..2e59bf39 100644 --- a/python/t/110-defaults.py +++ b/python/t/110-defaults.py @@ -21,6 +21,7 @@ assert h.get_export_name() == "" assert h.get_full_info() is False assert h.get_tls() == nbd.TLS_DISABLE +assert h.get_request_extended_headers() is True assert h.get_request_structured_replies() is True assert h.get_request_meta_context() is True assert h.get_request_block_size() is True diff --git a/python/t/120-set-non-defaults.py b/python/t/120-set-non-defaults.py index a2fafe0c..3320f8cf 100644 --- a/python/t/120-set-non-defaults.py +++ b/python/t/120-set-non-defaults.py @@ -31,6 +31,8 @@ if h.supports_tls(): h.set_tls(nbd.TLS_ALLOW) assert h.get_tls() == nbd.TLS_ALLOW +h.set_request_extended_headers(False) +assert h.get_request_extended_headers() is False h.set_request_structured_replies(False) assert h.get_request_structured_replies() is False h.set_request_meta_context(False) diff --git a/ocaml/tests/test_110_defaults.ml b/ocaml/tests/test_110_defaults.ml index b95cd1f0..d7b0463f 100644 --- a/ocaml/tests/test_110_defaults.ml +++ b/ocaml/tests/test_110_defaults.ml @@ -26,6 +26,8 @@ let assert (not info); let tls = NBD.get_tls nbd in assert (tls = NBD.TLS.DISABLE); + let eh = NBD.get_request_extended_headers nbd in + assert (eh = true); let sr = NBD.get_request_structured_replies nbd in assert sr; let meta = NBD.get_request_meta_context nbd in diff --git a/ocaml/tests/test_120_set_non_defaults.ml b/ocaml/tests/test_120_set_non_defaults.ml index 478a88f3..4f5c2eb5 100644 --- a/ocaml/tests/test_120_set_non_defaults.ml +++ b/ocaml/tests/test_120_set_non_defaults.ml @@ -39,6 +39,9 @@ let let tls = NBD.get_tls nbd in assert (tls = NBD.TLS.ALLOW); ); + NBD.set_request_extended_headers nbd false; + let eh = NBD.get_request_extended_headers nbd in + assert (eh = false); NBD.set_request_structured_replies nbd false; let sr = NBD.get_request_structured_replies nbd in assert (not sr); diff --git a/golang/libnbd_110_defaults_test.go b/golang/libnbd_110_defaults_test.go index 119cb7ca..510baa39 100644 --- a/golang/libnbd_110_defaults_test.go +++ b/golang/libnbd_110_defaults_test.go @@ -51,6 +51,14 @@ func Test110Defaults(t *testing.T) { t.Fatalf("unexpected tls state") } + eh, err := h.GetRequestExtendedHeaders() + if err != nil { + t.Fatalf("could not get extended headers state: %s", err) + } + if eh != true { + t.Fatalf("unexpected extended headers state") + } + sr, err := h.GetRequestStructuredReplies() if err != nil { t.Fatalf("could not get structured replies state: %s", err) diff --git a/golang/libnbd_120_set_non_defaults_test.go b/golang/libnbd_120_set_non_defaults_test.go index b096c4f6..4c6bb3ce 100644 --- a/golang/libnbd_120_set_non_defaults_test.go +++ b/golang/libnbd_120_set_non_defaults_test.go @@ -81,6 +81,18 @@ func Test120SetNonDefaults(t *testing.T) { } } + err = h.SetRequestExtendedHeaders(false) + if err != nil { + t.Fatalf("could not set extended headers state: %s", err) + } + eh, err := h.GetRequestExtendedHeaders() + if err != nil { + t.Fatalf("could not get extended headers state: %s", err) + } + if eh != false { + t.Fatalf("unexpected extended headers state") + } + err = h.SetRequestStructuredReplies(false) if err != nil { t.Fatalf("could not set structured replies state: %s", err) -- 2.40.1
Although our use of "base:allocation" doesn't require the use of the 64-bit API for flags, we might perform slightly faster for a server that does give us 64-bit extent lengths and honors larger nbd_zero lengths. Signed-off-by: Eric Blake <eblake@redhat.com> --- copy/nbd-ops.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/copy/nbd-ops.c b/copy/nbd-ops.c index f3b3bed3..71ee0a3b 100644 --- a/copy/nbd-ops.c +++ b/copy/nbd-ops.c @@ -428,7 +428,7 @@ nbd_ops_asynch_notify_write (struct rw *rw, size_t index) * request for extents in a single round trip. */ static int add_extent (void *vp, const char *metacontext, - uint64_t offset, uint32_t *entries, size_t nr_entries, + uint64_t offset, nbd_extent *entries, size_t nr_entries, int *error); static void @@ -449,11 +449,11 @@ nbd_ops_get_extents (struct rw *rw, size_t index, size_t i; exts.len = 0; - if (nbd_block_status (nbd, count, offset, - (nbd_extent_callback) { - .user_data = &exts, - .callback = add_extent - }, 0) == -1) { + if (nbd_block_status_64 (nbd, count, offset, + (nbd_extent64_callback) { + .user_data = &exts, + .callback = add_extent + }, 0) == -1) { /* XXX We could call default_get_extents, but unclear if it's * the right thing to do if the server is returning errors. */ @@ -496,7 +496,7 @@ nbd_ops_get_extents (struct rw *rw, size_t index, static int add_extent (void *vp, const char *metacontext, - uint64_t offset, uint32_t *entries, size_t nr_entries, + uint64_t offset, nbd_extent *entries, size_t nr_entries, int *error) { extent_list *ret = vp; @@ -505,25 +505,25 @@ add_extent (void *vp, const char *metacontext, if (strcmp (metacontext, "base:allocation") != 0 || *error) return 0; - for (i = 0; i < nr_entries; i += 2) { + for (i = 0; i < nr_entries; i++) { struct extent e; e.offset = offset; - e.length = entries[i]; + e.length = entries[i].length; /* Note we deliberately don't care about the HOLE flag. There is * no need to read extent that reads as zeroes. We will convert * to it to a hole or allocated extents based on the command line * arguments. */ - e.zero = (entries[i+1] & LIBNBD_STATE_ZERO) != 0; + e.zero = (entries[i].flags & LIBNBD_STATE_ZERO) != 0; if (extent_list_append (ret, e) == -1) { perror ("realloc"); exit (EXIT_FAILURE); } - offset += entries[i]; + offset += entries[i].length; } return 0; -- 2.40.1
On Thu, May 25, 2023 at 08:00:58AM -0500, Eric Blake wrote:
Although our use of "base:allocation" doesn't require the use of the 64-bit API for flags, we might perform slightly faster for a server that does give us 64-bit extent lengths and honors larger nbd_zero lengths.
I think this change may turn out to be more significant than you have describe here. Consider copying from a remote, very large, mostly blank NBD source, and also that our block status querying is synchronous. The old nbdcopy might end up doing lots of serialized round trips over 4GB segments of the source to discover that it is sparse. With this change as I understand it, those would be coalesced into a single RTT. (Similarly when zeroing on the output side, but to a lesser extent because those requests can be issued in parallel.)
Signed-off-by: Eric Blake <eblake@redhat.com> --- copy/nbd-ops.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
diff --git a/copy/nbd-ops.c b/copy/nbd-ops.c index f3b3bed3..71ee0a3b 100644 --- a/copy/nbd-ops.c +++ b/copy/nbd-ops.c @@ -428,7 +428,7 @@ nbd_ops_asynch_notify_write (struct rw *rw, size_t index) * request for extents in a single round trip. */ static int add_extent (void *vp, const char *metacontext, - uint64_t offset, uint32_t *entries, size_t nr_entries, + uint64_t offset, nbd_extent *entries, size_t nr_entries, int *error);
static void @@ -449,11 +449,11 @@ nbd_ops_get_extents (struct rw *rw, size_t index, size_t i;
exts.len = 0; - if (nbd_block_status (nbd, count, offset, - (nbd_extent_callback) { - .user_data = &exts, - .callback = add_extent - }, 0) == -1) { + if (nbd_block_status_64 (nbd, count, offset, + (nbd_extent64_callback) { + .user_data = &exts, + .callback = add_extent + }, 0) == -1) { /* XXX We could call default_get_extents, but unclear if it's * the right thing to do if the server is returning errors. */ @@ -496,7 +496,7 @@ nbd_ops_get_extents (struct rw *rw, size_t index,
static int add_extent (void *vp, const char *metacontext, - uint64_t offset, uint32_t *entries, size_t nr_entries, + uint64_t offset, nbd_extent *entries, size_t nr_entries, int *error) { extent_list *ret = vp; @@ -505,25 +505,25 @@ add_extent (void *vp, const char *metacontext, if (strcmp (metacontext, "base:allocation") != 0 || *error) return 0;
- for (i = 0; i < nr_entries; i += 2) { + for (i = 0; i < nr_entries; i++) { struct extent e;
e.offset = offset; - e.length = entries[i]; + e.length = entries[i].length;
/* Note we deliberately don't care about the HOLE flag. There is * no need to read extent that reads as zeroes. We will convert * to it to a hole or allocated extents based on the command line * arguments. */ - e.zero = (entries[i+1] & LIBNBD_STATE_ZERO) != 0; + e.zero = (entries[i].flags & LIBNBD_STATE_ZERO) != 0;
if (extent_list_append (ret, e) == -1) { perror ("realloc"); exit (EXIT_FAILURE); }
- offset += entries[i]; + offset += entries[i].length; }
return 0; -- 2.40.1
Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Although our use of "base:allocation" doesn't require the use of the 64-bit API for flags, we might perform slightly faster for a server that does give us 64-bit extent lengths. Signed-off-by: Eric Blake <eblake@redhat.com> --- dump/dump.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/dump/dump.c b/dump/dump.c index b4aebe84..71053277 100644 --- a/dump/dump.c +++ b/dump/dump.c @@ -38,7 +38,7 @@ #include "version.h" #include "vector.h" -DEFINE_VECTOR_TYPE (uint32_vector, uint32_t); +DEFINE_VECTOR_TYPE (uint64_vector, uint64_t); static const char *progname; static struct nbd_handle *nbd; @@ -262,10 +262,10 @@ catch_signal (int sig) static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, - uint32_t *entries, size_t nr_entries, + nbd_extent *entries, size_t nr_entries, int *error) { - uint32_vector *list = user_data; + uint64_vector *list = user_data; size_t i; if (strcmp (metacontext, LIBNBD_CONTEXT_BASE_ALLOCATION) != 0) @@ -273,7 +273,8 @@ extent_callback (void *user_data, const char *metacontext, /* Just append the entries we got to the list. */ for (i = 0; i < nr_entries; ++i) { - if (uint32_vector_append (list, entries[i]) == -1) { + if (uint64_vector_append (list, entries[i].length) == -1 || + uint64_vector_append (list, entries[i].flags) == -1) { perror ("realloc"); exit (EXIT_FAILURE); } @@ -284,7 +285,7 @@ extent_callback (void *user_data, const char *metacontext, static bool test_all_zeroes (uint64_t offset, size_t count) { - uint32_vector entries = empty_vector; + uint64_vector entries = empty_vector; size_t i; uint64_t count_read; @@ -296,22 +297,22 @@ test_all_zeroes (uint64_t offset, size_t count) * false, causing the main code to do a full read. We could be * smarter and keep asking the server (XXX). */ - if (nbd_block_status (nbd, count, offset, - (nbd_extent_callback) { - .callback = extent_callback, - .user_data = &entries }, - 0) == -1) { + if (nbd_block_status_64 (nbd, count, offset, + (nbd_extent64_callback) { + .callback = extent_callback, + .user_data = &entries }, + 0) == -1) { fprintf (stderr, "%s: %s\n", progname, nbd_get_error ()); exit (EXIT_FAILURE); } count_read = 0; for (i = 0; i < entries.len; i += 2) { - uint32_t len = entries.ptr[i]; - uint32_t type = entries.ptr[i+1]; + uint64_t len = entries.ptr[i]; + uint64_t type = entries.ptr[i+1]; count_read += len; - if (!(type & 2)) /* not zero */ + if (!(type & LIBNBD_STATE_ZERO)) /* not zero */ return false; } -- 2.40.1
On Thu, May 25, 2023 at 08:00:59AM -0500, Eric Blake wrote:
Although our use of "base:allocation" doesn't require the use of the 64-bit API for flags, we might perform slightly faster for a server that does give us 64-bit extent lengths.
Signed-off-by: Eric Blake <eblake@redhat.com> --- dump/dump.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-)
diff --git a/dump/dump.c b/dump/dump.c index b4aebe84..71053277 100644 --- a/dump/dump.c +++ b/dump/dump.c @@ -38,7 +38,7 @@ #include "version.h" #include "vector.h"
-DEFINE_VECTOR_TYPE (uint32_vector, uint32_t); +DEFINE_VECTOR_TYPE (uint64_vector, uint64_t);
static const char *progname; static struct nbd_handle *nbd; @@ -262,10 +262,10 @@ catch_signal (int sig) static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, - uint32_t *entries, size_t nr_entries, + nbd_extent *entries, size_t nr_entries, int *error) { - uint32_vector *list = user_data; + uint64_vector *list = user_data; size_t i;
if (strcmp (metacontext, LIBNBD_CONTEXT_BASE_ALLOCATION) != 0) @@ -273,7 +273,8 @@ extent_callback (void *user_data, const char *metacontext,
/* Just append the entries we got to the list. */ for (i = 0; i < nr_entries; ++i) { - if (uint32_vector_append (list, entries[i]) == -1) { + if (uint64_vector_append (list, entries[i].length) == -1 || + uint64_vector_append (list, entries[i].flags) == -1) { perror ("realloc"); exit (EXIT_FAILURE); } @@ -284,7 +285,7 @@ extent_callback (void *user_data, const char *metacontext, static bool test_all_zeroes (uint64_t offset, size_t count) { - uint32_vector entries = empty_vector; + uint64_vector entries = empty_vector; size_t i; uint64_t count_read;
@@ -296,22 +297,22 @@ test_all_zeroes (uint64_t offset, size_t count) * false, causing the main code to do a full read. We could be * smarter and keep asking the server (XXX). */ - if (nbd_block_status (nbd, count, offset, - (nbd_extent_callback) { - .callback = extent_callback, - .user_data = &entries }, - 0) == -1) { + if (nbd_block_status_64 (nbd, count, offset, + (nbd_extent64_callback) { + .callback = extent_callback, + .user_data = &entries }, + 0) == -1) { fprintf (stderr, "%s: %s\n", progname, nbd_get_error ()); exit (EXIT_FAILURE); }
count_read = 0; for (i = 0; i < entries.len; i += 2) { - uint32_t len = entries.ptr[i]; - uint32_t type = entries.ptr[i+1]; + uint64_t len = entries.ptr[i]; + uint64_t type = entries.ptr[i+1];
count_read += len; - if (!(type & 2)) /* not zero */ + if (!(type & LIBNBD_STATE_ZERO)) /* not zero */ return false; }
Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Add another bit of overall server information, as well as a '--can extended-headers' silent query. For now, the testsuite is written assuming that when nbdkit finally adds extended headers support, it will also add a --no-eh kill switch comparable to its existing --no-sr switch. Signed-off-by: Eric Blake <eblake@redhat.com> --- info/nbdinfo.pod | 11 ++++++++++- info/can.c | 9 +++++++++ info/info-can.sh | 27 +++++++++++++++++++++++++++ info/info-packets.sh | 17 ++++++++++++++++- info/main.c | 7 ++++++- 5 files changed, 68 insertions(+), 3 deletions(-) diff --git a/info/nbdinfo.pod b/info/nbdinfo.pod index 7eb3c1a0..9ea4a278 100644 --- a/info/nbdinfo.pod +++ b/info/nbdinfo.pod @@ -86,6 +86,7 @@ the I<--json> parameter: "protocol": "newstyle-fixed", "TLS": false, "structured": true, + "extended": false, "exports": [ { "export-name": "", @@ -165,6 +166,11 @@ Test if the NBD URI connection is using TLS. Test if server can respond with structured replies (a prerequisite for supporting block status commands). +=item nbdinfo --can extended-headers URI + +Test if server supports extended headers (a prerequisite for +supporting 64-bit commands; implies structured replies as well). + =item nbdinfo --is rotational URI Test if the server export is backed by something which behaves like a @@ -312,6 +318,8 @@ Display brief command line help and exit. =item B<--can df> +=item B<--can extended-headers> + =item B<--can fast-zero> =item B<--can flush> @@ -341,7 +349,8 @@ and the following libnbd functions: L<nbd_can_cache(3)>, L<nbd_can_df(3)>, L<nbd_can_fast_zero(3)>, L<nbd_can_flush(3)>, L<nbd_can_fua(3)>, L<nbd_can_multi_conn(3)>, L<nbd_can_trim(3)>, L<nbd_can_zero(3)>, L<nbd_is_read_only(3)>, -L<nbd_get_structured_replies_negotiated(3)>. +L<nbd_get_structured_replies_negotiated(3)>, +L<nbd_get_extended_headers_negotiated(3)>. =item B<--color> diff --git a/info/can.c b/info/can.c index 01ab4806..31c4a1ca 100644 --- a/info/can.c +++ b/info/can.c @@ -50,6 +50,15 @@ do_can (void) strcasecmp (can, "structured_replies") == 0) feature = nbd_get_structured_replies_negotiated (nbd); + else if (strcasecmp (can, "eh") == 0 || + strcasecmp (can, "extended header") == 0 || + strcasecmp (can, "extended-header") == 0 || + strcasecmp (can, "extended_header") == 0 || + strcasecmp (can, "extended headers") == 0 || + strcasecmp (can, "extended-headers") == 0 || + strcasecmp (can, "extended_headers") == 0) + feature = nbd_get_extended_headers_negotiated (nbd); + else if (strcasecmp (can, "readonly") == 0 || strcasecmp (can, "read-only") == 0 || strcasecmp (can, "read_only") == 0) diff --git a/info/info-can.sh b/info/info-can.sh index 6cc8cbf4..8154d1ce 100755 --- a/info/info-can.sh +++ b/info/info-can.sh @@ -61,6 +61,33 @@ esac EOF test $st = 2 +# --can extended-headers cannot be positively tested until nbdkit gains +# --no-eh support. Otherwise, it is similar to --can structured-reply. + +no_eh= +if nbdkit --no-eh --help >/dev/null 2>/dev/null; then + no_eh=--no-eh + nbdkit -v -U - sh - \ + --run '$VG nbdinfo --can extended-headers "nbd+unix:///?socket=$unixsocket"' <<'EOF' +case "$1" in + get_size) echo 1024 ;; + pread) ;; + *) exit 2 ;; +esac +EOF +fi + +st=0 +nbdkit -v -U - $no_eh sh - \ + --run '$VG nbdinfo --can extended-headers "nbd+unix:///?socket=$unixsocket"' <<'EOF' || st=$? +case "$1" in + get_size) echo 1024 ;; + pread) ;; + *) exit 2 ;; +esac +EOF +test $st = 2 + # --can cache and --can fua require special handling because in # nbdkit-sh-plugin we must print "native" or "none". Also the can_fua # flag is only sent if the export is writable (hence can_write below). diff --git a/info/info-packets.sh b/info/info-packets.sh index 2460052e..410faef8 100755 --- a/info/info-packets.sh +++ b/info/info-packets.sh @@ -27,12 +27,27 @@ requires nbdkit --no-sr memory --version out=info-packets.out cleanup_fn rm -f $out +# Older nbdkit does not support extended headers; --no-eh is a reliable +# witness of whether nbdkit is new enough. + +no_eh= +if nbdkit --no-eh --help >/dev/null 2>/dev/null; then + no_eh=--no-eh +fi + nbdkit --no-sr -U - memory size=1M \ --run '$VG nbdinfo "nbd+unix:///?socket=$unixsocket"' > $out cat $out grep "protocol: .*using simple packets" $out -nbdkit -U - memory size=1M \ +nbdkit $no_eh -U - memory size=1M \ --run '$VG nbdinfo "nbd+unix:///?socket=$unixsocket"' > $out cat $out grep "protocol: .*using structured packets" $out + +if test x != "x$no_eh"; then + nbdkit -U - memory size=1M \ + --run '$VG nbdinfo "nbd+unix:///?socket=$unixsocket"' > $out + cat $out + grep "protocol: .*using extended packets" $out +fi diff --git a/info/main.c b/info/main.c index 1b99e089..8c923266 100644 --- a/info/main.c +++ b/info/main.c @@ -302,11 +302,13 @@ main (int argc, char *argv[]) const char *protocol; int tls_negotiated; int sr_negotiated; + int eh_negotiated; /* Print per-connection fields. */ protocol = nbd_get_protocol (nbd); tls_negotiated = nbd_get_tls_negotiated (nbd); sr_negotiated = nbd_get_structured_replies_negotiated (nbd); + eh_negotiated = nbd_get_extended_headers_negotiated (nbd); if (!json_output) { if (protocol) { @@ -314,8 +316,9 @@ main (int argc, char *argv[]) fprintf (fp, "protocol: %s", protocol); if (tls_negotiated >= 0) fprintf (fp, " %s TLS", tls_negotiated ? "with" : "without"); - if (sr_negotiated >= 0) + if (eh_negotiated >= 0 && sr_negotiated >= 0) fprintf (fp, ", using %s packets", + eh_negotiated ? "extended" : sr_negotiated ? "structured" : "simple"); fprintf (fp, "\n"); ansi_restore (fp); @@ -333,6 +336,8 @@ main (int argc, char *argv[]) fprintf (fp, "\"TLS\": %s,\n", tls_negotiated ? "true" : "false"); if (sr_negotiated >= 0) fprintf (fp, "\"structured\": %s,\n", sr_negotiated ? "true" : "false"); + if (eh_negotiated >= 0) + fprintf (fp, "\"extended\": %s,\n", eh_negotiated ? "true" : "false"); } if (!list_all) -- 2.40.1
On Thu, May 25, 2023 at 08:01:00AM -0500, Eric Blake wrote:
Add another bit of overall server information, as well as a '--can extended-headers' silent query. For now, the testsuite is written assuming that when nbdkit finally adds extended headers support, it will also add a --no-eh kill switch comparable to its existing --no-sr switch.
Signed-off-by: Eric Blake <eblake@redhat.com> --- info/nbdinfo.pod | 11 ++++++++++- info/can.c | 9 +++++++++ info/info-can.sh | 27 +++++++++++++++++++++++++++ info/info-packets.sh | 17 ++++++++++++++++- info/main.c | 7 ++++++- 5 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/info/nbdinfo.pod b/info/nbdinfo.pod index 7eb3c1a0..9ea4a278 100644 --- a/info/nbdinfo.pod +++ b/info/nbdinfo.pod @@ -86,6 +86,7 @@ the I<--json> parameter: "protocol": "newstyle-fixed", "TLS": false, "structured": true, + "extended": false, "exports": [ { "export-name": "", @@ -165,6 +166,11 @@ Test if the NBD URI connection is using TLS. Test if server can respond with structured replies (a prerequisite for supporting block status commands).
+=item nbdinfo --can extended-headers URI + +Test if server supports extended headers (a prerequisite for +supporting 64-bit commands; implies structured replies as well).
Maybe we should add "--has" and "--have" as yet more synonyms for --is/--can. "--has extended-headers" reads better. It's only tiny extra change to this array: https://gitlab.com/nbdkit/libnbd/-/blob/27d4ea0bd833f17349e7696353c7a9df069a...
=item nbdinfo --is rotational URI
Test if the server export is backed by something which behaves like a @@ -312,6 +318,8 @@ Display brief command line help and exit.
=item B<--can df>
+=item B<--can extended-headers> + =item B<--can fast-zero>
=item B<--can flush> @@ -341,7 +349,8 @@ and the following libnbd functions: L<nbd_can_cache(3)>, L<nbd_can_df(3)>, L<nbd_can_fast_zero(3)>, L<nbd_can_flush(3)>, L<nbd_can_fua(3)>, L<nbd_can_multi_conn(3)>, L<nbd_can_trim(3)>, L<nbd_can_zero(3)>, L<nbd_is_read_only(3)>, -L<nbd_get_structured_replies_negotiated(3)>. +L<nbd_get_structured_replies_negotiated(3)>, +L<nbd_get_extended_headers_negotiated(3)>.
=item B<--color>
diff --git a/info/can.c b/info/can.c index 01ab4806..31c4a1ca 100644 --- a/info/can.c +++ b/info/can.c @@ -50,6 +50,15 @@ do_can (void) strcasecmp (can, "structured_replies") == 0) feature = nbd_get_structured_replies_negotiated (nbd);
+ else if (strcasecmp (can, "eh") == 0 || + strcasecmp (can, "extended header") == 0 || + strcasecmp (can, "extended-header") == 0 || + strcasecmp (can, "extended_header") == 0 || + strcasecmp (can, "extended headers") == 0 || + strcasecmp (can, "extended-headers") == 0 || + strcasecmp (can, "extended_headers") == 0) + feature = nbd_get_extended_headers_negotiated (nbd); + else if (strcasecmp (can, "readonly") == 0 || strcasecmp (can, "read-only") == 0 || strcasecmp (can, "read_only") == 0) diff --git a/info/info-can.sh b/info/info-can.sh index 6cc8cbf4..8154d1ce 100755 --- a/info/info-can.sh +++ b/info/info-can.sh @@ -61,6 +61,33 @@ esac EOF test $st = 2
+# --can extended-headers cannot be positively tested until nbdkit gains +# --no-eh support. Otherwise, it is similar to --can structured-reply. + +no_eh= +if nbdkit --no-eh --help >/dev/null 2>/dev/null; then + no_eh=--no-eh + nbdkit -v -U - sh - \ + --run '$VG nbdinfo --can extended-headers "nbd+unix:///?socket=$unixsocket"' <<'EOF' +case "$1" in + get_size) echo 1024 ;; + pread) ;; + *) exit 2 ;; +esac +EOF +fi + +st=0 +nbdkit -v -U - $no_eh sh - \ + --run '$VG nbdinfo --can extended-headers "nbd+unix:///?socket=$unixsocket"' <<'EOF' || st=$? +case "$1" in + get_size) echo 1024 ;; + pread) ;; + *) exit 2 ;; +esac +EOF +test $st = 2 + # --can cache and --can fua require special handling because in # nbdkit-sh-plugin we must print "native" or "none". Also the can_fua # flag is only sent if the export is writable (hence can_write below). diff --git a/info/info-packets.sh b/info/info-packets.sh index 2460052e..410faef8 100755 --- a/info/info-packets.sh +++ b/info/info-packets.sh @@ -27,12 +27,27 @@ requires nbdkit --no-sr memory --version out=info-packets.out cleanup_fn rm -f $out
+# Older nbdkit does not support extended headers; --no-eh is a reliable +# witness of whether nbdkit is new enough. + +no_eh= +if nbdkit --no-eh --help >/dev/null 2>/dev/null; then + no_eh=--no-eh +fi + nbdkit --no-sr -U - memory size=1M \ --run '$VG nbdinfo "nbd+unix:///?socket=$unixsocket"' > $out cat $out grep "protocol: .*using simple packets" $out
-nbdkit -U - memory size=1M \ +nbdkit $no_eh -U - memory size=1M \ --run '$VG nbdinfo "nbd+unix:///?socket=$unixsocket"' > $out cat $out grep "protocol: .*using structured packets" $out + +if test x != "x$no_eh"; then + nbdkit -U - memory size=1M \ + --run '$VG nbdinfo "nbd+unix:///?socket=$unixsocket"' > $out + cat $out + grep "protocol: .*using extended packets" $out +fi diff --git a/info/main.c b/info/main.c index 1b99e089..8c923266 100644 --- a/info/main.c +++ b/info/main.c @@ -302,11 +302,13 @@ main (int argc, char *argv[]) const char *protocol; int tls_negotiated; int sr_negotiated; + int eh_negotiated;
/* Print per-connection fields. */ protocol = nbd_get_protocol (nbd); tls_negotiated = nbd_get_tls_negotiated (nbd); sr_negotiated = nbd_get_structured_replies_negotiated (nbd); + eh_negotiated = nbd_get_extended_headers_negotiated (nbd);
if (!json_output) { if (protocol) { @@ -314,8 +316,9 @@ main (int argc, char *argv[]) fprintf (fp, "protocol: %s", protocol); if (tls_negotiated >= 0) fprintf (fp, " %s TLS", tls_negotiated ? "with" : "without"); - if (sr_negotiated >= 0) + if (eh_negotiated >= 0 && sr_negotiated >= 0) fprintf (fp, ", using %s packets", + eh_negotiated ? "extended" : sr_negotiated ? "structured" : "simple"); fprintf (fp, "\n"); ansi_restore (fp); @@ -333,6 +336,8 @@ main (int argc, char *argv[]) fprintf (fp, "\"TLS\": %s,\n", tls_negotiated ? "true" : "false"); if (sr_negotiated >= 0) fprintf (fp, "\"structured\": %s,\n", sr_negotiated ? "true" : "false"); + if (eh_negotiated >= 0) + fprintf (fp, "\"extended\": %s,\n", eh_negotiated ? "true" : "false"); }
if (!list_all) -- 2.40.1
Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On Thu, Jun 08, 2023 at 10:24:22AM +0100, Richard W.M. Jones wrote:
On Thu, May 25, 2023 at 08:01:00AM -0500, Eric Blake wrote:
Add another bit of overall server information, as well as a '--can extended-headers' silent query. For now, the testsuite is written assuming that when nbdkit finally adds extended headers support, it will also add a --no-eh kill switch comparable to its existing --no-sr switch.
+=item nbdinfo --can extended-headers URI + +Test if server supports extended headers (a prerequisite for +supporting 64-bit commands; implies structured replies as well).
Maybe we should add "--has" and "--have" as yet more synonyms for --is/--can. "--has extended-headers" reads better. It's only tiny extra change to this array:
https://gitlab.com/nbdkit/libnbd/-/blob/27d4ea0bd833f17349e7696353c7a9df069a...
I like the idea. Folding it into v4 as a separate prereq patch ('--has structured-replies' is another benefit).
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
-- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
Although we usually map "base:allocation" which doesn't require the use of the 64-bit API for flags, this application IS intended to map out other metacontexts that might have 64-bit flags. And when extended headers are in use, we might as well ask for the server to give us extents as large as it wants, rather than breaking things up at 4G boundaries. At the time this patch was written, there are no known servers that actually provide a metacontext with 64-bit flags. However, that is planned for the nbdkit v3 protocol. Signed-off-by: Eric Blake <eblake@redhat.com> --- info/map.c | 65 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 34 insertions(+), 31 deletions(-) diff --git a/info/map.c b/info/map.c index 1169fce4..50b058f2 100644 --- a/info/map.c +++ b/info/map.c @@ -36,13 +36,13 @@ #include "nbdinfo.h" -DEFINE_VECTOR_TYPE (uint32_vector, uint32_t); +DEFINE_VECTOR_TYPE (uint64_vector, uint64_t); -static void print_extents (uint32_vector *entries); -static void print_totals (uint32_vector *entries, int64_t size); +static void print_extents (uint64_vector *entries); +static void print_totals (uint64_vector *entries, int64_t size); static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, - uint32_t *entries, size_t nr_entries, + nbd_extent *entries, size_t nr_entries, int *error); void @@ -50,7 +50,7 @@ do_map (void) { size_t i; int64_t size; - uint32_vector entries = empty_vector; + uint64_vector entries = empty_vector; uint64_t offset, align, max_len; size_t prev_entries_size; @@ -69,14 +69,16 @@ do_map (void) fprintf (stderr, "%s: %s\n", progname, nbd_get_error ()); exit (EXIT_FAILURE); } + if (nbd_get_extended_headers_negotiated (nbd) == 1) + max_len = size; for (offset = 0; offset < size;) { prev_entries_size = entries.len; - if (nbd_block_status (nbd, MIN (size - offset, max_len), offset, - (nbd_extent_callback) { - .callback = extent_callback, - .user_data = &entries }, - 0) == -1) { + if (nbd_block_status_64 (nbd, MIN (size - offset, max_len), offset, + (nbd_extent64_callback) { + .callback = extent_callback, + .user_data = &entries }, + 0) == -1) { fprintf (stderr, "%s: %s\n", progname, nbd_get_error ()); exit (EXIT_FAILURE); } @@ -99,18 +101,18 @@ do_map (void) } /* Callback handling --map. */ -static void print_one_extent (uint64_t offset, uint64_t len, uint32_t type); -static void extent_description (const char *metacontext, uint32_t type, +static void print_one_extent (uint64_t offset, uint64_t len, uint64_t type); +static void extent_description (const char *metacontext, uint64_t type, char **descr, bool *free_descr, const char **fg, const char **bg); static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, - uint32_t *entries, size_t nr_entries, + nbd_extent *entries, size_t nr_entries, int *error) { - uint32_vector *list = user_data; + uint64_vector *list = user_data; size_t i; if (strcmp (metacontext, map) != 0) @@ -120,7 +122,8 @@ extent_callback (void *user_data, const char *metacontext, * print_extents below. */ for (i = 0; i < nr_entries; ++i) { - if (uint32_vector_append (list, entries[i]) == -1) { + if (uint64_vector_append (list, entries[i].length) == -1 || + uint64_vector_append (list, entries[i].flags) == -1) { perror ("realloc"); exit (EXIT_FAILURE); } @@ -129,7 +132,7 @@ extent_callback (void *user_data, const char *metacontext, } static void -print_extents (uint32_vector *entries) +print_extents (uint64_vector *entries) { size_t i, j; uint64_t offset = 0; /* end of last extent printed + 1 */ @@ -138,7 +141,7 @@ print_extents (uint32_vector *entries) if (json_output) fprintf (fp, "[\n"); for (i = 0; i < entries->len; i += 2) { - uint32_t type = entries->ptr[last+1]; + uint64_t type = entries->ptr[last+1]; /* If we're coalescing and the current type is different from the * previous one then we should print everything up to this entry. @@ -157,7 +160,7 @@ print_extents (uint32_vector *entries) /* Print the last extent if there is one. */ if (last != i) { - uint32_t type = entries->ptr[last+1]; + uint64_t type = entries->ptr[last+1]; uint64_t len; for (j = last, len = 0; j < i; j += 2) @@ -169,7 +172,7 @@ print_extents (uint32_vector *entries) } static void -print_one_extent (uint64_t offset, uint64_t len, uint32_t type) +print_one_extent (uint64_t offset, uint64_t len, uint64_t type) { static bool comma = false; char *descr; @@ -185,7 +188,7 @@ print_one_extent (uint64_t offset, uint64_t len, uint32_t type) ansi_colour (bg, fp); fprintf (fp, "%10" PRIu64 " " "%10" PRIu64 " " - "%3" PRIu32, + "%3" PRIu64, offset, len, type); if (descr) fprintf (fp, " %s", descr); @@ -199,7 +202,7 @@ print_one_extent (uint64_t offset, uint64_t len, uint32_t type) fprintf (fp, "{ \"offset\": %" PRIu64 ", " "\"length\": %" PRIu64 ", " - "\"type\": %" PRIu32, + "\"type\": %" PRIu64, offset, len, type); if (descr) { fprintf (fp, ", \"description\": "); @@ -215,9 +218,9 @@ print_one_extent (uint64_t offset, uint64_t len, uint32_t type) /* --map --totals suboption */ static void -print_totals (uint32_vector *entries, int64_t size) +print_totals (uint64_vector *entries, int64_t size) { - uint32_t type; + uint64_t type; bool comma = false; /* This is necessary to avoid a divide by zero below, but if the @@ -237,16 +240,16 @@ print_totals (uint32_vector *entries, int64_t size) */ type = 0; for (;;) { - uint64_t next_type = (uint64_t)UINT32_MAX + 1; + uint64_t next_type = 0; uint64_t c = 0; size_t i; for (i = 0; i < entries->len; i += 2) { - uint32_t t = entries->ptr[i+1]; + uint64_t t = entries->ptr[i+1]; if (t == type) c += entries->ptr[i]; - else if (type < t && t < next_type) + else if (type < t && (next_type == 0 || t < next_type)) next_type = t; } @@ -263,7 +266,7 @@ print_totals (uint32_vector *entries, int64_t size) ansi_colour (fg, fp); if (bg) ansi_colour (bg, fp); - fprintf (fp, "%10" PRIu64 " %5.1f%% %3" PRIu32, + fprintf (fp, "%10" PRIu64 " %5.1f%% %3" PRIu64, c, percent, type); if (descr) fprintf (fp, " %s", descr); @@ -278,7 +281,7 @@ print_totals (uint32_vector *entries, int64_t size) fprintf (fp, "{ \"size\": %" PRIu64 ", " "\"percent\": %g, " - "\"type\": %" PRIu32, + "\"type\": %" PRIu64, c, percent, type); if (descr) { fprintf (fp, ", \"description\": "); @@ -292,7 +295,7 @@ print_totals (uint32_vector *entries, int64_t size) free (descr); } - if (next_type == (uint64_t)UINT32_MAX + 1) + if (next_type == 0) break; type = next_type; } @@ -301,7 +304,7 @@ print_totals (uint32_vector *entries, int64_t size) } static void -extent_description (const char *metacontext, uint32_t type, +extent_description (const char *metacontext, uint64_t type, char **descr, bool *free_descr, const char **fg, const char **bg) { @@ -348,7 +351,7 @@ extent_description (const char *metacontext, uint32_t type, *fg = ANSI_FG_BRIGHT_WHITE; *bg = ANSI_BG_BLACK; return; default: - if (asprintf (descr, "backing depth %u", type) == -1) { + if (asprintf (descr, "backing depth %" PRIu64, type) == -1) { perror ("asprintf"); exit (EXIT_FAILURE); } -- 2.40.1
On Thu, May 25, 2023 at 08:01:01AM -0500, Eric Blake wrote:
Although we usually map "base:allocation" which doesn't require the use of the 64-bit API for flags, this application IS intended to map out other metacontexts that might have 64-bit flags. And when extended headers are in use, we might as well ask for the server to give us extents as large as it wants, rather than breaking things up at 4G boundaries.
At the time this patch was written, there are no known servers that actually provide a metacontext with 64-bit flags. However, that is planned for the nbdkit v3 protocol.
Signed-off-by: Eric Blake <eblake@redhat.com> --- info/map.c | 65 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 34 insertions(+), 31 deletions(-)
diff --git a/info/map.c b/info/map.c index 1169fce4..50b058f2 100644 --- a/info/map.c +++ b/info/map.c @@ -36,13 +36,13 @@
#include "nbdinfo.h"
-DEFINE_VECTOR_TYPE (uint32_vector, uint32_t); +DEFINE_VECTOR_TYPE (uint64_vector, uint64_t);
-static void print_extents (uint32_vector *entries); -static void print_totals (uint32_vector *entries, int64_t size); +static void print_extents (uint64_vector *entries); +static void print_totals (uint64_vector *entries, int64_t size); static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, - uint32_t *entries, size_t nr_entries, + nbd_extent *entries, size_t nr_entries, int *error);
void @@ -50,7 +50,7 @@ do_map (void) { size_t i; int64_t size; - uint32_vector entries = empty_vector; + uint64_vector entries = empty_vector; uint64_t offset, align, max_len; size_t prev_entries_size;
@@ -69,14 +69,16 @@ do_map (void) fprintf (stderr, "%s: %s\n", progname, nbd_get_error ()); exit (EXIT_FAILURE); } + if (nbd_get_extended_headers_negotiated (nbd) == 1) + max_len = size;
for (offset = 0; offset < size;) { prev_entries_size = entries.len; - if (nbd_block_status (nbd, MIN (size - offset, max_len), offset, - (nbd_extent_callback) { - .callback = extent_callback, - .user_data = &entries }, - 0) == -1) { + if (nbd_block_status_64 (nbd, MIN (size - offset, max_len), offset, + (nbd_extent64_callback) { + .callback = extent_callback, + .user_data = &entries }, + 0) == -1) { fprintf (stderr, "%s: %s\n", progname, nbd_get_error ()); exit (EXIT_FAILURE); } @@ -99,18 +101,18 @@ do_map (void) }
/* Callback handling --map. */ -static void print_one_extent (uint64_t offset, uint64_t len, uint32_t type); -static void extent_description (const char *metacontext, uint32_t type, +static void print_one_extent (uint64_t offset, uint64_t len, uint64_t type); +static void extent_description (const char *metacontext, uint64_t type, char **descr, bool *free_descr, const char **fg, const char **bg);
static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, - uint32_t *entries, size_t nr_entries, + nbd_extent *entries, size_t nr_entries, int *error) { - uint32_vector *list = user_data; + uint64_vector *list = user_data; size_t i;
if (strcmp (metacontext, map) != 0) @@ -120,7 +122,8 @@ extent_callback (void *user_data, const char *metacontext, * print_extents below. */ for (i = 0; i < nr_entries; ++i) { - if (uint32_vector_append (list, entries[i]) == -1) { + if (uint64_vector_append (list, entries[i].length) == -1 || + uint64_vector_append (list, entries[i].flags) == -1) { perror ("realloc"); exit (EXIT_FAILURE); } @@ -129,7 +132,7 @@ extent_callback (void *user_data, const char *metacontext, }
static void -print_extents (uint32_vector *entries) +print_extents (uint64_vector *entries) { size_t i, j; uint64_t offset = 0; /* end of last extent printed + 1 */ @@ -138,7 +141,7 @@ print_extents (uint32_vector *entries) if (json_output) fprintf (fp, "[\n");
for (i = 0; i < entries->len; i += 2) { - uint32_t type = entries->ptr[last+1]; + uint64_t type = entries->ptr[last+1];
/* If we're coalescing and the current type is different from the * previous one then we should print everything up to this entry. @@ -157,7 +160,7 @@ print_extents (uint32_vector *entries)
/* Print the last extent if there is one. */ if (last != i) { - uint32_t type = entries->ptr[last+1]; + uint64_t type = entries->ptr[last+1]; uint64_t len;
for (j = last, len = 0; j < i; j += 2) @@ -169,7 +172,7 @@ print_extents (uint32_vector *entries) }
static void -print_one_extent (uint64_t offset, uint64_t len, uint32_t type) +print_one_extent (uint64_t offset, uint64_t len, uint64_t type) { static bool comma = false; char *descr; @@ -185,7 +188,7 @@ print_one_extent (uint64_t offset, uint64_t len, uint32_t type) ansi_colour (bg, fp); fprintf (fp, "%10" PRIu64 " " "%10" PRIu64 " " - "%3" PRIu32, + "%3" PRIu64, offset, len, type); if (descr) fprintf (fp, " %s", descr); @@ -199,7 +202,7 @@ print_one_extent (uint64_t offset, uint64_t len, uint32_t type)
fprintf (fp, "{ \"offset\": %" PRIu64 ", " "\"length\": %" PRIu64 ", " - "\"type\": %" PRIu32, + "\"type\": %" PRIu64, offset, len, type); if (descr) { fprintf (fp, ", \"description\": "); @@ -215,9 +218,9 @@ print_one_extent (uint64_t offset, uint64_t len, uint32_t type)
/* --map --totals suboption */ static void -print_totals (uint32_vector *entries, int64_t size) +print_totals (uint64_vector *entries, int64_t size) { - uint32_t type; + uint64_t type; bool comma = false;
/* This is necessary to avoid a divide by zero below, but if the @@ -237,16 +240,16 @@ print_totals (uint32_vector *entries, int64_t size) */ type = 0; for (;;) { - uint64_t next_type = (uint64_t)UINT32_MAX + 1; + uint64_t next_type = 0; uint64_t c = 0; size_t i;
for (i = 0; i < entries->len; i += 2) { - uint32_t t = entries->ptr[i+1]; + uint64_t t = entries->ptr[i+1];
if (t == type) c += entries->ptr[i]; - else if (type < t && t < next_type) + else if (type < t && (next_type == 0 || t < next_type)) next_type = t; }
@@ -263,7 +266,7 @@ print_totals (uint32_vector *entries, int64_t size) ansi_colour (fg, fp); if (bg) ansi_colour (bg, fp); - fprintf (fp, "%10" PRIu64 " %5.1f%% %3" PRIu32, + fprintf (fp, "%10" PRIu64 " %5.1f%% %3" PRIu64, c, percent, type); if (descr) fprintf (fp, " %s", descr); @@ -278,7 +281,7 @@ print_totals (uint32_vector *entries, int64_t size) fprintf (fp, "{ \"size\": %" PRIu64 ", " "\"percent\": %g, " - "\"type\": %" PRIu32, + "\"type\": %" PRIu64, c, percent, type); if (descr) { fprintf (fp, ", \"description\": "); @@ -292,7 +295,7 @@ print_totals (uint32_vector *entries, int64_t size) free (descr); }
- if (next_type == (uint64_t)UINT32_MAX + 1) + if (next_type == 0) break; type = next_type; } @@ -301,7 +304,7 @@ print_totals (uint32_vector *entries, int64_t size) }
static void -extent_description (const char *metacontext, uint32_t type, +extent_description (const char *metacontext, uint64_t type, char **descr, bool *free_descr, const char **fg, const char **bg) { @@ -348,7 +351,7 @@ extent_description (const char *metacontext, uint32_t type, *fg = ANSI_FG_BRIGHT_WHITE; *bg = ANSI_BG_BLACK; return; default: - if (asprintf (descr, "backing depth %u", type) == -1) { + if (asprintf (descr, "backing depth %" PRIu64, type) == -1) { perror ("asprintf"); exit (EXIT_FAILURE); }
Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
Although our use of "base:allocation" doesn't require the use of the 64-bit API for flags, we might perform slightly faster for a server that does give us 64-bit extent lengths and honors larger nbd_zero lengths. Signed-off-by: Eric Blake <eblake@redhat.com> --- examples/copy-libev.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/examples/copy-libev.c b/examples/copy-libev.c index 32cb46b3..9346faf7 100644 --- a/examples/copy-libev.c +++ b/examples/copy-libev.c @@ -96,7 +96,7 @@ struct request { }; struct extent { - uint32_t length; + uint64_t length; bool zero; }; @@ -184,7 +184,7 @@ get_events (struct connection *c) static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, - uint32_t *entries, size_t nr_entries, int *error) + nbd_extent *entries, size_t nr_entries, int *error) { struct request *r = user_data; @@ -199,22 +199,21 @@ extent_callback (void *user_data, const char *metacontext, uint64_t offset, return 1; } - /* Libnbd returns uint32_t pair (length, flags) for each extent. */ - extents_len = nr_entries / 2; + extents_len = nr_entries; extents = malloc (extents_len * sizeof *extents); if (extents == NULL) FAIL ("Cannot allocated extents: %s", strerror (errno)); /* Copy libnbd entries to extents array. */ - for (int i = 0, j = 0; i < extents_len; i++, j=i*2) { - extents[i].length = entries[j]; + for (int i = 0; i < extents_len; i++) { + extents[i].length = entries[i].length; /* Libnbd exposes both ZERO and HOLE flags. We care only about * ZERO status, meaning we can copy this extent using efficinet * zero method. */ - extents[i].zero = (entries[j + 1] & LIBNBD_STATE_ZERO) != 0; + extents[i].zero = (entries[i].flags & LIBNBD_STATE_ZERO) != 0; } DEBUG ("r%zu: received %zu extents for %s", @@ -286,10 +285,10 @@ start_extents (struct request *r) DEBUG ("r%zu: start extents offset=%" PRIi64 " count=%zu", r->index, offset, count); - cookie = nbd_aio_block_status ( + cookie = nbd_aio_block_status_64 ( src.nbd, count, offset, - (nbd_extent_callback) { .callback=extent_callback, - .user_data=r }, + (nbd_extent64_callback) { .callback=extent_callback, + .user_data=r }, (nbd_completion_callback) { .callback=extents_completed, .user_data=r }, 0); @@ -324,7 +323,7 @@ next_extent (struct request *r) limit = MIN (REQUEST_SIZE, size - offset); while (length < limit) { - DEBUG ("e%zu: offset=%" PRIi64 " len=%" PRIu32 " zero=%d", + DEBUG ("e%zu: offset=%" PRIi64 " len=%" PRIu64 " zero=%d", extents_pos, offset, extents[extents_pos].length, is_zero); /* If this extent is too large, steal some data from it to -- 2.40.1
On Thu, May 25, 2023 at 08:01:02AM -0500, Eric Blake wrote:
Although our use of "base:allocation" doesn't require the use of the 64-bit API for flags, we might perform slightly faster for a server that does give us 64-bit extent lengths and honors larger nbd_zero lengths.
Signed-off-by: Eric Blake <eblake@redhat.com> --- examples/copy-libev.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-)
diff --git a/examples/copy-libev.c b/examples/copy-libev.c index 32cb46b3..9346faf7 100644 --- a/examples/copy-libev.c +++ b/examples/copy-libev.c @@ -96,7 +96,7 @@ struct request { };
struct extent { - uint32_t length; + uint64_t length; bool zero; };
@@ -184,7 +184,7 @@ get_events (struct connection *c)
static int extent_callback (void *user_data, const char *metacontext, uint64_t offset, - uint32_t *entries, size_t nr_entries, int *error) + nbd_extent *entries, size_t nr_entries, int *error) { struct request *r = user_data;
@@ -199,22 +199,21 @@ extent_callback (void *user_data, const char *metacontext, uint64_t offset, return 1; }
- /* Libnbd returns uint32_t pair (length, flags) for each extent. */ - extents_len = nr_entries / 2; + extents_len = nr_entries;
extents = malloc (extents_len * sizeof *extents); if (extents == NULL) FAIL ("Cannot allocated extents: %s", strerror (errno));
/* Copy libnbd entries to extents array. */ - for (int i = 0, j = 0; i < extents_len; i++, j=i*2) { - extents[i].length = entries[j]; + for (int i = 0; i < extents_len; i++) { + extents[i].length = entries[i].length;
/* Libnbd exposes both ZERO and HOLE flags. We care only about * ZERO status, meaning we can copy this extent using efficinet * zero method. */ - extents[i].zero = (entries[j + 1] & LIBNBD_STATE_ZERO) != 0; + extents[i].zero = (entries[i].flags & LIBNBD_STATE_ZERO) != 0; }
DEBUG ("r%zu: received %zu extents for %s", @@ -286,10 +285,10 @@ start_extents (struct request *r) DEBUG ("r%zu: start extents offset=%" PRIi64 " count=%zu", r->index, offset, count);
- cookie = nbd_aio_block_status ( + cookie = nbd_aio_block_status_64 ( src.nbd, count, offset, - (nbd_extent_callback) { .callback=extent_callback, - .user_data=r }, + (nbd_extent64_callback) { .callback=extent_callback, + .user_data=r }, (nbd_completion_callback) { .callback=extents_completed, .user_data=r }, 0); @@ -324,7 +323,7 @@ next_extent (struct request *r) limit = MIN (REQUEST_SIZE, size - offset);
while (length < limit) { - DEBUG ("e%zu: offset=%" PRIi64 " len=%" PRIu32 " zero=%d", + DEBUG ("e%zu: offset=%" PRIi64 " len=%" PRIu64 " zero=%d", extents_pos, offset, extents[extents_pos].length, is_zero);
/* If this extent is too large, steal some data from it to
Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
Since our example program for 32-bit extents is inherently limited to 32-bit lengths, it is also worth demonstrating the 64-bit extent API, including the difference in the array indexing being saner. Signed-off-by: Eric Blake <eblake@redhat.com> --- ocaml/examples/Makefile.am | 1 + ocaml/examples/extents64.ml | 42 +++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 ocaml/examples/extents64.ml diff --git a/ocaml/examples/Makefile.am b/ocaml/examples/Makefile.am index 28b4ab94..a4eb47a5 100644 --- a/ocaml/examples/Makefile.am +++ b/ocaml/examples/Makefile.am @@ -20,6 +20,7 @@ include $(top_srcdir)/subdir-rules.mk ml_examples = \ asynch_copy.ml \ extents.ml \ + extents64.ml \ get_size.ml \ open_qcow2.ml \ server_flags.ml \ diff --git a/ocaml/examples/extents64.ml b/ocaml/examples/extents64.ml new file mode 100644 index 00000000..8ee7e218 --- /dev/null +++ b/ocaml/examples/extents64.ml @@ -0,0 +1,42 @@ +open Printf + +let () = + NBD.with_handle ( + fun nbd -> + NBD.add_meta_context nbd "base:allocation"; + NBD.connect_command nbd + ["nbdkit"; "-s"; "--exit-with-parent"; "-r"; + "sparse-random"; "8G"]; + + (* Read the extents and print them. *) + let size = NBD.get_size nbd in + let cap = + match NBD.get_extended_headers_negotiated nbd with + | true -> size + | false -> 0x8000_0000_L in + let fetch_offset = ref 0_L in + while !fetch_offset < size do + let remaining = Int64.sub size !fetch_offset in + let fetch_size = min remaining cap in + NBD.block_status_64 nbd fetch_size !fetch_offset ( + fun meta _ entries err -> + printf "nbd_block_status callback: meta=%s err=%d\n" meta !err; + if meta = "base:allocation" then ( + printf "index\t%16s %16s %s\n" "offset" "length" "flags"; + for i = 0 to Array.length entries - 1 do + let len = fst entries.(i) + and flags = + match snd entries.(i) with + | 0_L -> "data" + | 1_L -> "hole" + | 2_L -> "zero" + | 3_L -> "hole+zero" + | unknown -> sprintf "unknown (%Ld)" unknown in + printf "%d:\t%16Ld %16Ld %s\n" i !fetch_offset len flags; + fetch_offset := Int64.add !fetch_offset len + done; + ); + 0 + ) (* NBD.block_status *) + done + ) -- 2.40.1
On Thu, May 25, 2023 at 08:01:03AM -0500, Eric Blake wrote:
Since our example program for 32-bit extents is inherently limited to 32-bit lengths, it is also worth demonstrating the 64-bit extent API, including the difference in the array indexing being saner.
Signed-off-by: Eric Blake <eblake@redhat.com> --- ocaml/examples/Makefile.am | 1 + ocaml/examples/extents64.ml | 42 +++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 ocaml/examples/extents64.ml
diff --git a/ocaml/examples/Makefile.am b/ocaml/examples/Makefile.am index 28b4ab94..a4eb47a5 100644 --- a/ocaml/examples/Makefile.am +++ b/ocaml/examples/Makefile.am @@ -20,6 +20,7 @@ include $(top_srcdir)/subdir-rules.mk ml_examples = \ asynch_copy.ml \ extents.ml \ + extents64.ml \ get_size.ml \ open_qcow2.ml \ server_flags.ml \ diff --git a/ocaml/examples/extents64.ml b/ocaml/examples/extents64.ml new file mode 100644 index 00000000..8ee7e218 --- /dev/null +++ b/ocaml/examples/extents64.ml @@ -0,0 +1,42 @@ +open Printf + +let () = + NBD.with_handle ( + fun nbd -> + NBD.add_meta_context nbd "base:allocation"; + NBD.connect_command nbd + ["nbdkit"; "-s"; "--exit-with-parent"; "-r"; + "sparse-random"; "8G"]; + + (* Read the extents and print them. *) + let size = NBD.get_size nbd in + let cap = + match NBD.get_extended_headers_negotiated nbd with + | true -> size + | false -> 0x8000_0000_L in + let fetch_offset = ref 0_L in + while !fetch_offset < size do + let remaining = Int64.sub size !fetch_offset in + let fetch_size = min remaining cap in + NBD.block_status_64 nbd fetch_size !fetch_offset ( + fun meta _ entries err -> + printf "nbd_block_status callback: meta=%s err=%d\n" meta !err; + if meta = "base:allocation" then ( + printf "index\t%16s %16s %s\n" "offset" "length" "flags"; + for i = 0 to Array.length entries - 1 do + let len = fst entries.(i) + and flags = + match snd entries.(i) with + | 0_L -> "data" + | 1_L -> "hole" + | 2_L -> "zero" + | 3_L -> "hole+zero" + | unknown -> sprintf "unknown (%Ld)" unknown in + printf "%d:\t%16Ld %16Ld %s\n" i !fetch_offset len flags; + fetch_offset := Int64.add !fetch_offset len + done; + ); + 0 + ) (* NBD.block_status *) + done + )
Yes, the API is nicer now we return the subelements as a list instead of having to iterate over the list in pairs. I might change that to an array or struct after these patches go upstream as that will be a tiny bit more efficient. Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Thu, Jun 08, 2023 at 10:37:59AM +0100, Richard W.M. Jones wrote:
Yes, the API is nicer now we return the subelements as a list instead of having to iterate over the list in pairs. I might change that to an array or struct after these patches go upstream as that will be a tiny bit more efficient.
Actually the second sentence here is wrong. It's allocated as a pair not a list, and a pair has the same internal representation as a 2-element array. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
This is the culmination of the previous patches' preparation work for using extended headers when possible. The new states in the state machine are copied extensively from our handling of OPT_STRUCTURED_REPLY. The next patch will then expose a new API nbd_opt_extended_headers() for manual control. At the same time I posted this patch, I had patches for qemu-nbd to support extended headers as server (nbdkit is a bit tougher). The next patches will add some interop tests that pass when using a new enough qemu-nbd, showing that we have cross-project interoperability and therefore an extension worth standardizing. Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/API.ml | 87 ++++++++--------- generator/Makefile.am | 1 + generator/state_machine.ml | 41 ++++++++ .../states-newstyle-opt-extended-headers.c | 94 +++++++++++++++++++ generator/states-newstyle-opt-starttls.c | 6 +- 5 files changed, 184 insertions(+), 45 deletions(-) create mode 100644 generator/states-newstyle-opt-extended-headers.c diff --git a/generator/API.ml b/generator/API.ml index 7616990a..078f140f 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -953,23 +953,24 @@ "set_request_meta_context", { (all C<nbd_connect_*> calls when L<nbd_set_opt_mode(3)> is false, or L<nbd_opt_go(3)> and L<nbd_opt_info(3)> when option mode is enabled) will also try to issue NBD_OPT_SET_META_CONTEXT when -the server supports structured replies and any contexts were -registered by L<nbd_add_meta_context(3)>. The default setting -is true; however the extra step of negotiating meta contexts is -not always desirable: performing both info and go on the same -export works without needing to re-negotiate contexts on the -second call; integration testing of other servers may benefit -from manual invocation of L<nbd_opt_set_meta_context(3)> at -other times in the negotiation sequence; and even when using just -L<nbd_opt_info(3)>, it can be faster to collect the server's +the server supports structured replies or extended headers and +any contexts were registered by L<nbd_add_meta_context(3)>. The +default setting is true; however the extra step of negotiating +meta contexts is not always desirable: performing both info and +go on the same export works without needing to re-negotiate +contexts on the second call; integration testing of other servers +may benefit from manual invocation of L<nbd_opt_set_meta_context(3)> +at other times in the negotiation sequence; and even when using +just L<nbd_opt_info(3)>, it can be faster to collect the server's results by relying on the callback function passed to L<nbd_opt_list_meta_context(3)> than a series of post-process calls to L<nbd_can_meta_context(3)>. Note that this control has no effect if the server does not -negotiate structured replies, or if the client did not request -any contexts via L<nbd_add_meta_context(3)>. Setting this -control to false may cause L<nbd_block_status(3)> to fail."; +negotiate structured replies or extended headers, or if the +client did not request any contexts via L<nbd_add_meta_context(3)>. +Setting this control to false may cause L<nbd_block_status(3)> +to fail."; see_also = [Link "set_opt_mode"; Link "opt_go"; Link "opt_info"; Link "opt_list_meta_context"; Link "opt_set_meta_context"; Link "get_structured_replies_negotiated"; @@ -1404,11 +1405,11 @@ "opt_info", { If successful, functions like L<nbd_is_read_only(3)> and L<nbd_get_size(3)> will report details about that export. If L<nbd_set_request_meta_context(3)> is set (the default) and -structured replies were negotiated, it is also valid to use -L<nbd_can_meta_context(3)> after this call. However, it may be -more efficient to clear that setting and manually utilize -L<nbd_opt_list_meta_context(3)> with its callback approach, for -learning which contexts an export supports. In general, if +structured replies or extended headers were negotiated, it is also +valid to use L<nbd_can_meta_context(3)> after this call. However, +it may be more efficient to clear that setting and manually +utilize L<nbd_opt_list_meta_context(3)> with its callback approach, +for learning which contexts an export supports. In general, if L<nbd_opt_go(3)> is called next, that call will likely succeed with the details remaining the same, although this is not guaranteed by all servers. @@ -1538,12 +1539,12 @@ "opt_set_meta_context", { recent L<nbd_set_export_name(3)> or L<nbd_connect_uri(3)>. This can only be used if L<nbd_set_opt_mode(3)> enabled option mode. Normally, this function is redundant, as L<nbd_opt_go(3)> -automatically does the same task if structured replies have -already been negotiated. But manual control over meta context -requests can be useful for fine-grained testing of how a server -handles unusual negotiation sequences. Often, use of this -function is coupled with L<nbd_set_request_meta_context(3)> to -bypass the automatic context request normally performed by +automatically does the same task if structured replies or extended +headers have already been negotiated. But manual control over +meta context requests can be useful for fine-grained testing of +how a server handles unusual negotiation sequences. Often, use +of this function is coupled with L<nbd_set_request_meta_context(3)> +to bypass the automatic context request normally performed by L<nbd_opt_go(3)>. The NBD protocol allows a client to decide how many queries to ask @@ -1597,12 +1598,13 @@ "opt_set_meta_context_queries", { or L<nbd_connect_uri(3)>. This can only be used if L<nbd_set_opt_mode(3)> enabled option mode. Normally, this function is redundant, as L<nbd_opt_go(3)> automatically does -the same task if structured replies have already been -negotiated. But manual control over meta context requests can -be useful for fine-grained testing of how a server handles -unusual negotiation sequences. Often, use of this function is -coupled with L<nbd_set_request_meta_context(3)> to bypass the -automatic context request normally performed by L<nbd_opt_go(3)>. +the same task if structured replies or extended headers have +already been negotiated. But manual control over meta context +requests can be useful for fine-grained testing of how a server +handles unusual negotiation sequences. Often, use of this +function is coupled with L<nbd_set_request_meta_context(3)> to +bypass the automatic context request normally performed by +L<nbd_opt_go(3)>. The NBD protocol allows a client to decide how many queries to ask the server. This function takes an explicit list of queries; to @@ -3281,13 +3283,13 @@ "aio_opt_set_meta_context", { recent L<nbd_set_export_name(3)> or L<nbd_connect_uri(3)>. This can only be used if L<nbd_set_opt_mode(3)> enabled option mode. Normally, this function is redundant, as L<nbd_opt_go(3)> -automatically does the same task if structured replies have -already been negotiated. But manual control over meta context -requests can be useful for fine-grained testing of how a server -handles unusual negotiation sequences. Often, use of this -function is coupled with L<nbd_set_request_meta_context(3)> to -bypass the automatic context request normally performed by -L<nbd_opt_go(3)>. +automatically does the same task if structured replies or +extended headers have already been negotiated. But manual +control over meta context requests can be useful for fine-grained +testing of how a server handles unusual negotiation sequences. +Often, use of this function is coupled with +L<nbd_set_request_meta_context(3)> to bypass the automatic +context request normally performed by L<nbd_opt_go(3)>. To determine when the request completes, wait for L<nbd_aio_is_connecting(3)> to return false. Or supply the optional @@ -3314,12 +3316,13 @@ "aio_opt_set_meta_context_queries", { or L<nbd_connect_uri(3)>. This can only be used if L<nbd_set_opt_mode(3)> enabled option mode. Normally, this function is redundant, as L<nbd_opt_go(3)> automatically does -the same task if structured replies have already been -negotiated. But manual control over meta context requests can -be useful for fine-grained testing of how a server handles -unusual negotiation sequences. Often, use of this function is -coupled with L<nbd_set_request_meta_context(3)> to bypass the -automatic context request normally performed by L<nbd_opt_go(3)>. +the same task if structured replies or extended headers have +already been negotiated. But manual control over meta context +requests can be useful for fine-grained testing of how a server +handles unusual negotiation sequences. Often, use of this +function is coupled with L<nbd_set_request_meta_context(3)> to +bypass the automatic context request normally performed by +L<nbd_opt_go(3)>. To determine when the request completes, wait for L<nbd_aio_is_connecting(3)> to return false. Or supply the optional diff --git a/generator/Makefile.am b/generator/Makefile.am index 91dbde5c..fc79b1b9 100644 --- a/generator/Makefile.am +++ b/generator/Makefile.am @@ -30,6 +30,7 @@ states_code = \ states-issue-command.c \ states-magic.c \ states-newstyle-opt-export-name.c \ + states-newstyle-opt-extended-headers.c \ states-newstyle-opt-list.c \ states-newstyle-opt-go.c \ states-newstyle-opt-meta-context.c \ diff --git a/generator/state_machine.ml b/generator/state_machine.ml index 1f0d00b0..d09ac792 100644 --- a/generator/state_machine.ml +++ b/generator/state_machine.ml @@ -297,6 +297,7 @@ and * NEGOTIATING after OPT_STRUCTURED_REPLY or any failed OPT_GO. *) Group ("OPT_STARTTLS", newstyle_opt_starttls_state_machine); + Group ("OPT_EXTENDED_HEADERS", newstyle_opt_extended_headers_state_machine); Group ("OPT_STRUCTURED_REPLY", newstyle_opt_structured_reply_state_machine); Group ("OPT_META_CONTEXT", newstyle_opt_meta_context_state_machine); Group ("OPT_GO", newstyle_opt_go_state_machine); @@ -441,6 +442,46 @@ and }; ] +(* Fixed newstyle NBD_OPT_EXTENDED_HEADERS option. + * Implementation: generator/states-newstyle-opt-extended-headers.c + *) +and newstyle_opt_extended_headers_state_machine = [ + State { + default_state with + name = "START"; + comment = "Try to negotiate newstyle NBD_OPT_EXTENDED_HEADERS"; + external_events = []; + }; + + State { + default_state with + name = "SEND"; + comment = "Send newstyle NBD_OPT_EXTENDED_HEADERS negotiation request"; + external_events = [ NotifyWrite, "" ]; + }; + + State { + default_state with + name = "RECV_REPLY"; + comment = "Receive newstyle NBD_OPT_EXTENDED_HEADERS option reply"; + external_events = [ NotifyRead, "" ]; + }; + + State { + default_state with + name = "RECV_REPLY_PAYLOAD"; + comment = "Receive any newstyle NBD_OPT_EXTENDED_HEADERS reply payload"; + external_events = [ NotifyRead, "" ]; + }; + + State { + default_state with + name = "CHECK_REPLY"; + comment = "Check newstyle NBD_OPT_EXTENDED_HEADERS option reply"; + external_events = []; + }; +] + (* Fixed newstyle NBD_OPT_STRUCTURED_REPLY option. * Implementation: generator/states-newstyle-opt-structured-reply.c *) diff --git a/generator/states-newstyle-opt-extended-headers.c b/generator/states-newstyle-opt-extended-headers.c new file mode 100644 index 00000000..1ec25e97 --- /dev/null +++ b/generator/states-newstyle-opt-extended-headers.c @@ -0,0 +1,94 @@ +/* nbd client library in userspace: state machine + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* State machine for negotiating NBD_OPT_EXTENDED_HEADERS. */ + +STATE_MACHINE { + NEWSTYLE.OPT_EXTENDED_HEADERS.START: + assert (h->gflags & LIBNBD_HANDSHAKE_FLAG_FIXED_NEWSTYLE); + assert (h->opt_current != NBD_OPT_EXTENDED_HEADERS); + assert (CALLBACK_IS_NULL (h->opt_cb.completion)); + if (!h->request_eh || !h->request_sr) { + SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + return 0; + } + + h->sbuf.option.version = htobe64 (NBD_NEW_VERSION); + h->sbuf.option.option = htobe32 (NBD_OPT_EXTENDED_HEADERS); + h->sbuf.option.optlen = htobe32 (0); + h->chunks_sent++; + h->wbuf = &h->sbuf; + h->wlen = sizeof h->sbuf.option; + SET_NEXT_STATE (%SEND); + return 0; + + NEWSTYLE.OPT_EXTENDED_HEADERS.SEND: + switch (send_from_wbuf (h)) { + case -1: SET_NEXT_STATE (%.DEAD); return 0; + case 0: + h->rbuf = &h->sbuf; + h->rlen = sizeof h->sbuf.or.option_reply; + SET_NEXT_STATE (%RECV_REPLY); + } + return 0; + + NEWSTYLE.OPT_EXTENDED_HEADERS.RECV_REPLY: + switch (recv_into_rbuf (h)) { + case -1: SET_NEXT_STATE (%.DEAD); return 0; + case 0: + if (prepare_for_reply_payload (h, NBD_OPT_EXTENDED_HEADERS) == -1) { + SET_NEXT_STATE (%.DEAD); + return 0; + } + SET_NEXT_STATE (%RECV_REPLY_PAYLOAD); + } + return 0; + + NEWSTYLE.OPT_EXTENDED_HEADERS.RECV_REPLY_PAYLOAD: + switch (recv_into_rbuf (h)) { + case -1: SET_NEXT_STATE (%.DEAD); return 0; + case 0: SET_NEXT_STATE (%CHECK_REPLY); + } + return 0; + + NEWSTYLE.OPT_EXTENDED_HEADERS.CHECK_REPLY: + uint32_t reply; + + reply = be32toh (h->sbuf.or.option_reply.reply); + switch (reply) { + case NBD_REP_ACK: + debug (h, "negotiated extended headers on this connection"); + h->extended_headers = true; + /* Extended headers trump structured replies, so skip ahead. */ + h->structured_replies = true; + break; + default: + if (handle_reply_error (h) == -1) { + SET_NEXT_STATE (%.DEAD); + return 0; + } + + debug (h, "extended headers are not supported by this server"); + break; + } + + /* Next option. */ + SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + return 0; + +} /* END STATE MACHINE */ diff --git a/generator/states-newstyle-opt-starttls.c b/generator/states-newstyle-opt-starttls.c index e497548c..1e2997a3 100644 --- a/generator/states-newstyle-opt-starttls.c +++ b/generator/states-newstyle-opt-starttls.c @@ -26,7 +26,7 @@ NEWSTYLE.OPT_STARTTLS.START: else { /* If TLS was not requested we skip this option and go to the next one. */ if (h->tls == LIBNBD_TLS_DISABLE) { - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + SET_NEXT_STATE (%^OPT_EXTENDED_HEADERS.START); return 0; } assert (CALLBACK_IS_NULL (h->opt_cb.completion)); @@ -128,7 +128,7 @@ NEWSTYLE.OPT_STARTTLS.CHECK_REPLY: SET_NEXT_STATE (%.NEGOTIATING); else { debug (h, "continuing with unencrypted connection"); - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + SET_NEXT_STATE (%^OPT_EXTENDED_HEADERS.START); } return 0; } @@ -185,7 +185,7 @@ NEWSTYLE.OPT_STARTTLS.TLS_HANDSHAKE_DONE: if (h->opt_current == NBD_OPT_STARTTLS) SET_NEXT_STATE (%.NEGOTIATING); else - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + SET_NEXT_STATE (%^OPT_EXTENDED_HEADERS.START); return 0; } /* END STATE MACHINE */ -- 2.40.1
On Thu, May 25, 2023 at 08:01:04AM -0500, Eric Blake wrote:
This is the culmination of the previous patches' preparation work for using extended headers when possible. The new states in the state machine are copied extensively from our handling of OPT_STRUCTURED_REPLY. The next patch will then expose a new API nbd_opt_extended_headers() for manual control.
At the same time I posted this patch, I had patches for qemu-nbd to support extended headers as server (nbdkit is a bit tougher). The next patches will add some interop tests that pass when using a new enough qemu-nbd, showing that we have cross-project interoperability and therefore an extension worth standardizing.
Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/API.ml | 87 ++++++++--------- generator/Makefile.am | 1 + generator/state_machine.ml | 41 ++++++++ .../states-newstyle-opt-extended-headers.c | 94 +++++++++++++++++++ generator/states-newstyle-opt-starttls.c | 6 +- 5 files changed, 184 insertions(+), 45 deletions(-) create mode 100644 generator/states-newstyle-opt-extended-headers.c
diff --git a/generator/API.ml b/generator/API.ml index 7616990a..078f140f 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -953,23 +953,24 @@ "set_request_meta_context", { (all C<nbd_connect_*> calls when L<nbd_set_opt_mode(3)> is false, or L<nbd_opt_go(3)> and L<nbd_opt_info(3)> when option mode is enabled) will also try to issue NBD_OPT_SET_META_CONTEXT when -the server supports structured replies and any contexts were -registered by L<nbd_add_meta_context(3)>. The default setting -is true; however the extra step of negotiating meta contexts is -not always desirable: performing both info and go on the same -export works without needing to re-negotiate contexts on the -second call; integration testing of other servers may benefit -from manual invocation of L<nbd_opt_set_meta_context(3)> at -other times in the negotiation sequence; and even when using just -L<nbd_opt_info(3)>, it can be faster to collect the server's +the server supports structured replies or extended headers and +any contexts were registered by L<nbd_add_meta_context(3)>. The +default setting is true; however the extra step of negotiating +meta contexts is not always desirable: performing both info and +go on the same export works without needing to re-negotiate +contexts on the second call; integration testing of other servers +may benefit from manual invocation of L<nbd_opt_set_meta_context(3)> +at other times in the negotiation sequence; and even when using +just L<nbd_opt_info(3)>, it can be faster to collect the server's results by relying on the callback function passed to L<nbd_opt_list_meta_context(3)> than a series of post-process calls to L<nbd_can_meta_context(3)>.
Note that this control has no effect if the server does not -negotiate structured replies, or if the client did not request -any contexts via L<nbd_add_meta_context(3)>. Setting this -control to false may cause L<nbd_block_status(3)> to fail."; +negotiate structured replies or extended headers, or if the +client did not request any contexts via L<nbd_add_meta_context(3)>. +Setting this control to false may cause L<nbd_block_status(3)> +to fail."; see_also = [Link "set_opt_mode"; Link "opt_go"; Link "opt_info"; Link "opt_list_meta_context"; Link "opt_set_meta_context"; Link "get_structured_replies_negotiated"; @@ -1404,11 +1405,11 @@ "opt_info", { If successful, functions like L<nbd_is_read_only(3)> and L<nbd_get_size(3)> will report details about that export. If L<nbd_set_request_meta_context(3)> is set (the default) and -structured replies were negotiated, it is also valid to use -L<nbd_can_meta_context(3)> after this call. However, it may be -more efficient to clear that setting and manually utilize -L<nbd_opt_list_meta_context(3)> with its callback approach, for -learning which contexts an export supports. In general, if +structured replies or extended headers were negotiated, it is also +valid to use L<nbd_can_meta_context(3)> after this call. However, +it may be more efficient to clear that setting and manually +utilize L<nbd_opt_list_meta_context(3)> with its callback approach, +for learning which contexts an export supports. In general, if L<nbd_opt_go(3)> is called next, that call will likely succeed with the details remaining the same, although this is not guaranteed by all servers. @@ -1538,12 +1539,12 @@ "opt_set_meta_context", { recent L<nbd_set_export_name(3)> or L<nbd_connect_uri(3)>. This can only be used if L<nbd_set_opt_mode(3)> enabled option mode. Normally, this function is redundant, as L<nbd_opt_go(3)> -automatically does the same task if structured replies have -already been negotiated. But manual control over meta context -requests can be useful for fine-grained testing of how a server -handles unusual negotiation sequences. Often, use of this -function is coupled with L<nbd_set_request_meta_context(3)> to -bypass the automatic context request normally performed by +automatically does the same task if structured replies or extended +headers have already been negotiated. But manual control over +meta context requests can be useful for fine-grained testing of +how a server handles unusual negotiation sequences. Often, use +of this function is coupled with L<nbd_set_request_meta_context(3)> +to bypass the automatic context request normally performed by L<nbd_opt_go(3)>.
The NBD protocol allows a client to decide how many queries to ask @@ -1597,12 +1598,13 @@ "opt_set_meta_context_queries", { or L<nbd_connect_uri(3)>. This can only be used if L<nbd_set_opt_mode(3)> enabled option mode. Normally, this function is redundant, as L<nbd_opt_go(3)> automatically does -the same task if structured replies have already been -negotiated. But manual control over meta context requests can -be useful for fine-grained testing of how a server handles -unusual negotiation sequences. Often, use of this function is -coupled with L<nbd_set_request_meta_context(3)> to bypass the -automatic context request normally performed by L<nbd_opt_go(3)>. +the same task if structured replies or extended headers have +already been negotiated. But manual control over meta context +requests can be useful for fine-grained testing of how a server +handles unusual negotiation sequences. Often, use of this +function is coupled with L<nbd_set_request_meta_context(3)> to +bypass the automatic context request normally performed by +L<nbd_opt_go(3)>.
The NBD protocol allows a client to decide how many queries to ask the server. This function takes an explicit list of queries; to @@ -3281,13 +3283,13 @@ "aio_opt_set_meta_context", { recent L<nbd_set_export_name(3)> or L<nbd_connect_uri(3)>. This can only be used if L<nbd_set_opt_mode(3)> enabled option mode. Normally, this function is redundant, as L<nbd_opt_go(3)> -automatically does the same task if structured replies have -already been negotiated. But manual control over meta context -requests can be useful for fine-grained testing of how a server -handles unusual negotiation sequences. Often, use of this -function is coupled with L<nbd_set_request_meta_context(3)> to -bypass the automatic context request normally performed by -L<nbd_opt_go(3)>. +automatically does the same task if structured replies or +extended headers have already been negotiated. But manual +control over meta context requests can be useful for fine-grained +testing of how a server handles unusual negotiation sequences. +Often, use of this function is coupled with +L<nbd_set_request_meta_context(3)> to bypass the automatic +context request normally performed by L<nbd_opt_go(3)>.
To determine when the request completes, wait for L<nbd_aio_is_connecting(3)> to return false. Or supply the optional @@ -3314,12 +3316,13 @@ "aio_opt_set_meta_context_queries", { or L<nbd_connect_uri(3)>. This can only be used if L<nbd_set_opt_mode(3)> enabled option mode. Normally, this function is redundant, as L<nbd_opt_go(3)> automatically does -the same task if structured replies have already been -negotiated. But manual control over meta context requests can -be useful for fine-grained testing of how a server handles -unusual negotiation sequences. Often, use of this function is -coupled with L<nbd_set_request_meta_context(3)> to bypass the -automatic context request normally performed by L<nbd_opt_go(3)>. +the same task if structured replies or extended headers have +already been negotiated. But manual control over meta context +requests can be useful for fine-grained testing of how a server +handles unusual negotiation sequences. Often, use of this +function is coupled with L<nbd_set_request_meta_context(3)> to +bypass the automatic context request normally performed by +L<nbd_opt_go(3)>.
To determine when the request completes, wait for L<nbd_aio_is_connecting(3)> to return false. Or supply the optional diff --git a/generator/Makefile.am b/generator/Makefile.am index 91dbde5c..fc79b1b9 100644 --- a/generator/Makefile.am +++ b/generator/Makefile.am @@ -30,6 +30,7 @@ states_code = \ states-issue-command.c \ states-magic.c \ states-newstyle-opt-export-name.c \ + states-newstyle-opt-extended-headers.c \ states-newstyle-opt-list.c \ states-newstyle-opt-go.c \ states-newstyle-opt-meta-context.c \ diff --git a/generator/state_machine.ml b/generator/state_machine.ml index 1f0d00b0..d09ac792 100644 --- a/generator/state_machine.ml +++ b/generator/state_machine.ml @@ -297,6 +297,7 @@ and * NEGOTIATING after OPT_STRUCTURED_REPLY or any failed OPT_GO. *) Group ("OPT_STARTTLS", newstyle_opt_starttls_state_machine); + Group ("OPT_EXTENDED_HEADERS", newstyle_opt_extended_headers_state_machine); Group ("OPT_STRUCTURED_REPLY", newstyle_opt_structured_reply_state_machine); Group ("OPT_META_CONTEXT", newstyle_opt_meta_context_state_machine); Group ("OPT_GO", newstyle_opt_go_state_machine); @@ -441,6 +442,46 @@ and }; ]
+(* Fixed newstyle NBD_OPT_EXTENDED_HEADERS option. + * Implementation: generator/states-newstyle-opt-extended-headers.c + *) +and newstyle_opt_extended_headers_state_machine = [ + State { + default_state with + name = "START"; + comment = "Try to negotiate newstyle NBD_OPT_EXTENDED_HEADERS"; + external_events = []; + }; + + State { + default_state with + name = "SEND"; + comment = "Send newstyle NBD_OPT_EXTENDED_HEADERS negotiation request"; + external_events = [ NotifyWrite, "" ]; + }; + + State { + default_state with + name = "RECV_REPLY"; + comment = "Receive newstyle NBD_OPT_EXTENDED_HEADERS option reply"; + external_events = [ NotifyRead, "" ]; + }; + + State { + default_state with + name = "RECV_REPLY_PAYLOAD"; + comment = "Receive any newstyle NBD_OPT_EXTENDED_HEADERS reply payload"; + external_events = [ NotifyRead, "" ]; + }; + + State { + default_state with + name = "CHECK_REPLY"; + comment = "Check newstyle NBD_OPT_EXTENDED_HEADERS option reply"; + external_events = []; + }; +] + (* Fixed newstyle NBD_OPT_STRUCTURED_REPLY option. * Implementation: generator/states-newstyle-opt-structured-reply.c *) diff --git a/generator/states-newstyle-opt-extended-headers.c b/generator/states-newstyle-opt-extended-headers.c new file mode 100644 index 00000000..1ec25e97 --- /dev/null +++ b/generator/states-newstyle-opt-extended-headers.c @@ -0,0 +1,94 @@ +/* nbd client library in userspace: state machine + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* State machine for negotiating NBD_OPT_EXTENDED_HEADERS. */ + +STATE_MACHINE { + NEWSTYLE.OPT_EXTENDED_HEADERS.START: + assert (h->gflags & LIBNBD_HANDSHAKE_FLAG_FIXED_NEWSTYLE); + assert (h->opt_current != NBD_OPT_EXTENDED_HEADERS); + assert (CALLBACK_IS_NULL (h->opt_cb.completion)); + if (!h->request_eh || !h->request_sr) { + SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + return 0; + } + + h->sbuf.option.version = htobe64 (NBD_NEW_VERSION); + h->sbuf.option.option = htobe32 (NBD_OPT_EXTENDED_HEADERS); + h->sbuf.option.optlen = htobe32 (0); + h->chunks_sent++; + h->wbuf = &h->sbuf; + h->wlen = sizeof h->sbuf.option; + SET_NEXT_STATE (%SEND); + return 0; + + NEWSTYLE.OPT_EXTENDED_HEADERS.SEND: + switch (send_from_wbuf (h)) { + case -1: SET_NEXT_STATE (%.DEAD); return 0; + case 0: + h->rbuf = &h->sbuf; + h->rlen = sizeof h->sbuf.or.option_reply; + SET_NEXT_STATE (%RECV_REPLY); + } + return 0; + + NEWSTYLE.OPT_EXTENDED_HEADERS.RECV_REPLY: + switch (recv_into_rbuf (h)) { + case -1: SET_NEXT_STATE (%.DEAD); return 0; + case 0: + if (prepare_for_reply_payload (h, NBD_OPT_EXTENDED_HEADERS) == -1) { + SET_NEXT_STATE (%.DEAD); + return 0; + } + SET_NEXT_STATE (%RECV_REPLY_PAYLOAD); + } + return 0; + + NEWSTYLE.OPT_EXTENDED_HEADERS.RECV_REPLY_PAYLOAD: + switch (recv_into_rbuf (h)) { + case -1: SET_NEXT_STATE (%.DEAD); return 0; + case 0: SET_NEXT_STATE (%CHECK_REPLY); + } + return 0; + + NEWSTYLE.OPT_EXTENDED_HEADERS.CHECK_REPLY: + uint32_t reply; + + reply = be32toh (h->sbuf.or.option_reply.reply); + switch (reply) { + case NBD_REP_ACK: + debug (h, "negotiated extended headers on this connection"); + h->extended_headers = true; + /* Extended headers trump structured replies, so skip ahead. */ + h->structured_replies = true; + break; + default: + if (handle_reply_error (h) == -1) { + SET_NEXT_STATE (%.DEAD); + return 0; + } + + debug (h, "extended headers are not supported by this server"); + break; + } + + /* Next option. */ + SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + return 0; + +} /* END STATE MACHINE */ diff --git a/generator/states-newstyle-opt-starttls.c b/generator/states-newstyle-opt-starttls.c index e497548c..1e2997a3 100644 --- a/generator/states-newstyle-opt-starttls.c +++ b/generator/states-newstyle-opt-starttls.c @@ -26,7 +26,7 @@ NEWSTYLE.OPT_STARTTLS.START: else { /* If TLS was not requested we skip this option and go to the next one. */ if (h->tls == LIBNBD_TLS_DISABLE) { - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + SET_NEXT_STATE (%^OPT_EXTENDED_HEADERS.START); return 0; } assert (CALLBACK_IS_NULL (h->opt_cb.completion)); @@ -128,7 +128,7 @@ NEWSTYLE.OPT_STARTTLS.CHECK_REPLY: SET_NEXT_STATE (%.NEGOTIATING); else { debug (h, "continuing with unencrypted connection"); - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + SET_NEXT_STATE (%^OPT_EXTENDED_HEADERS.START); } return 0; } @@ -185,7 +185,7 @@ NEWSTYLE.OPT_STARTTLS.TLS_HANDSHAKE_DONE: if (h->opt_current == NBD_OPT_STARTTLS) SET_NEXT_STATE (%.NEGOTIATING); else - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + SET_NEXT_STATE (%^OPT_EXTENDED_HEADERS.START); return 0;
} /* END STATE MACHINE */
Seems pretty straightforward. We add a new state machine between NEWSTYLE.OPT_STARTTLS and NEWSTYLE.OPT_STRUCTURED_REPLY which sends the new extended headers option (if enabled). Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Very similar to the recent addition of nbd_opt_structured_reply, giving us fine-grained control over an extended headers request. Because nbdkit does not yet support extended headers, testsuite coverage is limited to interop testing with qemu-nbd. It shows that extended headers imply structured replies, regardless of which order the two are manually negotiated in. Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/API.ml | 79 +++++++-- .../states-newstyle-opt-extended-headers.c | 30 +++- generator/states-newstyle.c | 3 + lib/opt.c | 44 +++++ interop/Makefile.am | 6 + interop/opt-extended-headers.c | 153 ++++++++++++++++++ interop/opt-extended-headers.sh | 29 ++++ .gitignore | 1 + 8 files changed, 329 insertions(+), 16 deletions(-) create mode 100644 interop/opt-extended-headers.c create mode 100755 interop/opt-extended-headers.sh diff --git a/generator/API.ml b/generator/API.ml index 078f140f..85625bbd 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -825,6 +825,7 @@ "set_request_extended_headers", { if L<nbd_set_request_structured_replies(3)> is also set to false, since the use of extended headers implies structured replies."; see_also = [Link "get_request_extended_headers"; + Link "opt_extended_headers"; Link "set_handshake_flags"; Link "set_strict_mode"; Link "get_extended_headers_negotiated"; Link "zero"; Link "trim"; Link "cache"; @@ -856,7 +857,9 @@ "get_extended_headers_negotiated", { shortdesc = "see if extended headers are in use"; longdesc = "\ After connecting you may call this to find out if the connection is -using extended headers. +using extended headers. Note that this setting is sticky; this +can return true even after a second L<nbd_opt_extended_headers(3)> +returns false because the server detected a duplicate request. When extended headers are not in use, commands are limited to a 32-bit length, even when the libnbd API uses a 64-bit parameter @@ -938,7 +941,7 @@ "get_structured_replies_negotiated", { attempted."; see_also = [Link "set_request_structured_replies"; Link "get_request_structured_replies"; - Link "opt_structured_reply"; + Link "opt_structured_reply"; Link "opt_extended_headers"; Link "get_protocol"; Link "get_extended_headers_negotiated"]; }; @@ -1211,12 +1214,13 @@ "set_opt_mode", { newstyle server. This setting has no effect when connecting to an oldstyle server. -Note that libnbd defaults to attempting C<NBD_OPT_STARTTLS> and -C<NBD_OPT_STRUCTURED_REPLY> before letting you control remaining -negotiation steps; if you need control over these steps as well, -first set L<nbd_set_tls(3)> to C<LIBNBD_TLS_DISABLE> and -L<nbd_set_request_structured_replies(3)> to false before starting -the connection attempt. +Note that libnbd defaults to attempting C<NBD_OPT_STARTTLS>, +C<NBD_OPT_EXTENDED_HEADERS>, and C<NBD_OPT_STRUCTURED_REPLY> +before letting you control remaining negotiation steps; if you +need control over these steps as well, first set L<nbd_set_tls(3)> +to C<LIBNBD_TLS_DISABLE>, and L<nbd_set_request_extended_headers(3)> +or L<nbd_set_request_structured_replies(3)> to false, before +starting the connection attempt. When option mode is enabled, you have fine-grained control over which options are negotiated, compared to the default of the server @@ -1324,6 +1328,35 @@ "opt_starttls", { Link "supports_tls"] }; + "opt_extended_headers", { + default_call with + args = []; ret = RBool; + permitted_states = [ Negotiating ]; + shortdesc = "request the server to enable extended headers"; + longdesc = "\ +Request that the server use extended headers, by sending +C<NBD_OPT_EXTENDED_HEADERS>. This can only be used if +L<nbd_set_opt_mode(3)> enabled option mode; furthermore, libnbd +defaults to automatically requesting this unless you use +L<nbd_set_request_extended_headers(3)> or +L<nbd_set_request_structured_replies(3)> prior to connecting. +This function is mainly useful for integration testing of corner +cases in server handling. + +This function returns true if the server replies with success, +false if the server replies with an error, and fails only if +the server does not reply (such as for a loss of connection). +Note that some servers fail a second request as redundant; +libnbd assumes that once one request has succeeded, then +extended headers are supported (as visible by +L<nbd_get_extended_headers_negotiated(3)>) regardless if +later calls to this function return false. If this function +returns true, the use of structured replies is implied."; + see_also = [Link "set_opt_mode"; Link "aio_opt_extended_headers"; + Link "opt_go"; Link "set_request_extended_headers"; + Link "set_request_structured_replies"] + }; + "opt_structured_reply", { default_call with args = []; ret = RBool; @@ -1345,7 +1378,9 @@ "opt_structured_reply", { libnbd assumes that once one request has succeeded, then structured replies are supported (as visible by L<nbd_get_structured_replies_negotiated(3)>) regardless if -later calls to this function return false."; +later calls to this function return false. Similarly, a +server may fail this request if extended headers are already +negotiated, since extended headers take priority."; see_also = [Link "set_opt_mode"; Link "aio_opt_structured_reply"; Link "opt_go"; Link "set_request_structured_replies"] }; @@ -3146,6 +3181,30 @@ "aio_opt_starttls", { see_also = [Link "set_opt_mode"; Link "opt_starttls"]; }; + "aio_opt_extended_headers", { + default_call with + args = []; + optargs = [ OClosure completion_closure ]; + ret = RErr; + permitted_states = [ Negotiating ]; + shortdesc = "request the server to enable extended headers"; + longdesc = "\ +Request that the server use extended headers, by sending +C<NBD_OPT_EXTENDED_HEADERS>. This behaves like the synchronous +counterpart L<nbd_opt_extended_headers(3)>, except that it does +not wait for the server's response. + +To determine when the request completes, wait for +L<nbd_aio_is_connecting(3)> to return false. Or supply the optional +C<completion_callback> which will be invoked as described in +L<libnbd(3)/Completion callbacks>, except that it is automatically +retired regardless of return value. Note that detecting whether the +server returns an error (as is done by the return value of the +synchronous counterpart) is only possible with a completion +callback."; + see_also = [Link "set_opt_mode"; Link "opt_extended_headers"]; + }; + "aio_opt_structured_reply", { default_call with args = []; @@ -4122,6 +4181,8 @@ let first_version = "set_request_extended_headers", (1, 18); "get_request_extended_headers", (1, 18); "get_extended_headers_negotiated", (1, 18); + "opt_extended_headers", (1, 18); + "aio_opt_extended_headers", (1, 18); (* These calls are proposed for a future version of libnbd, but * have not been added to any released version so far. diff --git a/generator/states-newstyle-opt-extended-headers.c b/generator/states-newstyle-opt-extended-headers.c index 1ec25e97..5017a629 100644 --- a/generator/states-newstyle-opt-extended-headers.c +++ b/generator/states-newstyle-opt-extended-headers.c @@ -21,11 +21,14 @@ STATE_MACHINE { NEWSTYLE.OPT_EXTENDED_HEADERS.START: assert (h->gflags & LIBNBD_HANDSHAKE_FLAG_FIXED_NEWSTYLE); - assert (h->opt_current != NBD_OPT_EXTENDED_HEADERS); - assert (CALLBACK_IS_NULL (h->opt_cb.completion)); - if (!h->request_eh || !h->request_sr) { - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); - return 0; + if (h->opt_current == NBD_OPT_EXTENDED_HEADERS) + assert (h->opt_mode); + else { + assert (CALLBACK_IS_NULL (h->opt_cb.completion)); + if (!h->request_eh || !h->request_sr) { + SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + return 0; + } } h->sbuf.option.version = htobe64 (NBD_NEW_VERSION); @@ -68,6 +71,7 @@ NEWSTYLE.OPT_EXTENDED_HEADERS.RECV_REPLY_PAYLOAD: NEWSTYLE.OPT_EXTENDED_HEADERS.CHECK_REPLY: uint32_t reply; + int err = ENOTSUP; reply = be32toh (h->sbuf.or.option_reply.reply); switch (reply) { @@ -76,19 +80,31 @@ NEWSTYLE.OPT_EXTENDED_HEADERS.CHECK_REPLY: h->extended_headers = true; /* Extended headers trump structured replies, so skip ahead. */ h->structured_replies = true; + err = 0; break; + case NBD_REP_ERR_INVALID: + err = EINVAL; + /* fallthrough */ default: if (handle_reply_error (h) == -1) { SET_NEXT_STATE (%.DEAD); return 0; } - debug (h, "extended headers are not supported by this server"); + if (h->extended_headers) + debug (h, "extended headers already negotiated"); + else + debug (h, "extended headers are not supported by this server"); break; } /* Next option. */ - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + if (h->opt_current == NBD_OPT_EXTENDED_HEADERS) + SET_NEXT_STATE (%.NEGOTIATING); + else + SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + CALL_CALLBACK (h->opt_cb.completion, &err); + nbd_internal_free_option (h); return 0; } /* END STATE MACHINE */ diff --git a/generator/states-newstyle.c b/generator/states-newstyle.c index ad5bbf72..45893a8b 100644 --- a/generator/states-newstyle.c +++ b/generator/states-newstyle.c @@ -146,6 +146,9 @@ NEWSTYLE.START: case NBD_OPT_STRUCTURED_REPLY: SET_NEXT_STATE (%OPT_STRUCTURED_REPLY.START); return 0; + case NBD_OPT_EXTENDED_HEADERS: + SET_NEXT_STATE (%OPT_EXTENDED_HEADERS.START); + return 0; case NBD_OPT_STARTTLS: SET_NEXT_STATE (%OPT_STARTTLS.START); return 0; diff --git a/lib/opt.c b/lib/opt.c index f58d5e19..d48acdd1 100644 --- a/lib/opt.c +++ b/lib/opt.c @@ -164,6 +164,31 @@ nbd_unlocked_opt_starttls (struct nbd_handle *h) return r; } +/* Issue NBD_OPT_EXTENDED_HEADERS and wait for the reply. */ +int +nbd_unlocked_opt_extended_headers (struct nbd_handle *h) +{ + int err; + nbd_completion_callback c = { .callback = go_complete, .user_data = &err }; + int r = nbd_unlocked_aio_opt_extended_headers (h, &c); + + if (r == -1) + return r; + + r = wait_for_option (h); + if (r == 0) { + if (nbd_internal_is_state_negotiating (get_next_state (h))) + r = err == 0; + else { + assert (nbd_internal_is_state_dead (get_next_state (h))); + set_error (err, + "failed to get response to opt_extended_headers request"); + r = -1; + } + } + return r; +} + /* Issue NBD_OPT_STRUCTURED_REPLY and wait for the reply. */ int nbd_unlocked_opt_structured_reply (struct nbd_handle *h) @@ -386,6 +411,25 @@ nbd_unlocked_aio_opt_starttls (struct nbd_handle *h, #endif } +/* Issue NBD_OPT_EXTENDED_HEADERS without waiting. */ +int +nbd_unlocked_aio_opt_extended_headers (struct nbd_handle *h, + nbd_completion_callback *complete) +{ + if ((h->gflags & LIBNBD_HANDSHAKE_FLAG_FIXED_NEWSTYLE) == 0) { + set_error (ENOTSUP, "server is not using fixed newstyle protocol"); + return -1; + } + + h->opt_current = NBD_OPT_EXTENDED_HEADERS; + h->opt_cb.completion = *complete; + SET_CALLBACK_TO_NULL (*complete); + + if (nbd_internal_run (h, cmd_issue) == -1) + debug (h, "option queued, ignoring state machine failure"); + return 0; +} + /* Issue NBD_OPT_STRUCTURED_REPLY without waiting. */ int nbd_unlocked_aio_opt_structured_reply (struct nbd_handle *h, diff --git a/interop/Makefile.am b/interop/Makefile.am index ec8ea0b2..3f81df0c 100644 --- a/interop/Makefile.am +++ b/interop/Makefile.am @@ -25,6 +25,7 @@ EXTRA_DIST = \ list-exports-test-dir/disk1 \ list-exports-test-dir/disk2 \ structured-read.sh \ + opt-extended-headers.sh \ $(NULL) TESTS_ENVIRONMENT = \ @@ -134,6 +135,7 @@ check_PROGRAMS += \ socket-activation-qemu-nbd \ dirty-bitmap \ structured-read \ + opt-extended-headers \ $(NULL) TESTS += \ interop-qemu-nbd \ @@ -144,6 +146,7 @@ TESTS += \ dirty-bitmap.sh \ structured-read.sh \ interop-qemu-block-size.sh \ + opt-extended-headers.sh \ $(NULL) interop_qemu_nbd_SOURCES = \ @@ -235,6 +238,9 @@ dirty_bitmap_LDADD = $(top_builddir)/lib/libnbd.la structured_read_SOURCES = structured-read.c structured_read_LDADD = $(top_builddir)/lib/libnbd.la +opt_extended_headers_SOURCES = opt-extended-headers.c +opt_extended_headers_LDADD = $(top_builddir)/lib/libnbd.la + endif HAVE_QEMU_NBD #---------------------------------------------------------------------- diff --git a/interop/opt-extended-headers.c b/interop/opt-extended-headers.c new file mode 100644 index 00000000..f50cd78f --- /dev/null +++ b/interop/opt-extended-headers.c @@ -0,0 +1,153 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Demonstrate low-level use of nbd_opt_extended_headers(). */ + +#include <config.h> + +#include <inttypes.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <unistd.h> +#include <sys/stat.h> + +#include <libnbd.h> + +#define check(got, exp) do_check (#got, got, exp) + +static void +do_check (const char *act, int64_t got, int64_t exp) +{ + fprintf (stderr, "trying %s\n", act); + if (got == -1) + fprintf (stderr, "%s\n", nbd_get_error ()); + else + fprintf (stderr, "succeeded, result %" PRId64 "\n", got); + if (got != exp) { + fprintf (stderr, "got %" PRId64 ", but expected %" PRId64 "\n", got, exp); + exit (EXIT_FAILURE); + } +} + +static int +cb (void *data, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t nr_entries, int *error) +{ + /* If we got here, extents worked, implying at least structured replies */ + bool *seen = data; + + *seen = true; + return 0; +} + +struct nbd_handle * +prep (bool sr, bool eh, char **cmd) +{ + struct nbd_handle *nbd; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Connect to the server in opt mode, disable client-side failsafes so + * that we are testing server response even when client breaks protocol. + */ + check (nbd_set_opt_mode (nbd, true), 0); + check (nbd_set_strict_mode (nbd, 0), 0); + check (nbd_add_meta_context (nbd, LIBNBD_CONTEXT_BASE_ALLOCATION), 0); + check (nbd_set_request_structured_replies (nbd, sr), 0); + check (nbd_set_request_extended_headers (nbd, eh), 0); + check (nbd_connect_systemd_socket_activation (nbd, cmd), 0); + + return nbd; +} + +void +cleanup (struct nbd_handle *nbd, bool extents_exp) +{ + bool extents = false; + + check (nbd_opt_go (nbd), 0); + check (nbd_can_meta_context (nbd, LIBNBD_CONTEXT_BASE_ALLOCATION), + extents_exp); + check (nbd_block_status_64 (nbd, 512, 0, + (nbd_extent64_callback) { .callback = cb, + .user_data = &extents }, + 0), extents_exp ? 0 : -1); + check (extents, extents_exp); + nbd_close (nbd); +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + int64_t bytes_sent; + + if (argc < 2) { + fprintf (stderr, "%s qemu-nbd [args ...]\n", argv[0]); + exit (EXIT_FAILURE); + } + + /* Default setup tries eh first, and skips sr request when eh works... */ + nbd = prep (true, true, &argv[1]); + bytes_sent = nbd_stats_bytes_sent (nbd); + check (nbd_get_extended_headers_negotiated (nbd), true); + check (nbd_get_structured_replies_negotiated (nbd), true); + /* Duplicate eh request is no-op as redundant, but does not change state */ + check (nbd_opt_extended_headers (nbd), false); + /* Trying sr after eh is no-op as redundant, but does not change state */ + check (nbd_opt_structured_reply (nbd), false); + check (nbd_get_extended_headers_negotiated (nbd), true); + check (nbd_get_structured_replies_negotiated (nbd), true); + cleanup (nbd, true); + + /* ...which should result in the same amount of initial negotiation + * traffic as explicitly requesting just structured replies, albeit + * with different results on what got negotiated. + */ + nbd = prep (true, false, &argv[1]); + check (nbd_stats_bytes_sent (nbd), bytes_sent); + check (nbd_get_extended_headers_negotiated (nbd), false); + check (nbd_get_structured_replies_negotiated (nbd), true); + cleanup (nbd, true); + + /* request_eh is ignored if request_sr is false. */ + nbd = prep (false, true, &argv[1]); + check (nbd_get_extended_headers_negotiated (nbd), false); + check (nbd_get_structured_replies_negotiated (nbd), false); + cleanup (nbd, false); + + /* Swap order, requesting structured replies before extended headers */ + nbd = prep (false, false, &argv[1]); + check (nbd_get_extended_headers_negotiated (nbd), false); + check (nbd_get_structured_replies_negotiated (nbd), false); + check (nbd_opt_structured_reply (nbd), true); + check (nbd_get_extended_headers_negotiated (nbd), false); + check (nbd_get_structured_replies_negotiated (nbd), true); + check (nbd_opt_extended_headers (nbd), true); + check (nbd_get_extended_headers_negotiated (nbd), true); + check (nbd_get_structured_replies_negotiated (nbd), true); + cleanup (nbd, true); + + exit (EXIT_SUCCESS); +} diff --git a/interop/opt-extended-headers.sh b/interop/opt-extended-headers.sh new file mode 100755 index 00000000..41322f36 --- /dev/null +++ b/interop/opt-extended-headers.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright Red Hat +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +# Test low-level nbd_opt_extended_headers() details with qemu-nbd + +source ../tests/functions.sh +set -e +set -x + +requires qemu-nbd --version +requires nbdinfo --can extended-headers -- [ qemu-nbd -r -f raw "$0" ] + +# Run the test. +$VG ./opt-extended-headers qemu-nbd -r -f raw "$0" diff --git a/.gitignore b/.gitignore index bc7c2c37..24642748 100644 --- a/.gitignore +++ b/.gitignore @@ -118,6 +118,7 @@ Makefile.in /interop/list-exports-nbdkit /interop/list-exports-qemu-nbd /interop/nbd-server-tls.conf +/interop/opt-extended-headers /interop/requires.c /interop/socket-activation-nbdkit /interop/socket-activation-qemu-nbd -- 2.40.1
On Thu, May 25, 2023 at 08:01:05AM -0500, Eric Blake wrote:
Very similar to the recent addition of nbd_opt_structured_reply, giving us fine-grained control over an extended headers request.
Because nbdkit does not yet support extended headers, testsuite coverage is limited to interop testing with qemu-nbd. It shows that extended headers imply structured replies, regardless of which order the two are manually negotiated in.
Signed-off-by: Eric Blake <eblake@redhat.com> --- generator/API.ml | 79 +++++++-- .../states-newstyle-opt-extended-headers.c | 30 +++- generator/states-newstyle.c | 3 + lib/opt.c | 44 +++++ interop/Makefile.am | 6 + interop/opt-extended-headers.c | 153 ++++++++++++++++++ interop/opt-extended-headers.sh | 29 ++++ .gitignore | 1 + 8 files changed, 329 insertions(+), 16 deletions(-) create mode 100644 interop/opt-extended-headers.c create mode 100755 interop/opt-extended-headers.sh
diff --git a/generator/API.ml b/generator/API.ml index 078f140f..85625bbd 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -825,6 +825,7 @@ "set_request_extended_headers", { if L<nbd_set_request_structured_replies(3)> is also set to false, since the use of extended headers implies structured replies."; see_also = [Link "get_request_extended_headers"; + Link "opt_extended_headers"; Link "set_handshake_flags"; Link "set_strict_mode"; Link "get_extended_headers_negotiated"; Link "zero"; Link "trim"; Link "cache"; @@ -856,7 +857,9 @@ "get_extended_headers_negotiated", { shortdesc = "see if extended headers are in use"; longdesc = "\ After connecting you may call this to find out if the connection is -using extended headers. +using extended headers. Note that this setting is sticky; this +can return true even after a second L<nbd_opt_extended_headers(3)> +returns false because the server detected a duplicate request.
When extended headers are not in use, commands are limited to a 32-bit length, even when the libnbd API uses a 64-bit parameter @@ -938,7 +941,7 @@ "get_structured_replies_negotiated", { attempted."; see_also = [Link "set_request_structured_replies"; Link "get_request_structured_replies"; - Link "opt_structured_reply"; + Link "opt_structured_reply"; Link "opt_extended_headers"; Link "get_protocol"; Link "get_extended_headers_negotiated"]; }; @@ -1211,12 +1214,13 @@ "set_opt_mode", { newstyle server. This setting has no effect when connecting to an oldstyle server.
-Note that libnbd defaults to attempting C<NBD_OPT_STARTTLS> and -C<NBD_OPT_STRUCTURED_REPLY> before letting you control remaining -negotiation steps; if you need control over these steps as well, -first set L<nbd_set_tls(3)> to C<LIBNBD_TLS_DISABLE> and -L<nbd_set_request_structured_replies(3)> to false before starting -the connection attempt. +Note that libnbd defaults to attempting C<NBD_OPT_STARTTLS>, +C<NBD_OPT_EXTENDED_HEADERS>, and C<NBD_OPT_STRUCTURED_REPLY> +before letting you control remaining negotiation steps; if you +need control over these steps as well, first set L<nbd_set_tls(3)> +to C<LIBNBD_TLS_DISABLE>, and L<nbd_set_request_extended_headers(3)> +or L<nbd_set_request_structured_replies(3)> to false, before +starting the connection attempt.
When option mode is enabled, you have fine-grained control over which options are negotiated, compared to the default of the server @@ -1324,6 +1328,35 @@ "opt_starttls", { Link "supports_tls"] };
+ "opt_extended_headers", { + default_call with + args = []; ret = RBool; + permitted_states = [ Negotiating ]; + shortdesc = "request the server to enable extended headers"; + longdesc = "\ +Request that the server use extended headers, by sending +C<NBD_OPT_EXTENDED_HEADERS>. This can only be used if +L<nbd_set_opt_mode(3)> enabled option mode; furthermore, libnbd +defaults to automatically requesting this unless you use +L<nbd_set_request_extended_headers(3)> or +L<nbd_set_request_structured_replies(3)> prior to connecting. +This function is mainly useful for integration testing of corner +cases in server handling. + +This function returns true if the server replies with success, +false if the server replies with an error, and fails only if +the server does not reply (such as for a loss of connection). +Note that some servers fail a second request as redundant; +libnbd assumes that once one request has succeeded, then +extended headers are supported (as visible by +L<nbd_get_extended_headers_negotiated(3)>) regardless if +later calls to this function return false. If this function +returns true, the use of structured replies is implied."; + see_also = [Link "set_opt_mode"; Link "aio_opt_extended_headers"; + Link "opt_go"; Link "set_request_extended_headers"; + Link "set_request_structured_replies"] + }; + "opt_structured_reply", { default_call with args = []; ret = RBool; @@ -1345,7 +1378,9 @@ "opt_structured_reply", { libnbd assumes that once one request has succeeded, then structured replies are supported (as visible by L<nbd_get_structured_replies_negotiated(3)>) regardless if -later calls to this function return false."; +later calls to this function return false. Similarly, a +server may fail this request if extended headers are already +negotiated, since extended headers take priority."; see_also = [Link "set_opt_mode"; Link "aio_opt_structured_reply"; Link "opt_go"; Link "set_request_structured_replies"] }; @@ -3146,6 +3181,30 @@ "aio_opt_starttls", { see_also = [Link "set_opt_mode"; Link "opt_starttls"]; };
+ "aio_opt_extended_headers", { + default_call with + args = []; + optargs = [ OClosure completion_closure ]; + ret = RErr; + permitted_states = [ Negotiating ]; + shortdesc = "request the server to enable extended headers"; + longdesc = "\ +Request that the server use extended headers, by sending +C<NBD_OPT_EXTENDED_HEADERS>. This behaves like the synchronous +counterpart L<nbd_opt_extended_headers(3)>, except that it does +not wait for the server's response. + +To determine when the request completes, wait for +L<nbd_aio_is_connecting(3)> to return false. Or supply the optional +C<completion_callback> which will be invoked as described in +L<libnbd(3)/Completion callbacks>, except that it is automatically +retired regardless of return value. Note that detecting whether the +server returns an error (as is done by the return value of the +synchronous counterpart) is only possible with a completion +callback."; + see_also = [Link "set_opt_mode"; Link "opt_extended_headers"]; + }; + "aio_opt_structured_reply", { default_call with args = []; @@ -4122,6 +4181,8 @@ let first_version = "set_request_extended_headers", (1, 18); "get_request_extended_headers", (1, 18); "get_extended_headers_negotiated", (1, 18); + "opt_extended_headers", (1, 18); + "aio_opt_extended_headers", (1, 18);
(* These calls are proposed for a future version of libnbd, but * have not been added to any released version so far. diff --git a/generator/states-newstyle-opt-extended-headers.c b/generator/states-newstyle-opt-extended-headers.c index 1ec25e97..5017a629 100644 --- a/generator/states-newstyle-opt-extended-headers.c +++ b/generator/states-newstyle-opt-extended-headers.c @@ -21,11 +21,14 @@ STATE_MACHINE { NEWSTYLE.OPT_EXTENDED_HEADERS.START: assert (h->gflags & LIBNBD_HANDSHAKE_FLAG_FIXED_NEWSTYLE); - assert (h->opt_current != NBD_OPT_EXTENDED_HEADERS); - assert (CALLBACK_IS_NULL (h->opt_cb.completion)); - if (!h->request_eh || !h->request_sr) { - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); - return 0; + if (h->opt_current == NBD_OPT_EXTENDED_HEADERS) + assert (h->opt_mode); + else { + assert (CALLBACK_IS_NULL (h->opt_cb.completion)); + if (!h->request_eh || !h->request_sr) { + SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + return 0; + } }
h->sbuf.option.version = htobe64 (NBD_NEW_VERSION); @@ -68,6 +71,7 @@ NEWSTYLE.OPT_EXTENDED_HEADERS.RECV_REPLY_PAYLOAD:
NEWSTYLE.OPT_EXTENDED_HEADERS.CHECK_REPLY: uint32_t reply; + int err = ENOTSUP;
reply = be32toh (h->sbuf.or.option_reply.reply); switch (reply) { @@ -76,19 +80,31 @@ NEWSTYLE.OPT_EXTENDED_HEADERS.CHECK_REPLY: h->extended_headers = true; /* Extended headers trump structured replies, so skip ahead. */ h->structured_replies = true; + err = 0; break; + case NBD_REP_ERR_INVALID: + err = EINVAL; + /* fallthrough */ default: if (handle_reply_error (h) == -1) { SET_NEXT_STATE (%.DEAD); return 0; }
- debug (h, "extended headers are not supported by this server"); + if (h->extended_headers) + debug (h, "extended headers already negotiated"); + else + debug (h, "extended headers are not supported by this server"); break; }
/* Next option. */ - SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + if (h->opt_current == NBD_OPT_EXTENDED_HEADERS) + SET_NEXT_STATE (%.NEGOTIATING); + else + SET_NEXT_STATE (%^OPT_STRUCTURED_REPLY.START); + CALL_CALLBACK (h->opt_cb.completion, &err); + nbd_internal_free_option (h); return 0;
} /* END STATE MACHINE */ diff --git a/generator/states-newstyle.c b/generator/states-newstyle.c index ad5bbf72..45893a8b 100644 --- a/generator/states-newstyle.c +++ b/generator/states-newstyle.c @@ -146,6 +146,9 @@ NEWSTYLE.START: case NBD_OPT_STRUCTURED_REPLY: SET_NEXT_STATE (%OPT_STRUCTURED_REPLY.START); return 0; + case NBD_OPT_EXTENDED_HEADERS: + SET_NEXT_STATE (%OPT_EXTENDED_HEADERS.START); + return 0; case NBD_OPT_STARTTLS: SET_NEXT_STATE (%OPT_STARTTLS.START); return 0; diff --git a/lib/opt.c b/lib/opt.c index f58d5e19..d48acdd1 100644 --- a/lib/opt.c +++ b/lib/opt.c @@ -164,6 +164,31 @@ nbd_unlocked_opt_starttls (struct nbd_handle *h) return r; }
+/* Issue NBD_OPT_EXTENDED_HEADERS and wait for the reply. */ +int +nbd_unlocked_opt_extended_headers (struct nbd_handle *h) +{ + int err; + nbd_completion_callback c = { .callback = go_complete, .user_data = &err }; + int r = nbd_unlocked_aio_opt_extended_headers (h, &c); + + if (r == -1) + return r; + + r = wait_for_option (h); + if (r == 0) { + if (nbd_internal_is_state_negotiating (get_next_state (h))) + r = err == 0; + else { + assert (nbd_internal_is_state_dead (get_next_state (h))); + set_error (err, + "failed to get response to opt_extended_headers request"); + r = -1; + } + } + return r; +} + /* Issue NBD_OPT_STRUCTURED_REPLY and wait for the reply. */ int nbd_unlocked_opt_structured_reply (struct nbd_handle *h) @@ -386,6 +411,25 @@ nbd_unlocked_aio_opt_starttls (struct nbd_handle *h, #endif }
+/* Issue NBD_OPT_EXTENDED_HEADERS without waiting. */ +int +nbd_unlocked_aio_opt_extended_headers (struct nbd_handle *h, + nbd_completion_callback *complete) +{ + if ((h->gflags & LIBNBD_HANDSHAKE_FLAG_FIXED_NEWSTYLE) == 0) { + set_error (ENOTSUP, "server is not using fixed newstyle protocol"); + return -1; + } + + h->opt_current = NBD_OPT_EXTENDED_HEADERS; + h->opt_cb.completion = *complete; + SET_CALLBACK_TO_NULL (*complete); + + if (nbd_internal_run (h, cmd_issue) == -1) + debug (h, "option queued, ignoring state machine failure"); + return 0; +} + /* Issue NBD_OPT_STRUCTURED_REPLY without waiting. */ int nbd_unlocked_aio_opt_structured_reply (struct nbd_handle *h, diff --git a/interop/Makefile.am b/interop/Makefile.am index ec8ea0b2..3f81df0c 100644 --- a/interop/Makefile.am +++ b/interop/Makefile.am @@ -25,6 +25,7 @@ EXTRA_DIST = \ list-exports-test-dir/disk1 \ list-exports-test-dir/disk2 \ structured-read.sh \ + opt-extended-headers.sh \ $(NULL)
TESTS_ENVIRONMENT = \ @@ -134,6 +135,7 @@ check_PROGRAMS += \ socket-activation-qemu-nbd \ dirty-bitmap \ structured-read \ + opt-extended-headers \ $(NULL) TESTS += \ interop-qemu-nbd \ @@ -144,6 +146,7 @@ TESTS += \ dirty-bitmap.sh \ structured-read.sh \ interop-qemu-block-size.sh \ + opt-extended-headers.sh \ $(NULL)
interop_qemu_nbd_SOURCES = \ @@ -235,6 +238,9 @@ dirty_bitmap_LDADD = $(top_builddir)/lib/libnbd.la structured_read_SOURCES = structured-read.c structured_read_LDADD = $(top_builddir)/lib/libnbd.la
+opt_extended_headers_SOURCES = opt-extended-headers.c +opt_extended_headers_LDADD = $(top_builddir)/lib/libnbd.la + endif HAVE_QEMU_NBD
#---------------------------------------------------------------------- diff --git a/interop/opt-extended-headers.c b/interop/opt-extended-headers.c new file mode 100644 index 00000000..f50cd78f --- /dev/null +++ b/interop/opt-extended-headers.c @@ -0,0 +1,153 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Demonstrate low-level use of nbd_opt_extended_headers(). */ + +#include <config.h> + +#include <inttypes.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <unistd.h> +#include <sys/stat.h> + +#include <libnbd.h> + +#define check(got, exp) do_check (#got, got, exp) + +static void +do_check (const char *act, int64_t got, int64_t exp) +{ + fprintf (stderr, "trying %s\n", act); + if (got == -1) + fprintf (stderr, "%s\n", nbd_get_error ()); + else + fprintf (stderr, "succeeded, result %" PRId64 "\n", got); + if (got != exp) { + fprintf (stderr, "got %" PRId64 ", but expected %" PRId64 "\n", got, exp); + exit (EXIT_FAILURE); + } +} + +static int +cb (void *data, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t nr_entries, int *error) +{ + /* If we got here, extents worked, implying at least structured replies */ + bool *seen = data; + + *seen = true; + return 0; +} + +struct nbd_handle * +prep (bool sr, bool eh, char **cmd) +{ + struct nbd_handle *nbd; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Connect to the server in opt mode, disable client-side failsafes so + * that we are testing server response even when client breaks protocol. + */ + check (nbd_set_opt_mode (nbd, true), 0); + check (nbd_set_strict_mode (nbd, 0), 0); + check (nbd_add_meta_context (nbd, LIBNBD_CONTEXT_BASE_ALLOCATION), 0); + check (nbd_set_request_structured_replies (nbd, sr), 0); + check (nbd_set_request_extended_headers (nbd, eh), 0); + check (nbd_connect_systemd_socket_activation (nbd, cmd), 0); + + return nbd; +} + +void +cleanup (struct nbd_handle *nbd, bool extents_exp) +{ + bool extents = false; + + check (nbd_opt_go (nbd), 0); + check (nbd_can_meta_context (nbd, LIBNBD_CONTEXT_BASE_ALLOCATION), + extents_exp); + check (nbd_block_status_64 (nbd, 512, 0, + (nbd_extent64_callback) { .callback = cb, + .user_data = &extents }, + 0), extents_exp ? 0 : -1); + check (extents, extents_exp); + nbd_close (nbd); +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + int64_t bytes_sent; + + if (argc < 2) { + fprintf (stderr, "%s qemu-nbd [args ...]\n", argv[0]); + exit (EXIT_FAILURE); + } + + /* Default setup tries eh first, and skips sr request when eh works... */ + nbd = prep (true, true, &argv[1]); + bytes_sent = nbd_stats_bytes_sent (nbd); + check (nbd_get_extended_headers_negotiated (nbd), true); + check (nbd_get_structured_replies_negotiated (nbd), true); + /* Duplicate eh request is no-op as redundant, but does not change state */ + check (nbd_opt_extended_headers (nbd), false); + /* Trying sr after eh is no-op as redundant, but does not change state */ + check (nbd_opt_structured_reply (nbd), false); + check (nbd_get_extended_headers_negotiated (nbd), true); + check (nbd_get_structured_replies_negotiated (nbd), true); + cleanup (nbd, true); + + /* ...which should result in the same amount of initial negotiation + * traffic as explicitly requesting just structured replies, albeit + * with different results on what got negotiated. + */ + nbd = prep (true, false, &argv[1]); + check (nbd_stats_bytes_sent (nbd), bytes_sent); + check (nbd_get_extended_headers_negotiated (nbd), false); + check (nbd_get_structured_replies_negotiated (nbd), true); + cleanup (nbd, true); + + /* request_eh is ignored if request_sr is false. */ + nbd = prep (false, true, &argv[1]); + check (nbd_get_extended_headers_negotiated (nbd), false); + check (nbd_get_structured_replies_negotiated (nbd), false); + cleanup (nbd, false); + + /* Swap order, requesting structured replies before extended headers */ + nbd = prep (false, false, &argv[1]); + check (nbd_get_extended_headers_negotiated (nbd), false); + check (nbd_get_structured_replies_negotiated (nbd), false); + check (nbd_opt_structured_reply (nbd), true); + check (nbd_get_extended_headers_negotiated (nbd), false); + check (nbd_get_structured_replies_negotiated (nbd), true); + check (nbd_opt_extended_headers (nbd), true); + check (nbd_get_extended_headers_negotiated (nbd), true); + check (nbd_get_structured_replies_negotiated (nbd), true); + cleanup (nbd, true); + + exit (EXIT_SUCCESS); +} diff --git a/interop/opt-extended-headers.sh b/interop/opt-extended-headers.sh new file mode 100755 index 00000000..41322f36 --- /dev/null +++ b/interop/opt-extended-headers.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright Red Hat +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +# Test low-level nbd_opt_extended_headers() details with qemu-nbd + +source ../tests/functions.sh +set -e +set -x + +requires qemu-nbd --version +requires nbdinfo --can extended-headers -- [ qemu-nbd -r -f raw "$0" ] + +# Run the test. +$VG ./opt-extended-headers qemu-nbd -r -f raw "$0" diff --git a/.gitignore b/.gitignore index bc7c2c37..24642748 100644 --- a/.gitignore +++ b/.gitignore @@ -118,6 +118,7 @@ Makefile.in /interop/list-exports-nbdkit /interop/list-exports-qemu-nbd /interop/nbd-server-tls.conf +/interop/opt-extended-headers /interop/requires.c /interop/socket-activation-nbdkit /interop/socket-activation-qemu-nbd
Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Prove that we can round-trip a block status request larger than 4G through a new-enough qemu-nbd. Also serves as a unit test of our shim for converting internal 64-bit representation back to the older 32-bit nbd_block_status callback interface. Signed-off-by: Eric Blake <eblake@redhat.com> --- interop/Makefile.am | 6 ++ interop/large-status.c | 186 ++++++++++++++++++++++++++++++++++++++++ interop/large-status.sh | 49 +++++++++++ .gitignore | 1 + 4 files changed, 242 insertions(+) create mode 100644 interop/large-status.c create mode 100755 interop/large-status.sh diff --git a/interop/Makefile.am b/interop/Makefile.am index 3f81df0c..9a7a5967 100644 --- a/interop/Makefile.am +++ b/interop/Makefile.am @@ -21,6 +21,7 @@ EXTRA_DIST = \ dirty-bitmap.sh \ interop-qemu-storage-daemon.sh \ interop-qemu-block-size.sh \ + large-status.sh \ list-exports-nbd-config \ list-exports-test-dir/disk1 \ list-exports-test-dir/disk2 \ @@ -134,6 +135,7 @@ check_PROGRAMS += \ list-exports-qemu-nbd \ socket-activation-qemu-nbd \ dirty-bitmap \ + large-status \ structured-read \ opt-extended-headers \ $(NULL) @@ -144,6 +146,7 @@ TESTS += \ list-exports-qemu-nbd \ socket-activation-qemu-nbd \ dirty-bitmap.sh \ + large-status.sh \ structured-read.sh \ interop-qemu-block-size.sh \ opt-extended-headers.sh \ @@ -235,6 +238,9 @@ socket_activation_qemu_nbd_LDADD = $(top_builddir)/lib/libnbd.la dirty_bitmap_SOURCES = dirty-bitmap.c dirty_bitmap_LDADD = $(top_builddir)/lib/libnbd.la +large_status_SOURCES = large-status.c +large_status_LDADD = $(top_builddir)/lib/libnbd.la + structured_read_SOURCES = structured-read.c structured_read_LDADD = $(top_builddir)/lib/libnbd.la diff --git a/interop/large-status.c b/interop/large-status.c new file mode 100644 index 00000000..36415653 --- /dev/null +++ b/interop/large-status.c @@ -0,0 +1,186 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Test 64-bit block status with qemu. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <assert.h> +#include <stdbool.h> +#include <errno.h> + +#include <libnbd.h> + +static const char *bitmap; + +struct data { + bool req_one; /* input: true if req_one was passed to request */ + int count; /* input: count of expected remaining calls */ + bool seen_base; /* output: true if base:allocation encountered */ + bool seen_dirty; /* output: true if qemu:dirty-bitmap encountered */ +}; + +static int +cb32 (void *opaque, const char *metacontext, uint64_t offset, + uint32_t *entries, size_t len, int *error) +{ + struct data *data = opaque; + + assert (offset == 0); + assert (data->count-- > 0); + + if (strcmp (metacontext, LIBNBD_CONTEXT_BASE_ALLOCATION) == 0) { + assert (!data->seen_base); + data->seen_base = true; + + /* Data block offset 0 size 64k, remainder is hole */ + assert (len == 4); + assert (entries[0] == 65536); + assert (entries[1] == 0); + /* libnbd had to truncate qemu's >4G answer */ + assert (entries[2] == 4227858432); + assert (entries[3] == (LIBNBD_STATE_HOLE|LIBNBD_STATE_ZERO)); + } + else if (strcmp (metacontext, bitmap) == 0) { + assert (!data->seen_dirty); + data->seen_dirty = true; + + /* Dirty block at offset 5G-64k, remainder is clean */ + /* libnbd had to truncate qemu's >4G answer */ + assert (len == 2); + assert (entries[0] == 4227858432); + assert (entries[1] == 0); + } + else { + fprintf (stderr, "unexpected context %s\n", metacontext); + exit (EXIT_FAILURE); + } + return 0; +} + +static int +cb64 (void *opaque, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t len, int *error) +{ + struct data *data = opaque; + + assert (offset == 0); + assert (data->count-- > 0); + + if (strcmp (metacontext, LIBNBD_CONTEXT_BASE_ALLOCATION) == 0) { + assert (!data->seen_base); + data->seen_base = true; + + /* Data block offset 0 size 64k, remainder is hole */ + assert (len == 2); + assert (entries[0].length == 65536); + assert (entries[0].flags == 0); + assert (entries[1].length == 5368643584ULL); + assert (entries[1].flags == (LIBNBD_STATE_HOLE|LIBNBD_STATE_ZERO)); + } + else if (strcmp (metacontext, bitmap) == 0) { + assert (!data->seen_dirty); + data->seen_dirty = true; + + /* Dirty block at offset 5G-64k, remainder is clean */ + assert (len == 2); + assert (entries[0].length == 5368643584ULL); + assert (entries[0].flags == 0); + assert (entries[1].length == 65536); + assert (entries[1].flags == 1); + } + else { + fprintf (stderr, "unexpected context %s\n", metacontext); + exit (EXIT_FAILURE); + } + return 0; +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + int64_t exportsize; + struct data data; + + if (argc < 3) { + fprintf (stderr, "%s bitmap qemu-nbd [args ...]\n", argv[0]); + exit (EXIT_FAILURE); + } + bitmap = argv[1]; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_add_meta_context (nbd, LIBNBD_CONTEXT_BASE_ALLOCATION); + nbd_add_meta_context (nbd, bitmap); + + if (nbd_connect_systemd_socket_activation (nbd, &argv[2]) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + exportsize = nbd_get_size (nbd); + if (exportsize == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + if (nbd_get_extended_headers_negotiated (nbd) != 1) { + fprintf (stderr, "skipping: qemu-nbd lacks extended headers\n"); + exit (77); + } + + /* Prove that we can round-trip a >4G block status request */ + data = (struct data) { .count = 2, }; + if (nbd_block_status_64 (nbd, exportsize, 0, + (nbd_extent64_callback) { .callback = cb64, + .user_data = &data }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (data.seen_base && data.seen_dirty); + + /* Check libnbd's handling of a >4G response through older interface */ + data = (struct data) { .count = 2, }; + if (nbd_block_status (nbd, exportsize, 0, + (nbd_extent_callback) { .callback = cb32, + .user_data = &data }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (data.seen_base && data.seen_dirty); + + if (nbd_shutdown (nbd, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + + exit (EXIT_SUCCESS); +} diff --git a/interop/large-status.sh b/interop/large-status.sh new file mode 100755 index 00000000..46810dc3 --- /dev/null +++ b/interop/large-status.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright Red Hat +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +# Test reading qemu dirty-bitmap. + +source ../tests/functions.sh +set -e +set -x + +requires qemu-img bitmap --help +requires qemu-nbd --version + +# This test uses the qemu-nbd -B option. +if ! qemu-nbd --help | grep -sq -- -B; then + echo "$0: skipping because qemu-nbd does not support the -B option" + exit 77 +fi + +files="large-status.qcow2" +rm -f $files +cleanup_fn rm -f $files + +# Create mostly-sparse file with intentionally different data vs. dirty areas +# (64k data, 5G-64k hole,zero; 5G-64k clean, 64k dirty) +qemu-img create -f qcow2 large-status.qcow2 5G +qemu-img bitmap --add --enable -f qcow2 large-status.qcow2 bitmap0 +qemu-io -f qcow2 -c "w -z $((5*1024*1024*1024 - 64*1024)) 64k" \ + large-status.qcow2 +qemu-img bitmap --disable -f qcow2 large-status.qcow2 bitmap0 +qemu-io -f qcow2 -c 'w 0 64k' large-status.qcow2 + +# Run the test. +$VG ./large-status qemu:dirty-bitmap:bitmap0 \ + qemu-nbd -f qcow2 -B bitmap0 large-status.qcow2 diff --git a/.gitignore b/.gitignore index 24642748..fd81357b 100644 --- a/.gitignore +++ b/.gitignore @@ -114,6 +114,7 @@ Makefile.in /interop/interop-qemu-nbd /interop/interop-qemu-nbd-tls-certs /interop/interop-qemu-nbd-tls-psk +/interop/large-status /interop/list-exports-nbd-server /interop/list-exports-nbdkit /interop/list-exports-qemu-nbd -- 2.40.1
On Thu, May 25, 2023 at 08:01:06AM -0500, Eric Blake wrote:
Prove that we can round-trip a block status request larger than 4G through a new-enough qemu-nbd. Also serves as a unit test of our shim for converting internal 64-bit representation back to the older 32-bit nbd_block_status callback interface.
I think it would be best to call this test "large-block-status.{c,sh}" as "large-status" is ambiguous. (Or even "block-status-64"?) The test itself is fine, so if renamed: Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com> --- interop/Makefile.am | 6 ++ interop/large-status.c | 186 ++++++++++++++++++++++++++++++++++++++++ interop/large-status.sh | 49 +++++++++++ .gitignore | 1 + 4 files changed, 242 insertions(+) create mode 100644 interop/large-status.c create mode 100755 interop/large-status.sh
diff --git a/interop/Makefile.am b/interop/Makefile.am index 3f81df0c..9a7a5967 100644 --- a/interop/Makefile.am +++ b/interop/Makefile.am @@ -21,6 +21,7 @@ EXTRA_DIST = \ dirty-bitmap.sh \ interop-qemu-storage-daemon.sh \ interop-qemu-block-size.sh \ + large-status.sh \ list-exports-nbd-config \ list-exports-test-dir/disk1 \ list-exports-test-dir/disk2 \ @@ -134,6 +135,7 @@ check_PROGRAMS += \ list-exports-qemu-nbd \ socket-activation-qemu-nbd \ dirty-bitmap \ + large-status \ structured-read \ opt-extended-headers \ $(NULL) @@ -144,6 +146,7 @@ TESTS += \ list-exports-qemu-nbd \ socket-activation-qemu-nbd \ dirty-bitmap.sh \ + large-status.sh \ structured-read.sh \ interop-qemu-block-size.sh \ opt-extended-headers.sh \ @@ -235,6 +238,9 @@ socket_activation_qemu_nbd_LDADD = $(top_builddir)/lib/libnbd.la dirty_bitmap_SOURCES = dirty-bitmap.c dirty_bitmap_LDADD = $(top_builddir)/lib/libnbd.la
+large_status_SOURCES = large-status.c +large_status_LDADD = $(top_builddir)/lib/libnbd.la + structured_read_SOURCES = structured-read.c structured_read_LDADD = $(top_builddir)/lib/libnbd.la
diff --git a/interop/large-status.c b/interop/large-status.c new file mode 100644 index 00000000..36415653 --- /dev/null +++ b/interop/large-status.c @@ -0,0 +1,186 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Test 64-bit block status with qemu. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <assert.h> +#include <stdbool.h> +#include <errno.h> + +#include <libnbd.h> + +static const char *bitmap; + +struct data { + bool req_one; /* input: true if req_one was passed to request */ + int count; /* input: count of expected remaining calls */ + bool seen_base; /* output: true if base:allocation encountered */ + bool seen_dirty; /* output: true if qemu:dirty-bitmap encountered */ +}; + +static int +cb32 (void *opaque, const char *metacontext, uint64_t offset, + uint32_t *entries, size_t len, int *error) +{ + struct data *data = opaque; + + assert (offset == 0); + assert (data->count-- > 0); + + if (strcmp (metacontext, LIBNBD_CONTEXT_BASE_ALLOCATION) == 0) { + assert (!data->seen_base); + data->seen_base = true; + + /* Data block offset 0 size 64k, remainder is hole */ + assert (len == 4); + assert (entries[0] == 65536); + assert (entries[1] == 0); + /* libnbd had to truncate qemu's >4G answer */ + assert (entries[2] == 4227858432); + assert (entries[3] == (LIBNBD_STATE_HOLE|LIBNBD_STATE_ZERO)); + } + else if (strcmp (metacontext, bitmap) == 0) { + assert (!data->seen_dirty); + data->seen_dirty = true; + + /* Dirty block at offset 5G-64k, remainder is clean */ + /* libnbd had to truncate qemu's >4G answer */ + assert (len == 2); + assert (entries[0] == 4227858432); + assert (entries[1] == 0); + } + else { + fprintf (stderr, "unexpected context %s\n", metacontext); + exit (EXIT_FAILURE); + } + return 0; +} + +static int +cb64 (void *opaque, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t len, int *error) +{ + struct data *data = opaque; + + assert (offset == 0); + assert (data->count-- > 0); + + if (strcmp (metacontext, LIBNBD_CONTEXT_BASE_ALLOCATION) == 0) { + assert (!data->seen_base); + data->seen_base = true; + + /* Data block offset 0 size 64k, remainder is hole */ + assert (len == 2); + assert (entries[0].length == 65536); + assert (entries[0].flags == 0); + assert (entries[1].length == 5368643584ULL); + assert (entries[1].flags == (LIBNBD_STATE_HOLE|LIBNBD_STATE_ZERO)); + } + else if (strcmp (metacontext, bitmap) == 0) { + assert (!data->seen_dirty); + data->seen_dirty = true; + + /* Dirty block at offset 5G-64k, remainder is clean */ + assert (len == 2); + assert (entries[0].length == 5368643584ULL); + assert (entries[0].flags == 0); + assert (entries[1].length == 65536); + assert (entries[1].flags == 1); + } + else { + fprintf (stderr, "unexpected context %s\n", metacontext); + exit (EXIT_FAILURE); + } + return 0; +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + int64_t exportsize; + struct data data; + + if (argc < 3) { + fprintf (stderr, "%s bitmap qemu-nbd [args ...]\n", argv[0]); + exit (EXIT_FAILURE); + } + bitmap = argv[1]; + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_add_meta_context (nbd, LIBNBD_CONTEXT_BASE_ALLOCATION); + nbd_add_meta_context (nbd, bitmap); + + if (nbd_connect_systemd_socket_activation (nbd, &argv[2]) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + exportsize = nbd_get_size (nbd); + if (exportsize == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + if (nbd_get_extended_headers_negotiated (nbd) != 1) { + fprintf (stderr, "skipping: qemu-nbd lacks extended headers\n"); + exit (77); + } + + /* Prove that we can round-trip a >4G block status request */ + data = (struct data) { .count = 2, }; + if (nbd_block_status_64 (nbd, exportsize, 0, + (nbd_extent64_callback) { .callback = cb64, + .user_data = &data }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (data.seen_base && data.seen_dirty); + + /* Check libnbd's handling of a >4G response through older interface */ + data = (struct data) { .count = 2, }; + if (nbd_block_status (nbd, exportsize, 0, + (nbd_extent_callback) { .callback = cb32, + .user_data = &data }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (data.seen_base && data.seen_dirty); + + if (nbd_shutdown (nbd, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + + exit (EXIT_SUCCESS); +} diff --git a/interop/large-status.sh b/interop/large-status.sh new file mode 100755 index 00000000..46810dc3 --- /dev/null +++ b/interop/large-status.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright Red Hat +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +# Test reading qemu dirty-bitmap. + +source ../tests/functions.sh +set -e +set -x + +requires qemu-img bitmap --help +requires qemu-nbd --version + +# This test uses the qemu-nbd -B option. +if ! qemu-nbd --help | grep -sq -- -B; then + echo "$0: skipping because qemu-nbd does not support the -B option" + exit 77 +fi + +files="large-status.qcow2" +rm -f $files +cleanup_fn rm -f $files + +# Create mostly-sparse file with intentionally different data vs. dirty areas +# (64k data, 5G-64k hole,zero; 5G-64k clean, 64k dirty) +qemu-img create -f qcow2 large-status.qcow2 5G +qemu-img bitmap --add --enable -f qcow2 large-status.qcow2 bitmap0 +qemu-io -f qcow2 -c "w -z $((5*1024*1024*1024 - 64*1024)) 64k" \ + large-status.qcow2 +qemu-img bitmap --disable -f qcow2 large-status.qcow2 bitmap0 +qemu-io -f qcow2 -c 'w 0 64k' large-status.qcow2 + +# Run the test. +$VG ./large-status qemu:dirty-bitmap:bitmap0 \ + qemu-nbd -f qcow2 -B bitmap0 large-status.qcow2 diff --git a/.gitignore b/.gitignore index 24642748..fd81357b 100644 --- a/.gitignore +++ b/.gitignore @@ -114,6 +114,7 @@ Makefile.in /interop/interop-qemu-nbd /interop/interop-qemu-nbd-tls-certs /interop/interop-qemu-nbd-tls-psk +/interop/large-status /interop/list-exports-nbd-server /interop/list-exports-nbdkit /interop/list-exports-qemu-nbd
Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
In the recent NBD protocol extensions to add 64-bit commands [1], an additional option was added to allow NBD_CMD_BLOCK_STATUS pass a client payload instructing the server to filter its answers in nbd.git commit e6f3b94a (mainly useful when the client requests more than one meta context with NBD_OPT_SET_META_CONTEXT). This patch lays the groundwork by exposing servers that advertise this capability, although libnbd does not yet actually utilize it until the next patch. At the time this patch was written, qemu-nbd was also patched to provide such support; hence, an interop/ test shows the API in action. [1] https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-header/doc/ Signed-off-by: Eric Blake <eblake@redhat.com> --- info/nbdinfo.pod | 10 ++- lib/nbd-protocol.h | 29 +++++--- generator/API.ml | 18 +++++ lib/flags.c | 12 +++ examples/server-flags.c | 7 +- interop/Makefile.am | 6 ++ interop/block-status-payload.c | 126 ++++++++++++++++++++++++++++++++ interop/block-status-payload.sh | 68 +++++++++++++++++ .gitignore | 1 + info/can.c | 5 ++ info/show.c | 9 ++- 11 files changed, 274 insertions(+), 17 deletions(-) create mode 100644 interop/block-status-payload.c create mode 100755 interop/block-status-payload.sh diff --git a/info/nbdinfo.pod b/info/nbdinfo.pod index 9ea4a278..f5dc53fa 100644 --- a/info/nbdinfo.pod +++ b/info/nbdinfo.pod @@ -178,6 +178,8 @@ rotating disk: accessing nearby blocks may be faster than random access and requests should be sorted to improve performance. Many servers do not or cannot report this accurately. +=item nbdinfo --can block-status-payload URI + =item nbdinfo --can cache URI =item nbdinfo --can df URI @@ -345,10 +347,10 @@ The command does not print anything. Instead it exits with success For further information see the L<NBD protocol|https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md> -and the following libnbd functions: L<nbd_can_cache(3)>, -L<nbd_can_df(3)>, L<nbd_can_fast_zero(3)>, L<nbd_can_flush(3)>, -L<nbd_can_fua(3)>, L<nbd_can_multi_conn(3)>, L<nbd_can_trim(3)>, -L<nbd_can_zero(3)>, L<nbd_is_read_only(3)>, +and the following libnbd functions: L<nbd_can_block_status_payload(3)>, +L<nbd_can_cache(3)>, L<nbd_can_df(3)>, L<nbd_can_fast_zero(3)>, +L<nbd_can_flush(3)>, L<nbd_can_fua(3)>, L<nbd_can_multi_conn(3)>, +L<nbd_can_trim(3)>, L<nbd_can_zero(3)>, L<nbd_is_read_only(3)>, L<nbd_get_structured_replies_negotiated(3)>, L<nbd_get_extended_headers_negotiated(3)>. diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h index b6fa9b8a..9e358122 100644 --- a/lib/nbd-protocol.h +++ b/lib/nbd-protocol.h @@ -102,17 +102,18 @@ struct nbd_fixed_new_option_reply { #define NBD_FLAG_NO_ZEROES (1 << 1) /* Per-export flags. */ -#define NBD_FLAG_HAS_FLAGS (1 << 0) -#define NBD_FLAG_READ_ONLY (1 << 1) -#define NBD_FLAG_SEND_FLUSH (1 << 2) -#define NBD_FLAG_SEND_FUA (1 << 3) -#define NBD_FLAG_ROTATIONAL (1 << 4) -#define NBD_FLAG_SEND_TRIM (1 << 5) -#define NBD_FLAG_SEND_WRITE_ZEROES (1 << 6) -#define NBD_FLAG_SEND_DF (1 << 7) -#define NBD_FLAG_CAN_MULTI_CONN (1 << 8) -#define NBD_FLAG_SEND_CACHE (1 << 10) -#define NBD_FLAG_SEND_FAST_ZERO (1 << 11) +#define NBD_FLAG_HAS_FLAGS (1 << 0) +#define NBD_FLAG_READ_ONLY (1 << 1) +#define NBD_FLAG_SEND_FLUSH (1 << 2) +#define NBD_FLAG_SEND_FUA (1 << 3) +#define NBD_FLAG_ROTATIONAL (1 << 4) +#define NBD_FLAG_SEND_TRIM (1 << 5) +#define NBD_FLAG_SEND_WRITE_ZEROES (1 << 6) +#define NBD_FLAG_SEND_DF (1 << 7) +#define NBD_FLAG_CAN_MULTI_CONN (1 << 8) +#define NBD_FLAG_SEND_CACHE (1 << 10) +#define NBD_FLAG_SEND_FAST_ZERO (1 << 11) +#define NBD_FLAG_BLOCK_STATUS_PAYLOAD (1 << 12) /* NBD options (new style handshake only). */ #define NBD_OPT_EXPORT_NAME 1 @@ -204,6 +205,12 @@ struct nbd_request_ext { uint64_t count; /* Request effect or payload length. */ } NBD_ATTRIBUTE_PACKED; +/* Extended request payload for NBD_CMD_BLOCK_STATUS, when supported. */ +struct nbd_block_status_payload { + uint64_t length; /* Effective length of client request */ + /* followed by array of uint32_t ids */ +} NBD_ATTRIBUTE_PACKED; + /* Simple reply (server -> client). */ struct nbd_simple_reply { uint32_t magic; /* NBD_SIMPLE_REPLY_MAGIC. */ diff --git a/generator/API.ml b/generator/API.ml index 85625bbd..5a31ce3b 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -2327,6 +2327,23 @@ "can_fast_zero", { example = Some "examples/server-flags.c"; }; + "can_block_status_payload", { + default_call with + args = []; ret = RBool; + permitted_states = [ Negotiating; Connected; Closed ]; + shortdesc = "does the server support the block status payload flag?"; + longdesc = "\ +Returns true if the server supports the use of the +C<LIBNBD_CMD_FLAG_PAYLOAD_LEN> flag to allow filtering of the +block status command. Returns +false if the server does not. Note that this will never return +true if L<nbd_get_extended_headers_negotiated(3)> is false." +^ non_blocking_test_call_description; + see_also = [SectionLink "Flag calls"; Link "opt_info"; + Link "get_extended_headers_negotiated"]; + example = Some "examples/server-flags.c"; + }; + "can_df", { default_call with args = []; ret = RBool; @@ -4183,6 +4200,7 @@ let first_version = "get_extended_headers_negotiated", (1, 18); "opt_extended_headers", (1, 18); "aio_opt_extended_headers", (1, 18); + "can_block_status_payload", (1, 18); (* These calls are proposed for a future version of libnbd, but * have not been added to any released version so far. diff --git a/lib/flags.c b/lib/flags.c index be880acf..7e6ddedd 100644 --- a/lib/flags.c +++ b/lib/flags.c @@ -66,6 +66,12 @@ nbd_internal_set_size_and_flags (struct nbd_handle *h, eflags &= ~NBD_FLAG_SEND_DF; } + if (eflags & NBD_FLAG_BLOCK_STATUS_PAYLOAD && !h->extended_headers) { + debug (h, "server lacks extended headers, ignoring claim " + "of block status payload"); + eflags &= ~NBD_FLAG_BLOCK_STATUS_PAYLOAD; + } + if (eflags & NBD_FLAG_SEND_FAST_ZERO && !(eflags & NBD_FLAG_SEND_WRITE_ZEROES)) { debug (h, "server lacks write zeroes, ignoring claim of fast zero"); @@ -213,6 +219,12 @@ nbd_unlocked_can_cache (struct nbd_handle *h) return get_flag (h, NBD_FLAG_SEND_CACHE); } +int +nbd_unlocked_can_block_status_payload (struct nbd_handle *h) +{ + return get_flag (h, NBD_FLAG_BLOCK_STATUS_PAYLOAD); +} + int nbd_unlocked_can_meta_context (struct nbd_handle *h, const char *name) { diff --git a/examples/server-flags.c b/examples/server-flags.c index d156aced..f53b86ed 100644 --- a/examples/server-flags.c +++ b/examples/server-flags.c @@ -78,8 +78,13 @@ main (int argc, char *argv[]) PRINT_FLAG (nbd_can_multi_conn); PRINT_FLAG (nbd_can_trim); PRINT_FLAG (nbd_can_zero); -#if LIBNBD_HAVE_NBD_CAN_FAST_ZERO /* Added in 1.2 */ +#if LIBNBD_HAVE_NBD_CAN_FAST_ZERO + /* Added in 1.2 */ PRINT_FLAG (nbd_can_fast_zero); +#endif +#if LIBNBD_HAVE_NBD_CAN_BLOCK_STATUS_PAYLOAD + /* Added in 1.18 */ + PRINT_FLAG (nbd_can_block_status_payload); #endif PRINT_FLAG (nbd_is_read_only); PRINT_FLAG (nbd_is_rotational); diff --git a/interop/Makefile.am b/interop/Makefile.am index 9a7a5967..f8c4cb7d 100644 --- a/interop/Makefile.am +++ b/interop/Makefile.am @@ -27,6 +27,7 @@ EXTRA_DIST = \ list-exports-test-dir/disk2 \ structured-read.sh \ opt-extended-headers.sh \ + block-status-payload.sh \ $(NULL) TESTS_ENVIRONMENT = \ @@ -138,6 +139,7 @@ check_PROGRAMS += \ large-status \ structured-read \ opt-extended-headers \ + block-status-payload \ $(NULL) TESTS += \ interop-qemu-nbd \ @@ -150,6 +152,7 @@ TESTS += \ structured-read.sh \ interop-qemu-block-size.sh \ opt-extended-headers.sh \ + block-status-payload.sh \ $(NULL) interop_qemu_nbd_SOURCES = \ @@ -247,6 +250,9 @@ structured_read_LDADD = $(top_builddir)/lib/libnbd.la opt_extended_headers_SOURCES = opt-extended-headers.c opt_extended_headers_LDADD = $(top_builddir)/lib/libnbd.la +block_status_payload_SOURCES = block-status-payload.c +block_status_payload_LDADD = $(top_builddir)/lib/libnbd.la + endif HAVE_QEMU_NBD #---------------------------------------------------------------------- diff --git a/interop/block-status-payload.c b/interop/block-status-payload.c new file mode 100644 index 00000000..9603dfe5 --- /dev/null +++ b/interop/block-status-payload.c @@ -0,0 +1,126 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Test interaction with qemu using block status payload filtering. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <assert.h> +#include <stdbool.h> +#include <errno.h> + +#include <libnbd.h> + +#include "array-size.h" + +static const char *contexts[] = { + "base:allocation", + "qemu:allocation-depth", + "qemu:dirty-bitmap:bitmap0", + "qemu:dirty-bitmap:bitmap1", +}; + +static int +cb (void *opaque, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t len, int *error) +{ + /* Adjust seen according to which context was visited */ + unsigned int *seen = opaque; + size_t i; + + for (i = 0; i < ARRAY_SIZE (contexts); i++) + if (strcmp (contexts[i], metacontext) == 0) + break; + *seen |= 1 << i; + return 0; +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + int64_t exportsize; + unsigned int seen; + size_t i; + int r; + + if (argc < 2) { + fprintf (stderr, "%s qemu-nbd [args ...]\n", argv[0]); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + assert (ARRAY_SIZE (contexts) == 4); + for (i = 0; i < ARRAY_SIZE (contexts); i++) { + if (nbd_add_meta_context (nbd, contexts[i]) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + } + + if (nbd_connect_systemd_socket_activation (nbd, &argv[1]) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + r = nbd_can_block_status_payload (nbd); + if (r == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (r != 1) { + fprintf (stderr, "expecting block status payload support from qemu\n"); + exit (EXIT_FAILURE); + } + + exportsize = nbd_get_size (nbd); + if (exportsize == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* An unfiltered call should see all four contexts */ + seen = 0; + if (nbd_block_status_64 (nbd, exportsize, 0, + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (seen == 0xf); + + /* FIXME: Test filtered calls once the API is added */ + if (nbd_shutdown (nbd, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + + exit (EXIT_SUCCESS); +} diff --git a/interop/block-status-payload.sh b/interop/block-status-payload.sh new file mode 100755 index 00000000..a12cfc8a --- /dev/null +++ b/interop/block-status-payload.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright Red Hat +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +# Test use of block status payload for server filtering + +source ../tests/functions.sh +set -e +set -x + +requires qemu-img bitmap --help +# This test uses the qemu-nbd -A and -B options. +requires qemu-nbd -A -BA --version + +file="block-status-payload.qcow2" +rm -f $file +cleanup_fn rm -f $file + +# Create sparse file with two bitmaps. +qemu-img create -f qcow2 $file 1M +qemu-img bitmap --add --enable -f qcow2 $file bitmap0 +qemu-img bitmap --add --enable -f qcow2 $file bitmap1 + +# Unconditional part of test: qemu should not advertise block status payload +# support if extended headers are not in use +nbdsh -c ' +h.set_request_extended_headers(False) +h.add_meta_context("base:allocation") +h.add_meta_context("qemu:allocation-depth") +h.add_meta_context("qemu:dirty-bitmap:bitmap0") +h.add_meta_context("qemu:dirty-bitmap:bitmap1") +h.set_opt_mode(True) +args = ["qemu-nbd", "-f", "qcow2", "-A", "-B", "bitmap0", "-B", "bitmap1", + "'"$file"'"] +h.connect_systemd_socket_activation(args) +assert h.aio_is_negotiating() is True +assert h.get_extended_headers_negotiated() is False +# Flag not available until info or go +try: + h.can_block_status_payload() + assert False +except nbd.Error: + pass +h.opt_info() +assert h.can_block_status_payload() is False +assert h.can_meta_context("base:allocation") is True +h.opt_abort() +' + +# Conditional part of test: if qemu is new enough to support extended +# headers, we assume it can also support block status payload. +requires nbdinfo --can extended-headers -- [ qemu-nbd -r -f qcow2 "$file" ] +$VG ./block-status-payload \ + qemu-nbd -f qcow2 -A -B bitmap0 -B bitmap1 $file diff --git a/.gitignore b/.gitignore index fd81357b..a2d052bd 100644 --- a/.gitignore +++ b/.gitignore @@ -101,6 +101,7 @@ Makefile.in /info/nbdinfo /info/nbdinfo.1 /install-sh +/interop/block-status-payload /interop/dirty-bitmap /interop/interop-nbd-server /interop/interop-nbd-server-tls diff --git a/info/can.c b/info/can.c index 31c4a1ca..6dd68eeb 100644 --- a/info/can.c +++ b/info/can.c @@ -72,6 +72,11 @@ do_can (void) else if (strcasecmp (can, "rotational") == 0) feature = nbd_is_rotational (nbd); + else if (strcasecmp (can, "block status payload") == 0 || + strcasecmp (can, "block-status-payload") == 0 || + strcasecmp (can, "block_status_payload") == 0) + feature = nbd_can_block_status_payload (nbd); + else if (strcasecmp (can, "cache") == 0) feature = nbd_can_cache (nbd); diff --git a/info/show.c b/info/show.c index 920bbb0a..8914f927 100644 --- a/info/show.c +++ b/info/show.c @@ -54,7 +54,7 @@ show_one_export (struct nbd_handle *nbd, const char *desc, char *uri = NULL; int is_rotational, is_read_only; int can_cache, can_df, can_fast_zero, can_flush, can_fua, - can_multi_conn, can_trim, can_zero; + can_multi_conn, can_trim, can_zero, can_block_status_payload; int64_t block_minimum, block_preferred, block_maximum; string_vector contexts = empty_vector; bool show_context = false; @@ -120,6 +120,7 @@ show_one_export (struct nbd_handle *nbd, const char *desc, can_multi_conn = nbd_can_multi_conn (nbd); can_trim = nbd_can_trim (nbd); can_zero = nbd_can_zero (nbd); + can_block_status_payload = nbd_can_block_status_payload (nbd); block_minimum = nbd_get_block_size (nbd, LIBNBD_SIZE_MINIMUM); block_preferred = nbd_get_block_size (nbd, LIBNBD_SIZE_PREFERRED); block_maximum = nbd_get_block_size (nbd, LIBNBD_SIZE_MAXIMUM); @@ -161,6 +162,8 @@ show_one_export (struct nbd_handle *nbd, const char *desc, if (is_read_only >= 0) fprintf (fp, "\t%s: %s\n", "is_read_only", is_read_only ? "true" : "false"); + if (can_block_status_payload >= 0) + show_boolean ("can_block_status_payload", can_block_status_payload); if (can_cache >= 0) show_boolean ("can_cache", can_cache); if (can_df >= 0) @@ -230,6 +233,10 @@ show_one_export (struct nbd_handle *nbd, const char *desc, if (is_read_only >= 0) fprintf (fp, "\t\"%s\": %s,\n", "is_read_only", is_read_only ? "true" : "false"); + if (can_block_status_payload >= 0) + fprintf (fp, "\t\"%s\": %s,\n", + "can_block_status_payload", + can_block_status_payload ? "true" : "false"); if (can_cache >= 0) fprintf (fp, "\t\"%s\": %s,\n", "can_cache", can_cache ? "true" : "false"); -- 2.40.1
On Thu, May 25, 2023 at 08:01:07AM -0500, Eric Blake wrote:
In the recent NBD protocol extensions to add 64-bit commands [1], an additional option was added to allow NBD_CMD_BLOCK_STATUS pass a
to pass
client payload instructing the server to filter its answers in nbd.git commit e6f3b94a (mainly useful when the client requests more than one meta context with NBD_OPT_SET_META_CONTEXT). This patch lays the groundwork by exposing servers that advertise this capability, although libnbd does not yet actually utilize it until the next patch.
At the time this patch was written, qemu-nbd was also patched to provide such support; hence, an interop/ test shows the API in action.
[1] https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-header/doc/
Signed-off-by: Eric Blake <eblake@redhat.com> --- info/nbdinfo.pod | 10 ++- lib/nbd-protocol.h | 29 +++++--- generator/API.ml | 18 +++++ lib/flags.c | 12 +++ examples/server-flags.c | 7 +- interop/Makefile.am | 6 ++ interop/block-status-payload.c | 126 ++++++++++++++++++++++++++++++++ interop/block-status-payload.sh | 68 +++++++++++++++++ .gitignore | 1 + info/can.c | 5 ++ info/show.c | 9 ++- 11 files changed, 274 insertions(+), 17 deletions(-) create mode 100644 interop/block-status-payload.c create mode 100755 interop/block-status-payload.sh
diff --git a/info/nbdinfo.pod b/info/nbdinfo.pod index 9ea4a278..f5dc53fa 100644 --- a/info/nbdinfo.pod +++ b/info/nbdinfo.pod @@ -178,6 +178,8 @@ rotating disk: accessing nearby blocks may be faster than random access and requests should be sorted to improve performance. Many servers do not or cannot report this accurately.
+=item nbdinfo --can block-status-payload URI
Another case where "--has" sounds better ...
=item nbdinfo --can cache URI
=item nbdinfo --can df URI @@ -345,10 +347,10 @@ The command does not print anything. Instead it exits with success
For further information see the L<NBD protocol|https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md> -and the following libnbd functions: L<nbd_can_cache(3)>, -L<nbd_can_df(3)>, L<nbd_can_fast_zero(3)>, L<nbd_can_flush(3)>, -L<nbd_can_fua(3)>, L<nbd_can_multi_conn(3)>, L<nbd_can_trim(3)>, -L<nbd_can_zero(3)>, L<nbd_is_read_only(3)>, +and the following libnbd functions: L<nbd_can_block_status_payload(3)>, +L<nbd_can_cache(3)>, L<nbd_can_df(3)>, L<nbd_can_fast_zero(3)>, +L<nbd_can_flush(3)>, L<nbd_can_fua(3)>, L<nbd_can_multi_conn(3)>, +L<nbd_can_trim(3)>, L<nbd_can_zero(3)>, L<nbd_is_read_only(3)>, L<nbd_get_structured_replies_negotiated(3)>, L<nbd_get_extended_headers_negotiated(3)>.
diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h index b6fa9b8a..9e358122 100644 --- a/lib/nbd-protocol.h +++ b/lib/nbd-protocol.h @@ -102,17 +102,18 @@ struct nbd_fixed_new_option_reply { #define NBD_FLAG_NO_ZEROES (1 << 1)
/* Per-export flags. */ -#define NBD_FLAG_HAS_FLAGS (1 << 0) -#define NBD_FLAG_READ_ONLY (1 << 1) -#define NBD_FLAG_SEND_FLUSH (1 << 2) -#define NBD_FLAG_SEND_FUA (1 << 3) -#define NBD_FLAG_ROTATIONAL (1 << 4) -#define NBD_FLAG_SEND_TRIM (1 << 5) -#define NBD_FLAG_SEND_WRITE_ZEROES (1 << 6) -#define NBD_FLAG_SEND_DF (1 << 7) -#define NBD_FLAG_CAN_MULTI_CONN (1 << 8) -#define NBD_FLAG_SEND_CACHE (1 << 10) -#define NBD_FLAG_SEND_FAST_ZERO (1 << 11) +#define NBD_FLAG_HAS_FLAGS (1 << 0) +#define NBD_FLAG_READ_ONLY (1 << 1) +#define NBD_FLAG_SEND_FLUSH (1 << 2) +#define NBD_FLAG_SEND_FUA (1 << 3) +#define NBD_FLAG_ROTATIONAL (1 << 4) +#define NBD_FLAG_SEND_TRIM (1 << 5) +#define NBD_FLAG_SEND_WRITE_ZEROES (1 << 6) +#define NBD_FLAG_SEND_DF (1 << 7) +#define NBD_FLAG_CAN_MULTI_CONN (1 << 8) +#define NBD_FLAG_SEND_CACHE (1 << 10) +#define NBD_FLAG_SEND_FAST_ZERO (1 << 11) +#define NBD_FLAG_BLOCK_STATUS_PAYLOAD (1 << 12)
/* NBD options (new style handshake only). */ #define NBD_OPT_EXPORT_NAME 1 @@ -204,6 +205,12 @@ struct nbd_request_ext { uint64_t count; /* Request effect or payload length. */ } NBD_ATTRIBUTE_PACKED;
+/* Extended request payload for NBD_CMD_BLOCK_STATUS, when supported. */ +struct nbd_block_status_payload { + uint64_t length; /* Effective length of client request */ + /* followed by array of uint32_t ids */ +} NBD_ATTRIBUTE_PACKED; + /* Simple reply (server -> client). */ struct nbd_simple_reply { uint32_t magic; /* NBD_SIMPLE_REPLY_MAGIC. */ diff --git a/generator/API.ml b/generator/API.ml index 85625bbd..5a31ce3b 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -2327,6 +2327,23 @@ "can_fast_zero", { example = Some "examples/server-flags.c"; };
+ "can_block_status_payload", { + default_call with + args = []; ret = RBool; + permitted_states = [ Negotiating; Connected; Closed ]; + shortdesc = "does the server support the block status payload flag?"; + longdesc = "\ +Returns true if the server supports the use of the +C<LIBNBD_CMD_FLAG_PAYLOAD_LEN> flag to allow filtering of the +block status command. Returns +false if the server does not. Note that this will never return +true if L<nbd_get_extended_headers_negotiated(3)> is false." +^ non_blocking_test_call_description; + see_also = [SectionLink "Flag calls"; Link "opt_info"; + Link "get_extended_headers_negotiated"]; + example = Some "examples/server-flags.c"; + }; + "can_df", { default_call with args = []; ret = RBool; @@ -4183,6 +4200,7 @@ let first_version = "get_extended_headers_negotiated", (1, 18); "opt_extended_headers", (1, 18); "aio_opt_extended_headers", (1, 18); + "can_block_status_payload", (1, 18);
(* These calls are proposed for a future version of libnbd, but * have not been added to any released version so far. diff --git a/lib/flags.c b/lib/flags.c index be880acf..7e6ddedd 100644 --- a/lib/flags.c +++ b/lib/flags.c @@ -66,6 +66,12 @@ nbd_internal_set_size_and_flags (struct nbd_handle *h, eflags &= ~NBD_FLAG_SEND_DF; }
+ if (eflags & NBD_FLAG_BLOCK_STATUS_PAYLOAD && !h->extended_headers) { + debug (h, "server lacks extended headers, ignoring claim " + "of block status payload"); + eflags &= ~NBD_FLAG_BLOCK_STATUS_PAYLOAD; + } + if (eflags & NBD_FLAG_SEND_FAST_ZERO && !(eflags & NBD_FLAG_SEND_WRITE_ZEROES)) { debug (h, "server lacks write zeroes, ignoring claim of fast zero"); @@ -213,6 +219,12 @@ nbd_unlocked_can_cache (struct nbd_handle *h) return get_flag (h, NBD_FLAG_SEND_CACHE); }
+int +nbd_unlocked_can_block_status_payload (struct nbd_handle *h) +{ + return get_flag (h, NBD_FLAG_BLOCK_STATUS_PAYLOAD); +} + int nbd_unlocked_can_meta_context (struct nbd_handle *h, const char *name) { diff --git a/examples/server-flags.c b/examples/server-flags.c index d156aced..f53b86ed 100644 --- a/examples/server-flags.c +++ b/examples/server-flags.c @@ -78,8 +78,13 @@ main (int argc, char *argv[]) PRINT_FLAG (nbd_can_multi_conn); PRINT_FLAG (nbd_can_trim); PRINT_FLAG (nbd_can_zero); -#if LIBNBD_HAVE_NBD_CAN_FAST_ZERO /* Added in 1.2 */ +#if LIBNBD_HAVE_NBD_CAN_FAST_ZERO + /* Added in 1.2 */ PRINT_FLAG (nbd_can_fast_zero); +#endif +#if LIBNBD_HAVE_NBD_CAN_BLOCK_STATUS_PAYLOAD + /* Added in 1.18 */ + PRINT_FLAG (nbd_can_block_status_payload); #endif PRINT_FLAG (nbd_is_read_only); PRINT_FLAG (nbd_is_rotational); diff --git a/interop/Makefile.am b/interop/Makefile.am index 9a7a5967..f8c4cb7d 100644 --- a/interop/Makefile.am +++ b/interop/Makefile.am @@ -27,6 +27,7 @@ EXTRA_DIST = \ list-exports-test-dir/disk2 \ structured-read.sh \ opt-extended-headers.sh \ + block-status-payload.sh \ $(NULL)
TESTS_ENVIRONMENT = \ @@ -138,6 +139,7 @@ check_PROGRAMS += \ large-status \ structured-read \ opt-extended-headers \ + block-status-payload \ $(NULL) TESTS += \ interop-qemu-nbd \ @@ -150,6 +152,7 @@ TESTS += \ structured-read.sh \ interop-qemu-block-size.sh \ opt-extended-headers.sh \ + block-status-payload.sh \ $(NULL)
interop_qemu_nbd_SOURCES = \ @@ -247,6 +250,9 @@ structured_read_LDADD = $(top_builddir)/lib/libnbd.la opt_extended_headers_SOURCES = opt-extended-headers.c opt_extended_headers_LDADD = $(top_builddir)/lib/libnbd.la
+block_status_payload_SOURCES = block-status-payload.c +block_status_payload_LDADD = $(top_builddir)/lib/libnbd.la + endif HAVE_QEMU_NBD
#---------------------------------------------------------------------- diff --git a/interop/block-status-payload.c b/interop/block-status-payload.c new file mode 100644 index 00000000..9603dfe5 --- /dev/null +++ b/interop/block-status-payload.c @@ -0,0 +1,126 @@ +/* NBD client library in userspace + * Copyright Red Hat + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Test interaction with qemu using block status payload filtering. */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <assert.h> +#include <stdbool.h> +#include <errno.h> + +#include <libnbd.h> + +#include "array-size.h" + +static const char *contexts[] = { + "base:allocation", + "qemu:allocation-depth", + "qemu:dirty-bitmap:bitmap0", + "qemu:dirty-bitmap:bitmap1", +}; + +static int +cb (void *opaque, const char *metacontext, uint64_t offset, + nbd_extent *entries, size_t len, int *error) +{ + /* Adjust seen according to which context was visited */ + unsigned int *seen = opaque; + size_t i; + + for (i = 0; i < ARRAY_SIZE (contexts); i++) + if (strcmp (contexts[i], metacontext) == 0) + break; + *seen |= 1 << i; + return 0; +} + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + int64_t exportsize; + unsigned int seen; + size_t i; + int r; + + if (argc < 2) { + fprintf (stderr, "%s qemu-nbd [args ...]\n", argv[0]); + exit (EXIT_FAILURE); + } + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + assert (ARRAY_SIZE (contexts) == 4); + for (i = 0; i < ARRAY_SIZE (contexts); i++) { + if (nbd_add_meta_context (nbd, contexts[i]) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + } + + if (nbd_connect_systemd_socket_activation (nbd, &argv[1]) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + r = nbd_can_block_status_payload (nbd); + if (r == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (r != 1) { + fprintf (stderr, "expecting block status payload support from qemu\n"); + exit (EXIT_FAILURE); + } + + exportsize = nbd_get_size (nbd); + if (exportsize == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* An unfiltered call should see all four contexts */ + seen = 0; + if (nbd_block_status_64 (nbd, exportsize, 0, + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (seen == 0xf); + + /* FIXME: Test filtered calls once the API is added */ + if (nbd_shutdown (nbd, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + + exit (EXIT_SUCCESS); +} diff --git a/interop/block-status-payload.sh b/interop/block-status-payload.sh new file mode 100755 index 00000000..a12cfc8a --- /dev/null +++ b/interop/block-status-payload.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright Red Hat +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +# Test use of block status payload for server filtering + +source ../tests/functions.sh +set -e +set -x + +requires qemu-img bitmap --help +# This test uses the qemu-nbd -A and -B options. +requires qemu-nbd -A -BA --version + +file="block-status-payload.qcow2" +rm -f $file +cleanup_fn rm -f $file + +# Create sparse file with two bitmaps. +qemu-img create -f qcow2 $file 1M +qemu-img bitmap --add --enable -f qcow2 $file bitmap0 +qemu-img bitmap --add --enable -f qcow2 $file bitmap1 + +# Unconditional part of test: qemu should not advertise block status payload +# support if extended headers are not in use +nbdsh -c ' +h.set_request_extended_headers(False) +h.add_meta_context("base:allocation") +h.add_meta_context("qemu:allocation-depth") +h.add_meta_context("qemu:dirty-bitmap:bitmap0") +h.add_meta_context("qemu:dirty-bitmap:bitmap1") +h.set_opt_mode(True) +args = ["qemu-nbd", "-f", "qcow2", "-A", "-B", "bitmap0", "-B", "bitmap1", + "'"$file"'"] +h.connect_systemd_socket_activation(args) +assert h.aio_is_negotiating() is True +assert h.get_extended_headers_negotiated() is False +# Flag not available until info or go +try: + h.can_block_status_payload() + assert False +except nbd.Error: + pass +h.opt_info() +assert h.can_block_status_payload() is False +assert h.can_meta_context("base:allocation") is True +h.opt_abort() +' + +# Conditional part of test: if qemu is new enough to support extended +# headers, we assume it can also support block status payload. +requires nbdinfo --can extended-headers -- [ qemu-nbd -r -f qcow2 "$file" ] +$VG ./block-status-payload \ + qemu-nbd -f qcow2 -A -B bitmap0 -B bitmap1 $file diff --git a/.gitignore b/.gitignore index fd81357b..a2d052bd 100644 --- a/.gitignore +++ b/.gitignore @@ -101,6 +101,7 @@ Makefile.in /info/nbdinfo /info/nbdinfo.1 /install-sh +/interop/block-status-payload /interop/dirty-bitmap /interop/interop-nbd-server /interop/interop-nbd-server-tls diff --git a/info/can.c b/info/can.c index 31c4a1ca..6dd68eeb 100644 --- a/info/can.c +++ b/info/can.c @@ -72,6 +72,11 @@ do_can (void) else if (strcasecmp (can, "rotational") == 0) feature = nbd_is_rotational (nbd);
+ else if (strcasecmp (can, "block status payload") == 0 || + strcasecmp (can, "block-status-payload") == 0 || + strcasecmp (can, "block_status_payload") == 0) + feature = nbd_can_block_status_payload (nbd); + else if (strcasecmp (can, "cache") == 0) feature = nbd_can_cache (nbd);
diff --git a/info/show.c b/info/show.c index 920bbb0a..8914f927 100644 --- a/info/show.c +++ b/info/show.c @@ -54,7 +54,7 @@ show_one_export (struct nbd_handle *nbd, const char *desc, char *uri = NULL; int is_rotational, is_read_only; int can_cache, can_df, can_fast_zero, can_flush, can_fua, - can_multi_conn, can_trim, can_zero; + can_multi_conn, can_trim, can_zero, can_block_status_payload; int64_t block_minimum, block_preferred, block_maximum; string_vector contexts = empty_vector; bool show_context = false; @@ -120,6 +120,7 @@ show_one_export (struct nbd_handle *nbd, const char *desc, can_multi_conn = nbd_can_multi_conn (nbd); can_trim = nbd_can_trim (nbd); can_zero = nbd_can_zero (nbd); + can_block_status_payload = nbd_can_block_status_payload (nbd); block_minimum = nbd_get_block_size (nbd, LIBNBD_SIZE_MINIMUM); block_preferred = nbd_get_block_size (nbd, LIBNBD_SIZE_PREFERRED); block_maximum = nbd_get_block_size (nbd, LIBNBD_SIZE_MAXIMUM); @@ -161,6 +162,8 @@ show_one_export (struct nbd_handle *nbd, const char *desc, if (is_read_only >= 0) fprintf (fp, "\t%s: %s\n", "is_read_only", is_read_only ? "true" : "false"); + if (can_block_status_payload >= 0) + show_boolean ("can_block_status_payload", can_block_status_payload); if (can_cache >= 0) show_boolean ("can_cache", can_cache); if (can_df >= 0) @@ -230,6 +233,10 @@ show_one_export (struct nbd_handle *nbd, const char *desc, if (is_read_only >= 0) fprintf (fp, "\t\"%s\": %s,\n", "is_read_only", is_read_only ? "true" : "false"); + if (can_block_status_payload >= 0) + fprintf (fp, "\t\"%s\": %s,\n", + "can_block_status_payload", + can_block_status_payload ? "true" : "false"); if (can_cache >= 0) fprintf (fp, "\t\"%s\": %s,\n", "can_cache", can_cache ? "true" : "false");
Reviewed-by: Richard W.M. Jones <rjones@redhat.com> -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
As part of extending NBD to support 64-bit lengths, the protocol also added an option for servers to allow clients to request filtered responses to NBD_CMD_BLOCK_STATUS when more than one meta-context is negotiated (see NBD commit e6f3b94a). At the same time as this patch, qemu-nbd was taught to support and advertise this feature as a server, but does not utilize it as a client (qemu doesn't yet need to connect to multiple contexts at once). Thus, addding generic client support and enhancing the interop/ test in libnbd is needed to prove that the feature is viable and worth standardizing. --- lib/internal.h | 5 +- generator/API.ml | 71 +++++++++++++++-- generator/states-issue-command.c | 4 +- lib/aio.c | 7 +- lib/rw.c | 127 ++++++++++++++++++++++++++++++- interop/block-status-payload.c | 117 +++++++++++++++++++++++++++- interop/block-status-payload.sh | 14 +++- info/info-can.sh | 3 + 8 files changed, 336 insertions(+), 12 deletions(-) diff --git a/lib/internal.h b/lib/internal.h index 2948b77b..64921de9 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -73,6 +73,8 @@ struct meta_context { }; DEFINE_VECTOR_TYPE (meta_vector, struct meta_context); +DEFINE_VECTOR_TYPE(uint32_vector, uint32_t); + struct export { char *name; char *description; @@ -380,7 +382,8 @@ struct command { uint64_t cookie; uint64_t offset; uint64_t count; - void *data; /* Buffer for read/write */ + void *data; /* Buffer for read/write, uint32_vector* for status payload */ + uint32_vector *ids; /* For block status with payload */ struct command_cb cb; bool initialized; /* For read, true if getting a hole may skip memset */ uint32_t data_seen; /* For read, cumulative size of data chunks seen */ diff --git a/generator/API.ml b/generator/API.ml index 5a31ce3b..a26ed1da 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -2335,12 +2335,13 @@ "can_block_status_payload", { longdesc = "\ Returns true if the server supports the use of the C<LIBNBD_CMD_FLAG_PAYLOAD_LEN> flag to allow filtering of the -block status command. Returns +block status command (see L<nbd_block_status_filter(3)>). Returns false if the server does not. Note that this will never return true if L<nbd_get_extended_headers_negotiated(3)> is false." ^ non_blocking_test_call_description; see_also = [SectionLink "Flag calls"; Link "opt_info"; - Link "get_extended_headers_negotiated"]; + Link "get_extended_headers_negotiated"; + Link "block_status_filter"]; example = Some "examples/server-flags.c"; }; @@ -2409,6 +2410,10 @@ "can_meta_context", { meta contexts were requested but there is a missing or failed attempt at NBD_OPT_SET_META_CONTEXT during option negotiation. +If the server supports block status filtering (see +L<nbd_can_block_status_payload(3)>, this function must return +true for any filter name passed to L<nbd_block_status_filter(3)>. + The single parameter is the name of the metadata context, for example C<LIBNBD_CONTEXT_BASE_ALLOCATION>. B<E<lt>libnbd.hE<gt>> includes defined constants for well-known @@ -2941,9 +2946,12 @@ "block_status_64", { information about blocks beginning from the specified offset to be returned. The C<count> parameter is a hint: the server may choose to return less status, or the final block -may extend beyond the requested range. If multiple contexts +may extend beyond the requested range. When multiple contexts are supported, the number of blocks and cumulative length -of those blocks need not be identical between contexts. +of those blocks need not be identical between contexts; this +command generally returns the status of all negotiated contexts, +while some servers also support a filtered request (see +L<nbd_can_block_status_payload(3)>, L<nbd_block_status_filter(3)>). Note that not all servers can support a C<count> of 4GiB or larger; L<nbd_get_extended_headers_negotiated(3)> indicates which servers @@ -2993,11 +3001,38 @@ "block_status_64", { does not exceed C<count> bytes; however, libnbd does not validate that the server obeyed the flag." ^ strict_call_description; - see_also = [Link "block_status"; + see_also = [Link "block_status"; Link "block_status_filter"; Link "add_meta_context"; Link "can_meta_context"; Link "aio_block_status_64"; Link "set_strict_mode"]; }; + "block_status_filter", { + default_call with + args = [ UInt64 "count"; UInt64 "offset"; StringList "contexts"; + Closure extent64_closure ]; + optargs = [ OFlags ("flags", cmd_flags, Some ["REQ_ONE"; "PAYLOAD_LEN"]) ]; + ret = RErr; + permitted_states = [ Connected ]; + shortdesc = "send filtered block status command, with 64-bit callback"; + longdesc = "\ +Issue a filtered block status command to the NBD server. If +supported by the server (see L<nbd_can_block_status_payload(3)>), +this causes metadata context information about blocks beginning +from the specified offset to be returned, and with the result +limited to just the contexts specified in C<filter>. Note that +all strings in C<filter> must be supported by +L<nbd_can_meta_context(3)>. + +All other parameters to this function have the same semantics +as in L<nbd_block_status_64(3)>; except that for convenience, +the C<flags> parameter may additionally contain or omit +C<LIBNBD_CMD_FLAG_PAYLOAD_LEN>." +^ strict_call_description; + see_also = [Link "block_status_64"; + Link "can_block_status_payload"; Link "can_meta_context"; + Link "aio_block_status_filter"; Link "set_strict_mode"]; + }; + "poll", { default_call with args = [ Int "timeout" ]; ret = RInt; @@ -3667,6 +3702,30 @@ "aio_block_status_64", { Link "set_strict_mode"]; }; + "aio_block_status_filter", { + default_call with + args = [ UInt64 "count"; UInt64 "offset"; StringList "contexts"; + Closure extent64_closure ]; + optargs = [ OClosure completion_closure; + OFlags ("flags", cmd_flags, Some ["REQ_ONE"; "PAYLOAD_LEN"]) ]; + ret = RCookie; + permitted_states = [ Connected ]; + shortdesc = "send filtered block status command to the NBD server"; + longdesc = "\ +Send a filtered block status command to the NBD server. + +To check if the command completed, call L<nbd_aio_command_completed(3)>. +Or supply the optional C<completion_callback> which will be invoked +as described in L<libnbd(3)/Completion callbacks>. + +Other parameters behave as documented in L<nbd_block_status_filter(3)>." +^ strict_call_description; + see_also = [SectionLink "Issuing asynchronous commands"; + Link "aio_block_status_64"; Link "block_status_filter"; + Link "can_meta_context"; Link "can_block_status_payload"; + Link "set_strict_mode"]; + }; + "aio_get_fd", { default_call with args = []; ret = RFd; @@ -4201,6 +4260,8 @@ let first_version = "opt_extended_headers", (1, 18); "aio_opt_extended_headers", (1, 18); "can_block_status_payload", (1, 18); + "block_status_filter", (1, 18); + "aio_block_status_filter", (1, 18); (* These calls are proposed for a future version of libnbd, but * have not been added to any released version so far. diff --git a/generator/states-issue-command.c b/generator/states-issue-command.c index 79136b61..5307731d 100644 --- a/generator/states-issue-command.c +++ b/generator/states-issue-command.c @@ -84,7 +84,9 @@ ISSUE_COMMAND.PREPARE_WRITE_PAYLOAD: assert (h->cmds_to_issue != NULL); cmd = h->cmds_to_issue; assert (cmd->cookie == be64toh (h->req.compact.handle)); - if (cmd->type == NBD_CMD_WRITE) { + if (cmd->type == NBD_CMD_WRITE || + (h->extended_headers && cmd->type == NBD_CMD_BLOCK_STATUS && + cmd->flags & NBD_CMD_FLAG_PAYLOAD_LEN)) { h->wbuf = cmd->data; h->wlen = cmd->count; if (cmd->next && cmd->count < 64 * 1024) diff --git a/lib/aio.c b/lib/aio.c index a419ac32..77b20c32 100644 --- a/lib/aio.c +++ b/lib/aio.c @@ -32,8 +32,13 @@ void nbd_internal_retire_and_free_command (struct command *cmd) { /* Free the callbacks. */ - if (cmd->type == NBD_CMD_BLOCK_STATUS) + if (cmd->type == NBD_CMD_BLOCK_STATUS) { + if (cmd->ids) { + uint32_vector_reset (cmd->ids); + free (cmd->ids); + } FREE_CALLBACK (cmd->cb.fn.extent); + } if (cmd->type == NBD_CMD_READ) FREE_CALLBACK (cmd->cb.fn.chunk); FREE_CALLBACK (cmd->cb.completion); diff --git a/lib/rw.c b/lib/rw.c index bea55fa1..db6bc0bc 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -242,6 +242,26 @@ nbd_unlocked_block_status_64 (struct nbd_handle *h, return wait_for_command (h, cookie); } +/* Issue a filtered block status command and wait for the reply. */ +int +nbd_unlocked_block_status_filter (struct nbd_handle *h, + uint64_t count, uint64_t offset, + char **filter, + nbd_extent64_callback *extent64, + uint32_t flags) +{ + int64_t cookie; + nbd_completion_callback c = NBD_NULL_COMPLETION; + + cookie = nbd_unlocked_aio_block_status_filter (h, count, offset, filter, + extent64, &c, flags); + if (cookie == -1) + return -1; + + assert (CALLBACK_IS_NULL (*extent64)); + return wait_for_command (h, cookie); +} + /* count_err represents the errno to return if bounds check fail */ int64_t nbd_internal_command_common (struct nbd_handle *h, @@ -250,6 +270,7 @@ nbd_internal_command_common (struct nbd_handle *h, void *data, struct command_cb *cb) { struct command *cmd; + uint32_vector *ids = NULL; if (h->disconnect_request) { set_error (EINVAL, "cannot request more commands after NBD_CMD_DISC"); @@ -297,10 +318,23 @@ nbd_internal_command_common (struct nbd_handle *h, } break; + case NBD_CMD_BLOCK_STATUS: + if (data) { + ids = data; + count = ids->len * sizeof (uint32_t); + data = ids->ptr; + if (count > MAX_REQUEST_SIZE || + (h->strict & LIBNBD_STRICT_PAYLOAD && count > h->payload_maximum)) { + set_error (ERANGE, "filter set too large"); + goto err; + } + break; + } + /* fallthrough */ + default: /* Other commands are limited by the 32 bit field in the command * structure on the wire, unless extended headers were negotiated. */ - default: if (!h->extended_headers && count > UINT32_MAX) { set_error (ERANGE, "request too large: maximum request size is %" PRIu32, UINT32_MAX); @@ -320,6 +354,7 @@ nbd_internal_command_common (struct nbd_handle *h, cmd->offset = offset; cmd->count = count; cmd->data = data; + cmd->ids = ids; if (cb) cmd->cb = *cb; @@ -364,8 +399,13 @@ nbd_internal_command_common (struct nbd_handle *h, err: /* Since we did not queue the command, we must free the callbacks. */ if (cb) { - if (type == NBD_CMD_BLOCK_STATUS) + if (type == NBD_CMD_BLOCK_STATUS) { + if (ids) { + uint32_vector_reset (ids); + free (ids); + } FREE_CALLBACK (cb->fn.extent); + } if (type == NBD_CMD_READ) FREE_CALLBACK (cb->fn.chunk); FREE_CALLBACK (cb->completion); @@ -609,3 +649,86 @@ nbd_unlocked_aio_block_status_64 (struct nbd_handle *h, return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb); } + +int64_t +nbd_unlocked_aio_block_status_filter (struct nbd_handle *h, + uint64_t count, uint64_t offset, + char **filter, + nbd_extent64_callback *extent64, + nbd_completion_callback *completion, + uint32_t flags) +{ + struct command_cb cb = { .fn.extent = *extent64, + .completion = *completion }; + uint32_vector *ids; + char *name; + size_t i; + + /* Because this affects wire format, it is more convenient to manage + * PAYLOAD_LEN by what was negotiated than to require the user to + * have to set it correctly. + */ + if (!h->extended_headers) { + set_error (ENOTSUP, "server does not support extended headers"); + return -1; + } + flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN; + + if (h->strict & LIBNBD_STRICT_COMMANDS) { + if (nbd_unlocked_can_block_status_payload (h) != 1) { + set_error (EINVAL, + "server does not support the block status payload flag"); + return -1; + } + + if (!h->meta_valid || h->meta_contexts.len == 0) { + set_error (ENOTSUP, "did not negotiate any metadata contexts, " + "either you did not call nbd_add_meta_context before " + "connecting or the server does not support it"); + return -1; + } + } + + ids = calloc (1, sizeof *ids); + if (ids == NULL) { + set_error (errno, "calloc"); + return -1; + } + if (uint32_vector_append (ids, htobe32 (count >> 32)) == -1 || + uint32_vector_append (ids, htobe32 (count)) == -1) { + set_error (errno, "realloc"); + goto fail; + } + + /* O(n^2) search - hopefully filter and negotiated contexts are both small */ + for ( ; (name = *filter) != NULL; filter++) { + if (!h->meta_valid) { + set_error (EINVAL, "context %s not negotiated", name); + goto fail; + } + for (i = 0; i < h->meta_contexts.len; i++) { + struct meta_context *meta = &h->meta_contexts.ptr[i]; + if (strcmp (name, meta->name) == 0) { + if (uint32_vector_append (ids, htobe32 (meta->context_id)) == -1) { + set_error (errno, "realloc"); + goto fail; + } + break; + } + } + if (i == h->meta_contexts.len) { + set_error (EINVAL, "context %s not negotiated", name); + goto fail; + } + } + + SET_CALLBACK_TO_NULL (*extent64); + SET_CALLBACK_TO_NULL (*completion); + return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, + count, EINVAL, ids, &cb); + + fail: + uint32_vector_reset (ids); + free (ids); + return -1; +} diff --git a/interop/block-status-payload.c b/interop/block-status-payload.c index 9603dfe5..704b25aa 100644 --- a/interop/block-status-payload.c +++ b/interop/block-status-payload.c @@ -54,11 +54,26 @@ cb (void *opaque, const char *metacontext, uint64_t offset, return 0; } +static char ** +list (unsigned int use) +{ + static const char *array[ARRAY_SIZE (contexts) + 1]; + size_t i, j; + + assert (use < 1 << ARRAY_SIZE (contexts)); + for (i = j = 0; i < ARRAY_SIZE (contexts); i++) + if (use & (1 << i)) + array[j++] = contexts[i]; + array[j] = NULL; + return (char **) array; +} + int main (int argc, char *argv[]) { struct nbd_handle *nbd; int64_t exportsize; + uint64_t bytes_sent; unsigned int seen; size_t i; int r; @@ -114,7 +129,107 @@ main (int argc, char *argv[]) } assert (seen == 0xf); - /* FIXME: Test filtered calls once the API is added */ + /* Filtering with all contexts listed, same effect as unfilitered call */ + seen = 0; + if (nbd_block_status_filter (nbd, exportsize, 0, list (0xf), + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (seen == 0xf); + + /* Filtering with just two out of four contexts; test optional flag */ + seen = 0; + if (nbd_block_status_filter (nbd, exportsize, 0, list (0x5), + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (seen == 0x5); + + /* Filtering with one context, near end of file (to make sure the + * payload length isn't confused with the effect length) + */ + seen = 0; + if (nbd_block_status_filter (nbd, 1, exportsize - 1, list (0x2), + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (seen == 0x2); + + /* Filtering with no contexts - pointless, so qemu rejects it */ + bytes_sent = nbd_stats_bytes_sent (nbd); + seen = 0; + if (nbd_block_status_filter (nbd, exportsize, 0, list (0x0), + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) != -1) { + fprintf (stderr, "expecting block status failure\n"); + exit (EXIT_FAILURE); + } + assert (seen == 0x0); + if (nbd_get_errno () != EINVAL) { + fprintf (stderr, "expecting EINVAL after block status failure\n"); + exit (EXIT_FAILURE); + } + if (nbd_stats_bytes_sent (nbd) <= bytes_sent) { + fprintf (stderr, "expecting server-side rejection of bad request\n"); + exit (EXIT_FAILURE); + } + + /* Giving unknown string triggers EINVAL from libnbd */ + bytes_sent = nbd_stats_bytes_sent (nbd); + seen = 0; + { + const char *bogus[] = { "qemu:dirty-bitmap:bitmap2", NULL }; + if (nbd_block_status_filter (nbd, exportsize, 0, (char **) bogus, + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) != -1) { + fprintf (stderr, "expecting block status failure\n"); + exit (EXIT_FAILURE); + } + } + if (nbd_get_errno () != EINVAL) { + fprintf (stderr, "expecting EINVAL after block status failure\n"); + exit (EXIT_FAILURE); + } + assert (seen == 0x0); + if (nbd_stats_bytes_sent (nbd) != bytes_sent) { + fprintf (stderr, "expecting client-side rejection of bad request\n"); + exit (EXIT_FAILURE); + } + + /* Giving same string twice triggers EINVAL from qemu */ + seen = 0; + { + const char *dupes[] = { "base:allocation", "base:allocation", NULL }; + if (nbd_block_status_filter (nbd, exportsize, 0, (char **) dupes, + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) != -1) { + fprintf (stderr, "expecting block status failure\n"); + exit (EXIT_FAILURE); + } + } + if (nbd_get_errno () != EINVAL) { + fprintf (stderr, "expecting EINVAL after block status failure\n"); + exit (EXIT_FAILURE); + } + assert (seen == 0x0); + if (nbd_stats_bytes_sent (nbd) <= bytes_sent) { + fprintf (stderr, "expecting server-side rejection of bad request\n"); + exit (EXIT_FAILURE); + } + + /* Done */ if (nbd_shutdown (nbd, 0) == -1) { fprintf (stderr, "%s\n", nbd_get_error ()); exit (EXIT_FAILURE); diff --git a/interop/block-status-payload.sh b/interop/block-status-payload.sh index a12cfc8a..0e6681b6 100755 --- a/interop/block-status-payload.sh +++ b/interop/block-status-payload.sh @@ -49,6 +49,7 @@ args = ["qemu-nbd", "-f", "qcow2", "-A", "-B", "bitmap0", "-B", "bitmap1", h.connect_systemd_socket_activation(args) assert h.aio_is_negotiating() is True assert h.get_extended_headers_negotiated() is False + # Flag not available until info or go try: h.can_block_status_payload() @@ -58,7 +59,18 @@ except nbd.Error: h.opt_info() assert h.can_block_status_payload() is False assert h.can_meta_context("base:allocation") is True -h.opt_abort() + +# Filter request not allowed if not advertised +def f(): + assert False +h.opt_go() +assert h.can_block_status_payload() is False +try: + h.block_status_filter(0, 512, ["base:allocation"], f) + assert False +except nbd.Error: + pass +h.shutdown() ' # Conditional part of test: if qemu is new enough to support extended diff --git a/info/info-can.sh b/info/info-can.sh index 8154d1ce..097837d2 100755 --- a/info/info-can.sh +++ b/info/info-can.sh @@ -38,6 +38,9 @@ requires bash -c "nbdkit sh --dump-plugin | grep has_can_cache=1" # and oldstyle never, but that feels like depending a bit too much on # the implementation. +# --can block-status-payload is not supported by nbdkit yet. Testing +# is done during interop with new-enough qemu. + # --can structured-reply is not a per-export setting, but rather # something set on the server as a whole. -- 2.40.1
On Thu, May 25, 2023 at 08:01:08AM -0500, Eric Blake wrote:
As part of extending NBD to support 64-bit lengths, the protocol also added an option for servers to allow clients to request filtered responses to NBD_CMD_BLOCK_STATUS when more than one meta-context is negotiated (see NBD commit e6f3b94a). At the same time as this patch, qemu-nbd was taught to support and advertise this feature as a server, but does not utilize it as a client (qemu doesn't yet need to connect to multiple contexts at once). Thus, addding generic client support and enhancing the interop/ test in libnbd is needed to prove that the feature is viable and worth standardizing. --- lib/internal.h | 5 +- generator/API.ml | 71 +++++++++++++++-- generator/states-issue-command.c | 4 +- lib/aio.c | 7 +- lib/rw.c | 127 ++++++++++++++++++++++++++++++- interop/block-status-payload.c | 117 +++++++++++++++++++++++++++- interop/block-status-payload.sh | 14 +++- info/info-can.sh | 3 + 8 files changed, 336 insertions(+), 12 deletions(-)
diff --git a/lib/internal.h b/lib/internal.h index 2948b77b..64921de9 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -73,6 +73,8 @@ struct meta_context { }; DEFINE_VECTOR_TYPE (meta_vector, struct meta_context);
+DEFINE_VECTOR_TYPE(uint32_vector, uint32_t); + struct export { char *name; char *description; @@ -380,7 +382,8 @@ struct command { uint64_t cookie; uint64_t offset; uint64_t count; - void *data; /* Buffer for read/write */ + void *data; /* Buffer for read/write, uint32_vector* for status payload */ + uint32_vector *ids; /* For block status with payload */ struct command_cb cb; bool initialized; /* For read, true if getting a hole may skip memset */ uint32_t data_seen; /* For read, cumulative size of data chunks seen */ diff --git a/generator/API.ml b/generator/API.ml index 5a31ce3b..a26ed1da 100644 --- a/generator/API.ml +++ b/generator/API.ml @@ -2335,12 +2335,13 @@ "can_block_status_payload", { longdesc = "\ Returns true if the server supports the use of the C<LIBNBD_CMD_FLAG_PAYLOAD_LEN> flag to allow filtering of the -block status command. Returns +block status command (see L<nbd_block_status_filter(3)>). Returns false if the server does not. Note that this will never return true if L<nbd_get_extended_headers_negotiated(3)> is false." ^ non_blocking_test_call_description; see_also = [SectionLink "Flag calls"; Link "opt_info"; - Link "get_extended_headers_negotiated"]; + Link "get_extended_headers_negotiated"; + Link "block_status_filter"]; example = Some "examples/server-flags.c"; };
@@ -2409,6 +2410,10 @@ "can_meta_context", { meta contexts were requested but there is a missing or failed attempt at NBD_OPT_SET_META_CONTEXT during option negotiation.
+If the server supports block status filtering (see +L<nbd_can_block_status_payload(3)>, this function must return +true for any filter name passed to L<nbd_block_status_filter(3)>. + The single parameter is the name of the metadata context, for example C<LIBNBD_CONTEXT_BASE_ALLOCATION>. B<E<lt>libnbd.hE<gt>> includes defined constants for well-known @@ -2941,9 +2946,12 @@ "block_status_64", { information about blocks beginning from the specified offset to be returned. The C<count> parameter is a hint: the server may choose to return less status, or the final block -may extend beyond the requested range. If multiple contexts +may extend beyond the requested range. When multiple contexts are supported, the number of blocks and cumulative length -of those blocks need not be identical between contexts. +of those blocks need not be identical between contexts; this +command generally returns the status of all negotiated contexts, +while some servers also support a filtered request (see +L<nbd_can_block_status_payload(3)>, L<nbd_block_status_filter(3)>).
Note that not all servers can support a C<count> of 4GiB or larger; L<nbd_get_extended_headers_negotiated(3)> indicates which servers @@ -2993,11 +3001,38 @@ "block_status_64", { does not exceed C<count> bytes; however, libnbd does not validate that the server obeyed the flag." ^ strict_call_description; - see_also = [Link "block_status"; + see_also = [Link "block_status"; Link "block_status_filter"; Link "add_meta_context"; Link "can_meta_context"; Link "aio_block_status_64"; Link "set_strict_mode"]; };
+ "block_status_filter", { + default_call with + args = [ UInt64 "count"; UInt64 "offset"; StringList "contexts"; + Closure extent64_closure ]; + optargs = [ OFlags ("flags", cmd_flags, Some ["REQ_ONE"; "PAYLOAD_LEN"]) ]; + ret = RErr; + permitted_states = [ Connected ]; + shortdesc = "send filtered block status command, with 64-bit callback"; + longdesc = "\ +Issue a filtered block status command to the NBD server. If +supported by the server (see L<nbd_can_block_status_payload(3)>), +this causes metadata context information about blocks beginning +from the specified offset to be returned, and with the result +limited to just the contexts specified in C<filter>. Note that +all strings in C<filter> must be supported by +L<nbd_can_meta_context(3)>. + +All other parameters to this function have the same semantics +as in L<nbd_block_status_64(3)>; except that for convenience, +the C<flags> parameter may additionally contain or omit +C<LIBNBD_CMD_FLAG_PAYLOAD_LEN>." +^ strict_call_description; + see_also = [Link "block_status_64"; + Link "can_block_status_payload"; Link "can_meta_context"; + Link "aio_block_status_filter"; Link "set_strict_mode"]; + }; + "poll", { default_call with args = [ Int "timeout" ]; ret = RInt; @@ -3667,6 +3702,30 @@ "aio_block_status_64", { Link "set_strict_mode"]; };
+ "aio_block_status_filter", { + default_call with + args = [ UInt64 "count"; UInt64 "offset"; StringList "contexts"; + Closure extent64_closure ]; + optargs = [ OClosure completion_closure; + OFlags ("flags", cmd_flags, Some ["REQ_ONE"; "PAYLOAD_LEN"]) ]; + ret = RCookie; + permitted_states = [ Connected ]; + shortdesc = "send filtered block status command to the NBD server"; + longdesc = "\ +Send a filtered block status command to the NBD server. + +To check if the command completed, call L<nbd_aio_command_completed(3)>. +Or supply the optional C<completion_callback> which will be invoked +as described in L<libnbd(3)/Completion callbacks>. + +Other parameters behave as documented in L<nbd_block_status_filter(3)>." +^ strict_call_description; + see_also = [SectionLink "Issuing asynchronous commands"; + Link "aio_block_status_64"; Link "block_status_filter"; + Link "can_meta_context"; Link "can_block_status_payload"; + Link "set_strict_mode"]; + }; + "aio_get_fd", { default_call with args = []; ret = RFd; @@ -4201,6 +4260,8 @@ let first_version = "opt_extended_headers", (1, 18); "aio_opt_extended_headers", (1, 18); "can_block_status_payload", (1, 18); + "block_status_filter", (1, 18); + "aio_block_status_filter", (1, 18);
(* These calls are proposed for a future version of libnbd, but * have not been added to any released version so far. diff --git a/generator/states-issue-command.c b/generator/states-issue-command.c index 79136b61..5307731d 100644 --- a/generator/states-issue-command.c +++ b/generator/states-issue-command.c @@ -84,7 +84,9 @@ ISSUE_COMMAND.PREPARE_WRITE_PAYLOAD: assert (h->cmds_to_issue != NULL); cmd = h->cmds_to_issue; assert (cmd->cookie == be64toh (h->req.compact.handle)); - if (cmd->type == NBD_CMD_WRITE) { + if (cmd->type == NBD_CMD_WRITE || + (h->extended_headers && cmd->type == NBD_CMD_BLOCK_STATUS && + cmd->flags & NBD_CMD_FLAG_PAYLOAD_LEN)) { h->wbuf = cmd->data; h->wlen = cmd->count; if (cmd->next && cmd->count < 64 * 1024) diff --git a/lib/aio.c b/lib/aio.c index a419ac32..77b20c32 100644 --- a/lib/aio.c +++ b/lib/aio.c @@ -32,8 +32,13 @@ void nbd_internal_retire_and_free_command (struct command *cmd) { /* Free the callbacks. */ - if (cmd->type == NBD_CMD_BLOCK_STATUS) + if (cmd->type == NBD_CMD_BLOCK_STATUS) { + if (cmd->ids) { + uint32_vector_reset (cmd->ids); + free (cmd->ids); + } FREE_CALLBACK (cmd->cb.fn.extent); + } if (cmd->type == NBD_CMD_READ) FREE_CALLBACK (cmd->cb.fn.chunk); FREE_CALLBACK (cmd->cb.completion); diff --git a/lib/rw.c b/lib/rw.c index bea55fa1..db6bc0bc 100644 --- a/lib/rw.c +++ b/lib/rw.c @@ -242,6 +242,26 @@ nbd_unlocked_block_status_64 (struct nbd_handle *h, return wait_for_command (h, cookie); }
+/* Issue a filtered block status command and wait for the reply. */ +int +nbd_unlocked_block_status_filter (struct nbd_handle *h, + uint64_t count, uint64_t offset, + char **filter, + nbd_extent64_callback *extent64, + uint32_t flags) +{ + int64_t cookie; + nbd_completion_callback c = NBD_NULL_COMPLETION; + + cookie = nbd_unlocked_aio_block_status_filter (h, count, offset, filter, + extent64, &c, flags); + if (cookie == -1) + return -1; + + assert (CALLBACK_IS_NULL (*extent64)); + return wait_for_command (h, cookie); +} + /* count_err represents the errno to return if bounds check fail */ int64_t nbd_internal_command_common (struct nbd_handle *h, @@ -250,6 +270,7 @@ nbd_internal_command_common (struct nbd_handle *h, void *data, struct command_cb *cb) { struct command *cmd; + uint32_vector *ids = NULL;
if (h->disconnect_request) { set_error (EINVAL, "cannot request more commands after NBD_CMD_DISC"); @@ -297,10 +318,23 @@ nbd_internal_command_common (struct nbd_handle *h, } break;
+ case NBD_CMD_BLOCK_STATUS: + if (data) { + ids = data; + count = ids->len * sizeof (uint32_t); + data = ids->ptr; + if (count > MAX_REQUEST_SIZE || + (h->strict & LIBNBD_STRICT_PAYLOAD && count > h->payload_maximum)) { + set_error (ERANGE, "filter set too large"); + goto err; + } + break; + } + /* fallthrough */ + default: /* Other commands are limited by the 32 bit field in the command * structure on the wire, unless extended headers were negotiated. */ - default: if (!h->extended_headers && count > UINT32_MAX) { set_error (ERANGE, "request too large: maximum request size is %" PRIu32, UINT32_MAX); @@ -320,6 +354,7 @@ nbd_internal_command_common (struct nbd_handle *h, cmd->offset = offset; cmd->count = count; cmd->data = data; + cmd->ids = ids; if (cb) cmd->cb = *cb;
@@ -364,8 +399,13 @@ nbd_internal_command_common (struct nbd_handle *h, err: /* Since we did not queue the command, we must free the callbacks. */ if (cb) { - if (type == NBD_CMD_BLOCK_STATUS) + if (type == NBD_CMD_BLOCK_STATUS) { + if (ids) { + uint32_vector_reset (ids); + free (ids); + } FREE_CALLBACK (cb->fn.extent); + } if (type == NBD_CMD_READ) FREE_CALLBACK (cb->fn.chunk); FREE_CALLBACK (cb->completion); @@ -609,3 +649,86 @@ nbd_unlocked_aio_block_status_64 (struct nbd_handle *h, return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, count, EINVAL, NULL, &cb); } + +int64_t +nbd_unlocked_aio_block_status_filter (struct nbd_handle *h, + uint64_t count, uint64_t offset, + char **filter, + nbd_extent64_callback *extent64, + nbd_completion_callback *completion, + uint32_t flags) +{ + struct command_cb cb = { .fn.extent = *extent64, + .completion = *completion }; + uint32_vector *ids; + char *name; + size_t i; + + /* Because this affects wire format, it is more convenient to manage + * PAYLOAD_LEN by what was negotiated than to require the user to + * have to set it correctly. + */ + if (!h->extended_headers) { + set_error (ENOTSUP, "server does not support extended headers"); + return -1; + } + flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN; + + if (h->strict & LIBNBD_STRICT_COMMANDS) { + if (nbd_unlocked_can_block_status_payload (h) != 1) { + set_error (EINVAL, + "server does not support the block status payload flag"); + return -1; + } + + if (!h->meta_valid || h->meta_contexts.len == 0) { + set_error (ENOTSUP, "did not negotiate any metadata contexts, " + "either you did not call nbd_add_meta_context before " + "connecting or the server does not support it"); + return -1; + } + } + + ids = calloc (1, sizeof *ids); + if (ids == NULL) { + set_error (errno, "calloc"); + return -1; + } + if (uint32_vector_append (ids, htobe32 (count >> 32)) == -1 || + uint32_vector_append (ids, htobe32 (count)) == -1) { + set_error (errno, "realloc"); + goto fail; + } + + /* O(n^2) search - hopefully filter and negotiated contexts are both small */
This is (sort of) remotely exploitable? If the server sends back an insane number of contexts? I'm not sure. But I wonder if it would work to keep the context strings sorted, then sort the filter strings, which could reduce this loop to O(n)-ish (apart from the sorts).
+ for ( ; (name = *filter) != NULL; filter++) { + if (!h->meta_valid) { + set_error (EINVAL, "context %s not negotiated", name); + goto fail; + } + for (i = 0; i < h->meta_contexts.len; i++) { + struct meta_context *meta = &h->meta_contexts.ptr[i]; + if (strcmp (name, meta->name) == 0) { + if (uint32_vector_append (ids, htobe32 (meta->context_id)) == -1) { + set_error (errno, "realloc"); + goto fail; + } + break; + } + } + if (i == h->meta_contexts.len) { + set_error (EINVAL, "context %s not negotiated", name); + goto fail; + } + } + + SET_CALLBACK_TO_NULL (*extent64); + SET_CALLBACK_TO_NULL (*completion); + return nbd_internal_command_common (h, flags, NBD_CMD_BLOCK_STATUS, offset, + count, EINVAL, ids, &cb); + + fail: + uint32_vector_reset (ids); + free (ids); + return -1; +} diff --git a/interop/block-status-payload.c b/interop/block-status-payload.c index 9603dfe5..704b25aa 100644 --- a/interop/block-status-payload.c +++ b/interop/block-status-payload.c @@ -54,11 +54,26 @@ cb (void *opaque, const char *metacontext, uint64_t offset, return 0; }
+static char ** +list (unsigned int use) +{ + static const char *array[ARRAY_SIZE (contexts) + 1]; + size_t i, j; + + assert (use < 1 << ARRAY_SIZE (contexts)); + for (i = j = 0; i < ARRAY_SIZE (contexts); i++) + if (use & (1 << i)) + array[j++] = contexts[i]; + array[j] = NULL; + return (char **) array; +} + int main (int argc, char *argv[]) { struct nbd_handle *nbd; int64_t exportsize; + uint64_t bytes_sent; unsigned int seen; size_t i; int r; @@ -114,7 +129,107 @@ main (int argc, char *argv[]) } assert (seen == 0xf);
- /* FIXME: Test filtered calls once the API is added */ + /* Filtering with all contexts listed, same effect as unfilitered call */ + seen = 0; + if (nbd_block_status_filter (nbd, exportsize, 0, list (0xf), + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (seen == 0xf); + + /* Filtering with just two out of four contexts; test optional flag */ + seen = 0; + if (nbd_block_status_filter (nbd, exportsize, 0, list (0x5), + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (seen == 0x5); + + /* Filtering with one context, near end of file (to make sure the + * payload length isn't confused with the effect length) + */ + seen = 0; + if (nbd_block_status_filter (nbd, 1, exportsize - 1, list (0x2), + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + assert (seen == 0x2); + + /* Filtering with no contexts - pointless, so qemu rejects it */ + bytes_sent = nbd_stats_bytes_sent (nbd); + seen = 0; + if (nbd_block_status_filter (nbd, exportsize, 0, list (0x0), + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) != -1) { + fprintf (stderr, "expecting block status failure\n"); + exit (EXIT_FAILURE); + } + assert (seen == 0x0); + if (nbd_get_errno () != EINVAL) { + fprintf (stderr, "expecting EINVAL after block status failure\n"); + exit (EXIT_FAILURE); + } + if (nbd_stats_bytes_sent (nbd) <= bytes_sent) { + fprintf (stderr, "expecting server-side rejection of bad request\n"); + exit (EXIT_FAILURE); + } + + /* Giving unknown string triggers EINVAL from libnbd */ + bytes_sent = nbd_stats_bytes_sent (nbd); + seen = 0; + { + const char *bogus[] = { "qemu:dirty-bitmap:bitmap2", NULL }; + if (nbd_block_status_filter (nbd, exportsize, 0, (char **) bogus, + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) != -1) { + fprintf (stderr, "expecting block status failure\n"); + exit (EXIT_FAILURE); + } + } + if (nbd_get_errno () != EINVAL) { + fprintf (stderr, "expecting EINVAL after block status failure\n"); + exit (EXIT_FAILURE); + } + assert (seen == 0x0); + if (nbd_stats_bytes_sent (nbd) != bytes_sent) { + fprintf (stderr, "expecting client-side rejection of bad request\n"); + exit (EXIT_FAILURE); + } + + /* Giving same string twice triggers EINVAL from qemu */ + seen = 0; + { + const char *dupes[] = { "base:allocation", "base:allocation", NULL }; + if (nbd_block_status_filter (nbd, exportsize, 0, (char **) dupes, + (nbd_extent64_callback) { .callback = cb, + .user_data = &seen }, + 0) != -1) { + fprintf (stderr, "expecting block status failure\n"); + exit (EXIT_FAILURE); + } + } + if (nbd_get_errno () != EINVAL) { + fprintf (stderr, "expecting EINVAL after block status failure\n"); + exit (EXIT_FAILURE); + } + assert (seen == 0x0); + if (nbd_stats_bytes_sent (nbd) <= bytes_sent) { + fprintf (stderr, "expecting server-side rejection of bad request\n"); + exit (EXIT_FAILURE); + } + + /* Done */ if (nbd_shutdown (nbd, 0) == -1) { fprintf (stderr, "%s\n", nbd_get_error ()); exit (EXIT_FAILURE); diff --git a/interop/block-status-payload.sh b/interop/block-status-payload.sh index a12cfc8a..0e6681b6 100755 --- a/interop/block-status-payload.sh +++ b/interop/block-status-payload.sh @@ -49,6 +49,7 @@ args = ["qemu-nbd", "-f", "qcow2", "-A", "-B", "bitmap0", "-B", "bitmap1", h.connect_systemd_socket_activation(args) assert h.aio_is_negotiating() is True assert h.get_extended_headers_negotiated() is False + # Flag not available until info or go try: h.can_block_status_payload() @@ -58,7 +59,18 @@ except nbd.Error: h.opt_info() assert h.can_block_status_payload() is False assert h.can_meta_context("base:allocation") is True -h.opt_abort() + +# Filter request not allowed if not advertised +def f(): + assert False +h.opt_go() +assert h.can_block_status_payload() is False +try: + h.block_status_filter(0, 512, ["base:allocation"], f) + assert False +except nbd.Error: + pass +h.shutdown() '
# Conditional part of test: if qemu is new enough to support extended diff --git a/info/info-can.sh b/info/info-can.sh index 8154d1ce..097837d2 100755 --- a/info/info-can.sh +++ b/info/info-can.sh @@ -38,6 +38,9 @@ requires bash -c "nbdkit sh --dump-plugin | grep has_can_cache=1" # and oldstyle never, but that feels like depending a bit too much on # the implementation.
+# --can block-status-payload is not supported by nbdkit yet. Testing +# is done during interop with new-enough qemu. + # --can structured-reply is not a per-export setting, but rather # something set on the server as a whole.
Seems generally OK, so: Acked-by: Richard W.M. Jones <rjones@redhat.com> Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Sorry it took me so long to get around to this one. Hopefully the next version will need only cursory approval. I did read all of the patches, and I basically agree with Laszlo on the ones he reviewed already. Therefore I didn't add any comment except where necessary. You can assume Acked-by on those patches. Thanks! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
participants (4)
-
Eric Blake -
Laszlo Ersek -
Richard W.M. Jones -
Wouter Verhelst