[libnbd PATCH 0/4] Work towards fixing nbdcopy silent corruption
These are the trivial patches modeled after Nir's commit 7ba6ef67, fixing further places that fail to check *error during callbacks. nbdcopy still has a bigger bug in that it is NOT properly checking for *error in the pread/pwrite completion handlers, but fixing that gracefully (rather than just quitting the process immediately, as we can do in the examples), requires more invasive patches which I will be posting separately for review, while the ones posted here are trivial enough that I'm pushing them now (commits 23bcea4a..1373d423). I also audited other uses of completion callbacks: - in tests, closure-lifetimes, errors, server-death, and shutdown-flags are all safe (either the callback returns 0 and the code later calls nbd_aio_command_completed, or we are explicitly testing aspects of completion callbacks) - in fuse/operations.c, the completion callback returns 0 and calls nbd_aio_command_completed later - in examples/strict-structured-reads.c, the completion callback properly checks *error I also checked that nbdkit's nbd plugin properly checks *error. Eric Blake (4): examples: aio-connect-read.c: Fix error handling examples: glib-main-loop: don't strand commands when gsource disappears examples: glib-main-loop: Fix error handling copy: Ignore extents if we encountered earlier error examples/aio-connect-read.c | 7 +++++++ examples/glib-main-loop.c | 14 ++++++++++++-- copy/nbd-ops.c | 4 ++-- 3 files changed, 21 insertions(+), 4 deletions(-) -- 2.34.1
This example failed to check the *error parameter to the read completion callback. Although the buffers in this example start life zero-initialized due to living in .bss (so there is no leak of heap contents), later reads could repeat the decoding of a buffer returned from an earlier read rather than reporting the failure. Fixed by reporting the error and exiting early. Here is an example failure fixed by this patch: $ ./run nbdkit -U - eval get_size='echo 64k' pread='echo EIO >&2; exit 1' \ --run 'examples/aio-connect-read $unixsocket' nbdkit: eval[1]: error: /tmp/nbdkit5kfDY2/pread: failed to read: Input/output error nbdkit: eval[1]: error: /tmp/nbdkit5kfDY2/pread: nbdkit: eval[1]: error: write error reply: Broken pipe $ echo $? 1 which previously dumped 64k of all zeroes before exiting with status 0. --- examples/aio-connect-read.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/examples/aio-connect-read.c b/examples/aio-connect-read.c index 799525ad..26ac3f99 100644 --- a/examples/aio-connect-read.c +++ b/examples/aio-connect-read.c @@ -17,6 +17,7 @@ #include <stdlib.h> #include <stdint.h> #include <inttypes.h> +#include <errno.h> #include <assert.h> #include <libnbd.h> @@ -35,6 +36,12 @@ hexdump (void *user_data, int *error) struct data *data = user_data; FILE *pp; + if (*error) { + errno = *error; + perror ("failed to read"); + exit (EXIT_FAILURE); + } + printf ("sector at offset 0x%" PRIx64 ":\n", data->offset); pp = popen ("hexdump -C", "w"); -- 2.34.1
Returning 0 from a callback handler implies that we will later call nbd_aio_command_completed(), but we weren't doing that, and our finalize() handler instead asserts that no outstanding commands remain. Even when we are going to ignore a late completion (due to the gsource already being torn down), we should still auto-retire the command. Fixes: 7e53d48e ("api: Allow completion callbacks to auto-retire", v0.1.9) --- examples/glib-main-loop.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/glib-main-loop.c b/examples/glib-main-loop.c index 9c279d39..2df27878 100644 --- a/examples/glib-main-loop.c +++ b/examples/glib-main-loop.c @@ -400,7 +400,7 @@ finished_read (void *vp, int *error) struct buffer *buffer = vp; if (gssrc == NULL) - return 0; + return 1; /* Nothing we can do, auto-retire the callback */ DEBUG (gssrc, "finished_read: read completed"); @@ -446,7 +446,7 @@ finished_write (void *vp, int *error) struct buffer *buffer = vp; if (gsdest == NULL) - return 0; + return 1; /* Nothing we can do, auto-retire the callback */ DEBUG (gsdest, "finished_write: write completed"); -- 2.34.1
This example failed to check the *error parameter to the read and write completion callbacks. What's more, the buffers start life uninitialized, so a failed read can end up leaking heap contents to the corresponding write. Fortunately, the example program is hard-coded to two particular nbdkit invocations, and not usable to corrupt a user's data. However, as this example is likely to be copied elsewhere, we should at least perform rudimentary error checking to avoid bad copy-and-paste that could turn into a more severe problem in code derived from this. Easiest for the example is to just punt and exit the process on any detected errors from either the source or destination. Since the nbdkit invocations are hard-coded, testing that this patch works involves using a temporary patch, such as: | diff --git i/examples/glib-main-loop.c w/examples/glib-main-loop.c | index 1982f941..f60a4a38 100644 | --- i/examples/glib-main-loop.c | +++ w/examples/glib-main-loop.c | @@ -232,7 +232,8 @@ static struct NBDSource *gssrc, *gsdest; | #define SIZE (1024*1024*1024) | | static const char *src_args[] = { | - "nbdkit", "-s", "--exit-with-parent", "-r", "pattern", "size=1G", NULL | + "nbdkit", "-s", "--exit-with-parent", "-r", "--filter=error", | + "pattern", "size=1G", "error-pread-rate=0.1", NULL | }; | | static const char *dest_args[] = { Fixes: bc3b4230 ("examples: Include an example of integrating with the glib main loop", v0.1.9) --- examples/glib-main-loop.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/examples/glib-main-loop.c b/examples/glib-main-loop.c index 2df27878..1982f941 100644 --- a/examples/glib-main-loop.c +++ b/examples/glib-main-loop.c @@ -402,6 +402,11 @@ finished_read (void *vp, int *error) if (gssrc == NULL) return 1; /* Nothing we can do, auto-retire the callback */ + if (*error) { + fprintf (stderr, "finished_read: read failed: %s\n", strerror (*error)); + exit (EXIT_FAILURE); + } + DEBUG (gssrc, "finished_read: read completed"); assert (buffer->state == BUFFER_READING); @@ -448,6 +453,11 @@ finished_write (void *vp, int *error) if (gsdest == NULL) return 1; /* Nothing we can do, auto-retire the callback */ + if (*error) { + fprintf (stderr, "finished_write: write failed: %s\n", strerror (*error)); + exit (EXIT_FAILURE); + } + DEBUG (gsdest, "finished_write: write completed"); assert (buffer->state == BUFFER_WRITING); -- 2.34.1
On 02/01/22 20:44, Eric Blake wrote:
This example failed to check the *error parameter to the read and write completion callbacks. What's more, the buffers start life uninitialized, so a failed read can end up leaking heap contents to the corresponding write.
Fortunately, the example program is hard-coded to two particular nbdkit invocations, and not usable to corrupt a user's data. However, as this example is likely to be copied elsewhere, we should at least perform rudimentary error checking to avoid bad copy-and-paste that could turn into a more severe problem in code derived from this. Easiest for the example is to just punt and exit the process on any detected errors from either the source or destination.
Since the nbdkit invocations are hard-coded, testing that this patch works involves using a temporary patch, such as:
| diff --git i/examples/glib-main-loop.c w/examples/glib-main-loop.c | index 1982f941..f60a4a38 100644 | --- i/examples/glib-main-loop.c | +++ w/examples/glib-main-loop.c | @@ -232,7 +232,8 @@ static struct NBDSource *gssrc, *gsdest; | #define SIZE (1024*1024*1024) | | static const char *src_args[] = { | - "nbdkit", "-s", "--exit-with-parent", "-r", "pattern", "size=1G", NULL | + "nbdkit", "-s", "--exit-with-parent", "-r", "--filter=error", | + "pattern", "size=1G", "error-pread-rate=0.1", NULL | }; | | static const char *dest_args[] = {
Fixes: bc3b4230 ("examples: Include an example of integrating with the glib main loop", v0.1.9) --- examples/glib-main-loop.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/examples/glib-main-loop.c b/examples/glib-main-loop.c index 2df27878..1982f941 100644 --- a/examples/glib-main-loop.c +++ b/examples/glib-main-loop.c @@ -402,6 +402,11 @@ finished_read (void *vp, int *error) if (gssrc == NULL) return 1; /* Nothing we can do, auto-retire the callback */
+ if (*error) { + fprintf (stderr, "finished_read: read failed: %s\n", strerror (*error)); + exit (EXIT_FAILURE); + } + DEBUG (gssrc, "finished_read: read completed");
assert (buffer->state == BUFFER_READING); @@ -448,6 +453,11 @@ finished_write (void *vp, int *error) if (gsdest == NULL) return 1; /* Nothing we can do, auto-retire the callback */
+ if (*error) { + fprintf (stderr, "finished_write: write failed: %s\n", strerror (*error)); + exit (EXIT_FAILURE); + } + DEBUG (gsdest, "finished_write: write completed");
assert (buffer->state == BUFFER_WRITING);
Slightly inconsistent with + errno = *error; + perror ("failed to read"); + exit (EXIT_FAILURE); from the first patch, but does effectively the same thing. Thanks, Laszlo
In practice, the nbd extents callback will only be called once per nbd_aio_block_status, because we have only requested exactly one metacontext id, and because compliant servers send only one reply chunk per id. In theory, a server could reply first with an error chunk (perhaps meaning "I'm unable to get status beyond offset X") followed by an extent chunk ("but here's what I was able to get") while still being compliant. And a non-compliant server can send unexpected chunks, just to try and confuse us. Right now, it would take a custom NBD server to actually trigger any scenario where *error would ever be non-zero on entry to next_extent() (nbdkit does not have that much power). So while *error ever being non-zero is unlikely, our easiest course is to handle it the same way we would for a server sending us an unexpected id: ignore the rest of the extent callback. The later completion callback will handle the overall command failing. --- copy/nbd-ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/copy/nbd-ops.c b/copy/nbd-ops.c index dfc62f20..ab30badd 100644 --- a/copy/nbd-ops.c +++ b/copy/nbd-ops.c @@ -1,5 +1,5 @@ /* NBD client library in userspace. - * Copyright (C) 2020 Red Hat Inc. + * Copyright (C) 2020-2022 Red Hat Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -337,7 +337,7 @@ add_extent (void *vp, const char *metacontext, extent_list *ret = vp; size_t i; - if (strcmp (metacontext, "base:allocation") != 0) + if (strcmp (metacontext, "base:allocation") != 0 || *error) return 0; for (i = 0; i < nr_entries; i += 2) { -- 2.34.1
On 02/01/22 20:44, Eric Blake wrote:
In practice, the nbd extents callback will only be called once per nbd_aio_block_status, because we have only requested exactly one metacontext id, and because compliant servers send only one reply chunk per id. In theory, a server could reply first with an error chunk (perhaps meaning "I'm unable to get status beyond offset X") followed by an extent chunk ("but here's what I was able to get") while still being compliant. And a non-compliant server can send unexpected chunks, just to try and confuse us. Right now, it would take a custom NBD server to actually trigger any scenario where *error would ever be non-zero on entry to next_extent() (nbdkit does not have that much power). So while *error ever being non-zero is unlikely, our easiest course is to handle it the same way we would for a server sending us an unexpected id: ignore the rest of the extent callback. The later completion callback will handle the overall command failing.
This commit message mostly answers my question under the cover letter, and indeed it's quite the opposite of what I expected. I'd have expected the command (the exchange with the server) completing, upon which time the completion callback would have run; then the results (a series of extent callbacks) would be passed to the extent callback. So, that's precisely *not* the case in reality. :/
--- copy/nbd-ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/copy/nbd-ops.c b/copy/nbd-ops.c index dfc62f20..ab30badd 100644 --- a/copy/nbd-ops.c +++ b/copy/nbd-ops.c @@ -1,5 +1,5 @@ /* NBD client library in userspace. - * Copyright (C) 2020 Red Hat Inc. + * Copyright (C) 2020-2022 Red Hat Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -337,7 +337,7 @@ add_extent (void *vp, const char *metacontext, extent_list *ret = vp; size_t i;
- if (strcmp (metacontext, "base:allocation") != 0) + if (strcmp (metacontext, "base:allocation") != 0 || *error) return 0;
for (i = 0; i < nr_entries; i += 2) {
So, I don't understand this. First, here we have a non-aio (i.e., synchronous) nbd_block_status call. in case add_extent (the extents callback) is entered with *error !=0, that means (per docs) "the error parameter contains the errno value of any previously detected error". In case we responded by returning -1, then the following (per docs) would apply: "If the callback returns -1, and no earlier error has been detected, then the overall block status command will fail with any non-zero value stored into the callback's error parameter" But we don't do that, we say "everything is OK, I'll just ignore this", like we ignore metadata contexts that we don't need. So how is the "later completion callback" going to fail the entire nbd_block_status() call? First, because this is not an AIO nbd_block_status() call, we don't provide the completion callback ourselves -- I guess it is some default function somewhere in libnbd. Or is it the following: upon getting (*error != 0) on entry, we just decline looking at the (likely nonsensical) "entries" array, and permit the error we got via *error to "survive" through all further add_extent() calls, and to finally fail the outermost nbd_block_status() call? Next question: even if that's the case, should we *first* check *error, and only check "metacontext" if *error is zero? What says that "metacontext" is defined if *error is nonzero? Thanks, Laszlo
On Wed, Feb 02, 2022 at 09:42:46AM +0100, Laszlo Ersek wrote:
On 02/01/22 20:44, Eric Blake wrote:
In practice, the nbd extents callback will only be called once per nbd_aio_block_status, because we have only requested exactly one metacontext id, and because compliant servers send only one reply chunk per id. In theory, a server could reply first with an error chunk (perhaps meaning "I'm unable to get status beyond offset X") followed by an extent chunk ("but here's what I was able to get") while still being compliant. And a non-compliant server can send unexpected chunks, just to try and confuse us. Right now, it would take a custom NBD server to actually trigger any scenario where *error would ever be non-zero on entry to next_extent() (nbdkit does not have that much power). So while *error ever being non-zero is unlikely, our easiest course is to handle it the same way we would for a server sending us an unexpected id: ignore the rest of the extent callback. The later completion callback will handle the overall command failing.
This commit message mostly answers my question under the cover letter, and indeed it's quite the opposite of what I expected. I'd have expected the command (the exchange with the server) completing, upon which time the completion callback would have run; then the results (a series of extent callbacks) would be passed to the extent callback.
So, that's precisely *not* the case in reality. :/
Yeah, understanding the control flow is helpful. The first thing to note: every client command has a sticky error field (see struct command->error); this field starts life 0, and while the command is live, it is set to non-zero in the following scenarios: if the client never sends the command to the server (for example, settings from nbd_set_strict_mode() determined it would be an out-of-bounds request); when the server replies with NBD_REPLY_TYPE_ERROR in a structured reply (although the command may still be live if it wasn't the final reply chunk); when the server replies with an error in a simple reply (with simple replies, the command is necessarily complete); or when a client mid-command callback returns -1 (only pread_structured and block_status have these, and the command may still be live if it wasn't the final reply chunk). Once the command is complete, then we call the completion callback (if there is one) with *error set to the value of command->error, giving one more chance to change the value stored in command->error; and then nbd_aio_command_completed returns the value of command->error as the error status of the overall command (unless the completion callback returned 1 to auto-retire the command first). Then we have nbd_aio_block_status which kicks off a transaction that will eventually send NBD_CMD_BLOCK_STATUS, and is associated with TWO callbacks: the mandatory mid-command nbd_extents callback to read block status information as it comes over the wire from the server, and the optional nbd_completion callback to mark when the server has nothing more to send. The documentation states that the nbd_extents callback can be called 0 or more times (0 if the server replies with an error, or even if nbd_set_strict_mode() determines that the command violates the NBD protocol and is never even sent to the server). If you only have one metacontext id (as in nbdcopy only caring about base:allocation), one time is all the more we expect from a compliant server (a noncompliant server can send more than one reply, but we'd have to code up such a custom server, as nbdkit does not behave that way). The *error parameter passed to nbd_extents is non-zero only if there is a running error detected so far (generally unlikely; a server would have to send an error chunk before a status chunk, or send more than one status chunk where the libnbd client set *error during the first time the callback was reached). Then the nbd_completion callback is always called exactly once, after there is nothing further to be done with the command. The *error parameter passed to the nbd_completion callback is the same value that will be returned from nbd_aio_command_comppleted (unless the callback returns 1 for auto-retiring). And finally we have nbd_block_status: a synchronous thin wrapper that calls nbd_aio_block_status, nbd_aio_poll, nbd_aio_command_completed under the hood. The nbd_extents callback is still mandatory, and is called mid-command prior to the function call completing, in response to server information as it comes over the wire. The optional nbd_completion callback of nbd_aio_block_status is not exposed (the command completes synchronously, instead; whether we used the completion callback under the hood is an implementation detail that doesn't matter to the public interface). I'm open to a documentation patch that gives more clarity on this sequence of callbacks.
--- copy/nbd-ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/copy/nbd-ops.c b/copy/nbd-ops.c index dfc62f20..ab30badd 100644 --- a/copy/nbd-ops.c +++ b/copy/nbd-ops.c @@ -1,5 +1,5 @@ /* NBD client library in userspace. - * Copyright (C) 2020 Red Hat Inc. + * Copyright (C) 2020-2022 Red Hat Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -337,7 +337,7 @@ add_extent (void *vp, const char *metacontext, extent_list *ret = vp; size_t i;
- if (strcmp (metacontext, "base:allocation") != 0) + if (strcmp (metacontext, "base:allocation") != 0 || *error) return 0;
for (i = 0; i < nr_entries; i += 2) {
So, I don't understand this. First, here we have a non-aio (i.e., synchronous) nbd_block_status call. in case add_extent (the extents callback) is entered with *error !=0, that means (per docs) "the error parameter contains the errno value of any previously detected error".
I guess what the docs are missing is the disclaimer that if the mid-command callback is reached, there is usually no previously detected error. The only two situations where there is a previous error is if the server replies with NBD_REPLY_TYPE_ERROR first and then NBD_REPLY_TYPE_BLOCK_STATUS second (a custom server), or if the client negotiated more than one context and the first use of the callback assigns an error (by returning -1) before the second callback is reached. The other cases where an error could be detected never reach the mid-command callback (if nbd_set_strict_mode() prevented the command from ever going to the server, or if the server replied with an error as a final chunk).
In case we responded by returning -1, then the following (per docs) would apply:
"If the callback returns -1, and no earlier error has been detected, then the overall block status command will fail with any non-zero value stored into the callback's error parameter"
But we don't do that, we say "everything is OK, I'll just ignore this", like we ignore metadata contexts that we don't need.
Yes. But even though the mid-command callback ignores the error, the synchronous wrapper will still see the same error via nbd_aio_command_completed under the hood, so the overall nbd_blocK_status will fail with the same value.
So how is the "later completion callback" going to fail the entire nbd_block_status() call? First, because this is not an AIO nbd_block_status() call, we don't provide the completion callback ourselves -- I guess it is some default function somewhere in libnbd.
Right - the synchronous wrappers return the same value as would be present in *error given to the completion callback of an asynchronous API (whether the synchronous wrappers actually use the completion callback, or get at the value directly, is an implementation detail). And remember that the completion callback to the asynchronous APIs is optional; if it is not provided, that error value is visible through nbd_aio_command_completed.
Or is it the following: upon getting (*error != 0) on entry, we just decline looking at the (likely nonsensical) "entries" array, and permit the error we got via *error to "survive" through all further add_extent() calls, and to finally fail the outermost nbd_block_status() call?
Yes, that is the best way of looking at it. In fact, the "entries" array is most likely valid, rather than non-sensical, but since we know that our response to a failing nbd_block_status() is to die, there's no point wasting time processing the valid extents to prolong the time before we die. In other words, this patch is optional; it does not add safety (as the error will resurface as the result of nbd_block_status, where we will fail), but merely speed (since we are going to fail anyways, why waste the time getting to that failure).
Next question: even if that's the case, should we *first* check *error, and only check "metacontext" if *error is zero? What says that "metacontext" is defined if *error is nonzero?
The nbd_extents callback is only reached any time the server replies with NBD_REPLY_TYPE_BLOCK_STATUS with a metacontext id that we recognized. If we did not recognize the id, or if the server never replies with that chunk, then we don't call the mid-command callback in the first place. So the "metacontext" and "extents" parameters to nbd_extents are always valid, even if *error is nonzero. And now that I've written this email, I wish I could incorporate some of it into the commit message, even though it is already pushed. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 02/02/22 15:54, Eric Blake wrote:
On Wed, Feb 02, 2022 at 09:42:46AM +0100, Laszlo Ersek wrote:
On 02/01/22 20:44, Eric Blake wrote:
In practice, the nbd extents callback will only be called once per nbd_aio_block_status, because we have only requested exactly one metacontext id, and because compliant servers send only one reply chunk per id. In theory, a server could reply first with an error chunk (perhaps meaning "I'm unable to get status beyond offset X") followed by an extent chunk ("but here's what I was able to get") while still being compliant. And a non-compliant server can send unexpected chunks, just to try and confuse us. Right now, it would take a custom NBD server to actually trigger any scenario where *error would ever be non-zero on entry to next_extent() (nbdkit does not have that much power). So while *error ever being non-zero is unlikely, our easiest course is to handle it the same way we would for a server sending us an unexpected id: ignore the rest of the extent callback. The later completion callback will handle the overall command failing.
This commit message mostly answers my question under the cover letter, and indeed it's quite the opposite of what I expected. I'd have expected the command (the exchange with the server) completing, upon which time the completion callback would have run; then the results (a series of extent callbacks) would be passed to the extent callback.
So, that's precisely *not* the case in reality. :/
Yeah, understanding the control flow is helpful. The first thing to note: every client command has a sticky error field (see struct command->error); this field starts life 0, and while the command is live, it is set to non-zero in the following scenarios: if the client never sends the command to the server (for example, settings from nbd_set_strict_mode() determined it would be an out-of-bounds request); when the server replies with NBD_REPLY_TYPE_ERROR in a structured reply (although the command may still be live if it wasn't the final reply chunk); when the server replies with an error in a simple reply (with simple replies, the command is necessarily complete); or when a client mid-command callback returns -1 (only pread_structured and block_status have these, and the command may still be live if it wasn't the final reply chunk). Once the command is complete, then we call the completion callback (if there is one) with *error set to the value of command->error, giving one more chance to change the value stored in command->error; and then nbd_aio_command_completed returns the value of command->error as the error status of the overall command (unless the completion callback returned 1 to auto-retire the command first).
Then we have nbd_aio_block_status which kicks off a transaction that will eventually send NBD_CMD_BLOCK_STATUS, and is associated with TWO callbacks: the mandatory mid-command nbd_extents callback to read block status information as it comes over the wire from the server, and the optional nbd_completion callback to mark when the server has nothing more to send.
The documentation states that the nbd_extents callback can be called 0 or more times (0 if the server replies with an error, or even if nbd_set_strict_mode() determines that the command violates the NBD protocol and is never even sent to the server). If you only have one metacontext id (as in nbdcopy only caring about base:allocation), one time is all the more we expect from a compliant server (a noncompliant server can send more than one reply, but we'd have to code up such a custom server, as nbdkit does not behave that way). The *error parameter passed to nbd_extents is non-zero only if there is a running error detected so far (generally unlikely; a server would have to send an error chunk before a status chunk, or send more than one status chunk where the libnbd client set *error during the first time the callback was reached).
Then the nbd_completion callback is always called exactly once, after there is nothing further to be done with the command. The *error parameter passed to the nbd_completion callback is the same value that will be returned from nbd_aio_command_comppleted (unless the callback returns 1 for auto-retiring).
And finally we have nbd_block_status: a synchronous thin wrapper that calls nbd_aio_block_status, nbd_aio_poll, nbd_aio_command_completed under the hood. The nbd_extents callback is still mandatory, and is called mid-command prior to the function call completing, in response to server information as it comes over the wire. The optional nbd_completion callback of nbd_aio_block_status is not exposed (the command completes synchronously, instead; whether we used the completion callback under the hood is an implementation detail that doesn't matter to the public interface).
Thanks. This is also really helpful. And I've peeked at the code ("generator/states-reply-structured.c") and it seems like a callback has no way to "clear" an error that was made sticky earlier; good! (Upon second reading, Rich's email also stated the same.)
I'm open to a documentation patch that gives more clarity on this sequence of callbacks.
Haha, that's my line! :) As in, "please write this up in a documentation patch" :) But, I know how this works -- I've been on the answering side. So I thank you and Rich very much this info, and when I next need it, I will find it in my (indexed) email archive.
--- copy/nbd-ops.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/copy/nbd-ops.c b/copy/nbd-ops.c index dfc62f20..ab30badd 100644 --- a/copy/nbd-ops.c +++ b/copy/nbd-ops.c @@ -1,5 +1,5 @@ /* NBD client library in userspace. - * Copyright (C) 2020 Red Hat Inc. + * Copyright (C) 2020-2022 Red Hat Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -337,7 +337,7 @@ add_extent (void *vp, const char *metacontext, extent_list *ret = vp; size_t i;
- if (strcmp (metacontext, "base:allocation") != 0) + if (strcmp (metacontext, "base:allocation") != 0 || *error) return 0;
for (i = 0; i < nr_entries; i += 2) {
So, I don't understand this. First, here we have a non-aio (i.e., synchronous) nbd_block_status call. in case add_extent (the extents callback) is entered with *error !=0, that means (per docs) "the error parameter contains the errno value of any previously detected error".
I guess what the docs are missing is the disclaimer that if the mid-command callback is reached, there is usually no previously detected error. The only two situations where there is a previous error is if the server replies with NBD_REPLY_TYPE_ERROR first and then NBD_REPLY_TYPE_BLOCK_STATUS second (a custom server), or if the client negotiated more than one context and the first use of the callback assigns an error (by returning -1) before the second callback is reached. The other cases where an error could be detected never reach the mid-command callback (if nbd_set_strict_mode() prevented the command from ever going to the server, or if the server replied with an error as a final chunk).
OK! Clear now!
In case we responded by returning -1, then the following (per docs) would apply:
"If the callback returns -1, and no earlier error has been detected, then the overall block status command will fail with any non-zero value stored into the callback's error parameter"
But we don't do that, we say "everything is OK, I'll just ignore this", like we ignore metadata contexts that we don't need.
Yes. But even though the mid-command callback ignores the error, the synchronous wrapper will still see the same error via nbd_aio_command_completed under the hood, so the overall nbd_blocK_status will fail with the same value.
So how is the "later completion callback" going to fail the entire nbd_block_status() call? First, because this is not an AIO nbd_block_status() call, we don't provide the completion callback ourselves -- I guess it is some default function somewhere in libnbd.
Right - the synchronous wrappers return the same value as would be present in *error given to the completion callback of an asynchronous API (whether the synchronous wrappers actually use the completion callback, or get at the value directly, is an implementation detail). And remember that the completion callback to the asynchronous APIs is optional; if it is not provided, that error value is visible through nbd_aio_command_completed.
Or is it the following: upon getting (*error != 0) on entry, we just decline looking at the (likely nonsensical) "entries" array, and permit the error we got via *error to "survive" through all further add_extent() calls, and to finally fail the outermost nbd_block_status() call?
Yes, that is the best way of looking at it. In fact, the "entries" array is most likely valid, rather than non-sensical, but since we know that our response to a failing nbd_block_status() is to die, there's no point wasting time processing the valid extents to prolong the time before we die. In other words, this patch is optional; it does not add safety (as the error will resurface as the result of nbd_block_status, where we will fail), but merely speed (since we are going to fail anyways, why waste the time getting to that failure).
Next question: even if that's the case, should we *first* check *error, and only check "metacontext" if *error is zero? What says that "metacontext" is defined if *error is nonzero?
The nbd_extents callback is only reached any time the server replies with NBD_REPLY_TYPE_BLOCK_STATUS with a metacontext id that we recognized. If we did not recognize the id, or if the server never replies with that chunk, then we don't call the mid-command callback in the first place. So the "metacontext" and "extents" parameters to nbd_extents are always valid, even if *error is nonzero.
And now that I've written this email, I wish I could incorporate some of it into the commit message, even though it is already pushed.
Your response will stay in the mailing list archive :) (This is BTW one reason why I like to apply my own patches from the list, for pushing: with "git am -m" (and "-m" can be made the default config), Message-Id footers are created in the commit messages automatically. It's always nice to have a permanent link from the commit to the posting / thread!) Thank you! Laszlo
The patch series looks fine, with or without Laszlo's small suggested change. ACK Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On 02/01/22 20:44, Eric Blake wrote:
These are the trivial patches modeled after Nir's commit 7ba6ef67, fixing further places that fail to check *error during callbacks.
I find these interfaces very difficult to use. https://libguestfs.org/nbd_aio_block_status.3.html We have two callbacks here, each takes a *error, and I have zero idea of the order the callbacks are supposed to be called in. The last paragraph of the commit message speaks about this, and the code updates seem to suggest some ordering, but I'm still unsure. ... In fact, I'm now uncertain whether ignoring "err" in virt-v2v's "lib/utils.ml", function "get_disk_allocated", is safe. It's a "sync" (not aio) nbd_block_status() call, but the documentation <https://libguestfs.org/nbd_block_status.3.html> says: "On entry to the callback, the error parameter contains the errno value of any previously detected error." So what should we do then, in "get_disk_allocated"? Return -1 immediately? And if we do so, what is that going to mean for "NBD.block_status"? Will it raise an exception? Thanks, Laszlo
nbdcopy still has a bigger bug in that it is NOT properly checking for *error in the pread/pwrite completion handlers, but fixing that gracefully (rather than just quitting the process immediately, as we can do in the examples), requires more invasive patches which I will be posting separately for review, while the ones posted here are trivial enough that I'm pushing them now (commits 23bcea4a..1373d423).
I also audited other uses of completion callbacks: - in tests, closure-lifetimes, errors, server-death, and shutdown-flags are all safe (either the callback returns 0 and the code later calls nbd_aio_command_completed, or we are explicitly testing aspects of completion callbacks) - in fuse/operations.c, the completion callback returns 0 and calls nbd_aio_command_completed later - in examples/strict-structured-reads.c, the completion callback properly checks *error
I also checked that nbdkit's nbd plugin properly checks *error.
Eric Blake (4): examples: aio-connect-read.c: Fix error handling examples: glib-main-loop: don't strand commands when gsource disappears examples: glib-main-loop: Fix error handling copy: Ignore extents if we encountered earlier error
examples/aio-connect-read.c | 7 +++++++ examples/glib-main-loop.c | 14 ++++++++++++-- copy/nbd-ops.c | 4 ++-- 3 files changed, 21 insertions(+), 4 deletions(-)
On 02/02/22 09:23, Laszlo Ersek wrote:
On 02/01/22 20:44, Eric Blake wrote:
These are the trivial patches modeled after Nir's commit 7ba6ef67, fixing further places that fail to check *error during callbacks.
I find these interfaces very difficult to use.
https://libguestfs.org/nbd_aio_block_status.3.html
We have two callbacks here, each takes a *error, and I have zero idea of the order the callbacks are supposed to be called in.
The last paragraph of the commit message speaks about this, and the code updates seem to suggest some ordering, but I'm still unsure.
... In fact, I'm now uncertain whether ignoring "err" in virt-v2v's "lib/utils.ml", function "get_disk_allocated", is safe. It's a "sync" (not aio) nbd_block_status() call, but the documentation <https://libguestfs.org/nbd_block_status.3.html> says:
"On entry to the callback, the error parameter contains the errno value of any previously detected error."
So what should we do then, in "get_disk_allocated"? Return -1 immediately?
From patch#4, it seems that we need exactly this. Here's the function:
let get_disk_allocated ~dir ~disknr = let socket = sprintf "%s/out%d" dir disknr and alloc_ctx = "base:allocation" in with_nbd_connect_unix ~socket ~meta_contexts:[alloc_ctx] (fun nbd -> if NBD.can_meta_context nbd alloc_ctx then ( (* Get the list of extents, using a 2GiB chunk size as hint. *) let size = NBD.get_size nbd and allocated = ref 0_L and fetch_offset = ref 0_L in while !fetch_offset < size do let remaining = size -^ !fetch_offset in let fetch_size = min 0x8000_0000_L remaining in NBD.block_status nbd fetch_size !fetch_offset (fun ctx offset entries err -> assert (ctx = alloc_ctx); for i = 0 to Array.length entries / 2 - 1 do let len = entries.(i * 2) and typ = entries.(i * 2 + 1) in assert (len > 0_L); if typ &^ 1_L = 0_L then allocated := !allocated +^ len; fetch_offset := !fetch_offset +^ len done; 0 ) done; Some !allocated ) else None ) If our (nameless) function is entered with a nonzero "err", then we cannot trust the "entries" array. Which means not only "allocated" may be miscalculated, but even "fetch_offset" -- and the latter can as well break the chunking (= outer) loop. This seems to imply we *must not* continue the outer loop. So how do we handle this in OCaml? If we return -1 from the (nameless) extents callback, will NBD.block_status raise an exception? Hmmmm: maybe. In the generated nbd_internal_ocaml_nbd_block_status() function, in file "ocaml/nbd-c.c", we have: caml_enter_blocking_section (); r = nbd_block_status (h, count, offset, extent_callback, flags); caml_leave_blocking_section (); if (r == -1) nbd_internal_ocaml_raise_error (); However, in extent_wrapper() -> extent_wrapper_locked(), there is more confusion for me: metacontextv = caml_copy_string (metacontext); offsetv = caml_copy_int64 (offset); entriesv = nbd_internal_ocaml_alloc_int64_from_uint32_array (entries, nr_entries); errorv = caml_alloc_tuple (1); Store_field (errorv, 0, Val_int (*error)); args[0] = metacontextv; args[1] = offsetv; args[2] = entriesv; args[3] = errorv; rv = caml_callbackN_exn (data->fnv, 4, args); *error = Int_val (Field (errorv, 0)); if (Is_exception_result (rv)) { nbd_internal_ocaml_exception_in_wrapper ("extent", rv); CAMLreturnT (int, -1); } r = Int_val (rv); assert (r >= 0); CAMLreturnT (int, r); If I understand correctly, if we returned (-1) from the (nameless) extents callback, that would trip the assert(); would it not? Thus, we would not reach the nbd_internal_ocaml_raise_error() in the above-quoted nbd_internal_ocaml_nbd_block_status() function. Thanks, Laszlo
And if we do so, what is that going to mean for "NBD.block_status"? Will it raise an exception?
Thanks, Laszlo
nbdcopy still has a bigger bug in that it is NOT properly checking for *error in the pread/pwrite completion handlers, but fixing that gracefully (rather than just quitting the process immediately, as we can do in the examples), requires more invasive patches which I will be posting separately for review, while the ones posted here are trivial enough that I'm pushing them now (commits 23bcea4a..1373d423).
I also audited other uses of completion callbacks: - in tests, closure-lifetimes, errors, server-death, and shutdown-flags are all safe (either the callback returns 0 and the code later calls nbd_aio_command_completed, or we are explicitly testing aspects of completion callbacks) - in fuse/operations.c, the completion callback returns 0 and calls nbd_aio_command_completed later - in examples/strict-structured-reads.c, the completion callback properly checks *error
I also checked that nbdkit's nbd plugin properly checks *error.
Eric Blake (4): examples: aio-connect-read.c: Fix error handling examples: glib-main-loop: don't strand commands when gsource disappears examples: glib-main-loop: Fix error handling copy: Ignore extents if we encountered earlier error
examples/aio-connect-read.c | 7 +++++++ examples/glib-main-loop.c | 14 ++++++++++++-- copy/nbd-ops.c | 4 ++-- 3 files changed, 21 insertions(+), 4 deletions(-)
On Wed, Feb 02, 2022 at 09:59:04AM +0100, Laszlo Ersek wrote:
However, in extent_wrapper() -> extent_wrapper_locked(), there is more confusion for me:
metacontextv = caml_copy_string (metacontext); offsetv = caml_copy_int64 (offset); entriesv = nbd_internal_ocaml_alloc_int64_from_uint32_array (entries, nr_entries); errorv = caml_alloc_tuple (1); Store_field (errorv, 0, Val_int (*error)); args[0] = metacontextv; args[1] = offsetv; args[2] = entriesv; args[3] = errorv; rv = caml_callbackN_exn (data->fnv, 4, args); *error = Int_val (Field (errorv, 0)); if (Is_exception_result (rv)) { nbd_internal_ocaml_exception_in_wrapper ("extent", rv); CAMLreturnT (int, -1); }
r = Int_val (rv); assert (r >= 0); CAMLreturnT (int, r);
If I understand correctly, if we returned (-1) from the (nameless) extents callback, that would trip the assert(); would it not? Thus, we would not reach the nbd_internal_ocaml_raise_error() in the above-quoted nbd_internal_ocaml_nbd_block_status() function.
There are five cases: - Callback returns > 1 => The program shouldn't do this, but it's treated the same as the next case. - Callback returns 1 => For completion callbacks this indicates that the callback was successful and you want to autoretire the command. For other types of callbacks the program shouldn't do this but it is the same as the next case. - Callback returns 0 => Indicates that the callback was successful. - Callback throws an exception => nbd_internal_ocaml_exception_in_wrapper is called => Callback returns -1 indicating (to libnbd) that there was an error. cmd->error is updated with !err || EPROTO. - Callback returns < 0 => assert fails! The assert is a bit harsh, but it indicates a bug in the OCaml code. It probably means the OCaml code thought that returning -1 indicates an error, whereas the OCaml code should throw an exception instead. For the history of why that assert was added see: https://gitlab.com/nbdkit/libnbd/-/commit/d881d160e1cd9c9964782300a7652ffb4e... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On 02/02/22 11:55, Richard W.M. Jones wrote:
On Wed, Feb 02, 2022 at 09:59:04AM +0100, Laszlo Ersek wrote:
However, in extent_wrapper() -> extent_wrapper_locked(), there is more confusion for me:
metacontextv = caml_copy_string (metacontext); offsetv = caml_copy_int64 (offset); entriesv = nbd_internal_ocaml_alloc_int64_from_uint32_array (entries, nr_entries); errorv = caml_alloc_tuple (1); Store_field (errorv, 0, Val_int (*error)); args[0] = metacontextv; args[1] = offsetv; args[2] = entriesv; args[3] = errorv; rv = caml_callbackN_exn (data->fnv, 4, args); *error = Int_val (Field (errorv, 0)); if (Is_exception_result (rv)) { nbd_internal_ocaml_exception_in_wrapper ("extent", rv); CAMLreturnT (int, -1); }
r = Int_val (rv); assert (r >= 0); CAMLreturnT (int, r);
If I understand correctly, if we returned (-1) from the (nameless) extents callback, that would trip the assert(); would it not? Thus, we would not reach the nbd_internal_ocaml_raise_error() in the above-quoted nbd_internal_ocaml_nbd_block_status() function.
There are five cases:
- Callback returns > 1 => The program shouldn't do this, but it's treated the same as the next case.
- Callback returns 1 => For completion callbacks this indicates that the callback was successful and you want to autoretire the command. For other types of callbacks the program shouldn't do this but it is the same as the next case.
- Callback returns 0 => Indicates that the callback was successful.
- Callback throws an exception => nbd_internal_ocaml_exception_in_wrapper is called => Callback returns -1 indicating (to libnbd) that there was an error. cmd->error is updated with !err || EPROTO.
- Callback returns < 0 => assert fails!
The assert is a bit harsh, but it indicates a bug in the OCaml code. It probably means the OCaml code thought that returning -1 indicates an error, whereas the OCaml code should throw an exception instead.
For the history of why that assert was added see:
https://gitlab.com/nbdkit/libnbd/-/commit/d881d160e1cd9c9964782300a7652ffb4e...
Thank you! Laszlo
On Wed, Feb 02, 2022 at 09:23:09AM +0100, Laszlo Ersek wrote:
On 02/01/22 20:44, Eric Blake wrote:
These are the trivial patches modeled after Nir's commit 7ba6ef67, fixing further places that fail to check *error during callbacks.
I find these interfaces very difficult to use.
https://libguestfs.org/nbd_aio_block_status.3.html
We have two callbacks here, each takes a *error, and I have zero idea of the order the callbacks are supposed to be called in.
The documentation is not great, but I can tell you that they are called in the order: - extent_cb (usually once, but in theory several times) - completion_cb (once at the end) The *error parameter is also not well documented, but the behaviour is: - There is a single error value stored for the asynch block status command (cmd->error). - *error is initialized from cmd->error on entry to the callback, ie. *error != 0 if an error has been reported by a previous callback. - If the callback sets *error != 0 _and_ returns -1 then cmd->error is updated. - The callback can also fail by returning -1 and not setting *error, in which case cmd->error is set to EPROTO (but don't do this as it means your program is buggy - but see below about OCaml). - Since there's only one error value, updating it overwrites previous values. - Returning -1 doesn't stop future callbacks, because basically we're pulling data off the wire and there's no way to tell the server to stop. So extent and completion callbacks are still called. - Don't forget that its not just callbacks that can fail, commands can fail for other reasons, eg. network errors. In this case the whole connection will drop into what we call the %DEAD state (because NBD can't resynchronize). Every call returns -1 and you have to close and reopen the connection. It's possible to explicitly test for this (nbd_aio_is_dead). https://gitlab.com/nbdkit/libnbd/-/blob/1373d423f5ac42255fe606d2a7ab1caf18a5... https://gitlab.com/nbdkit/libnbd/-/blob/1373d423f5ac42255fe606d2a7ab1caf18a5... To find out if a command had an error you can do either (but not both): - Check for *error != 0 in completion_cb. Return 1 from completion_cb which auto-retires the command. - Call nbd_aio_command_completed which will return -1 and then call nbd_get_errno which will return cmd->error. This also retires the command. (Note in this case you also have to deal with nbd_aio_command_completed returning 0, meaning the command has not completed, or 1, meaning the command completed without error). The first method is what nbdcopy should be doing but isn't.
The last paragraph of the commit message speaks about this, and the code updates seem to suggest some ordering, but I'm still unsure.
... In fact, I'm now uncertain whether ignoring "err" in virt-v2v's "lib/utils.ml", function "get_disk_allocated", is safe. It's a "sync" (not aio) nbd_block_status() call, but the documentation <https://libguestfs.org/nbd_block_status.3.html> says:
"On entry to the callback, the error parameter contains the errno value of any previously detected error."
The above applies to synch commands as well, with the difference that: - There's no completion callback (because there's no need for it in a synch call). - Once the command completes, it is retired and the error status is copied to nbd_get_errno. - The synch command returns -1 if there was an error or 0 if successful. See lib/rw.c: wait_for_command and nbd_unlocked_block_status. So in answer to your question, virt-v2v can ignore !err, unless there is a reason why you want to know if previous callbacks encountered an error (maybe you want to save some work). You'll still get an exception thrown by NBD.block_status.
So what should we do then, in "get_disk_allocated"? Return -1 immediately? And if we do so, what is that going to mean for "NBD.block_status"? Will it raise an exception?
All OCaml wrappers check for the underlying nbd_* call failing and turn that into an exception which carries the errno and error string. See ocaml/nbd-c.c and ocaml/helpers.c:nbd_internal_ocaml_raise_error In addition, when calling back into OCaml code (ie. into the extent callback), if the callback throws an exception then that is automatically turned into setting *error from the err reference and returning -1. OCaml code ought to do this when encountering an error inside the callback: if some_error then ( err := errno; raise AnException ) But there are a number of difficulties with that in practice: - OCaml functions that you call might raise exceptions which you don't catch, so you don't end up setting !err on all exceptions. - OCaml code doesn't have access to C errno values! I wouldn't worry too much about this. What happens in practice is that nbd_internal_ocaml_exception_in_wrapper prints this on stderr: libnbd: <operation>: uncaught OCaml exception: <exception name> cmd->error is set to EPROTO and the function as a whole throws some NBD.Error exception. The error doesn't get lost in this case, it's just that the output is not ideal. See ocaml/nbd-c.c:extent_wrapper_locked and ocaml/helpers.c:nbd_internal_ocaml_exception_in_wrapper Rich.
Thanks, Laszlo
nbdcopy still has a bigger bug in that it is NOT properly checking for *error in the pread/pwrite completion handlers, but fixing that gracefully (rather than just quitting the process immediately, as we can do in the examples), requires more invasive patches which I will be posting separately for review, while the ones posted here are trivial enough that I'm pushing them now (commits 23bcea4a..1373d423).
I also audited other uses of completion callbacks: - in tests, closure-lifetimes, errors, server-death, and shutdown-flags are all safe (either the callback returns 0 and the code later calls nbd_aio_command_completed, or we are explicitly testing aspects of completion callbacks) - in fuse/operations.c, the completion callback returns 0 and calls nbd_aio_command_completed later - in examples/strict-structured-reads.c, the completion callback properly checks *error
I also checked that nbdkit's nbd plugin properly checks *error.
Eric Blake (4): examples: aio-connect-read.c: Fix error handling examples: glib-main-loop: don't strand commands when gsource disappears examples: glib-main-loop: Fix error handling copy: Ignore extents if we encountered earlier error
examples/aio-connect-read.c | 7 +++++++ examples/glib-main-loop.c | 14 ++++++++++++-- copy/nbd-ops.c | 4 ++-- 3 files changed, 21 insertions(+), 4 deletions(-)
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On 02/02/22 11:43, Richard W.M. Jones wrote:
On Wed, Feb 02, 2022 at 09:23:09AM +0100, Laszlo Ersek wrote:
On 02/01/22 20:44, Eric Blake wrote:
These are the trivial patches modeled after Nir's commit 7ba6ef67, fixing further places that fail to check *error during callbacks.
I find these interfaces very difficult to use.
https://libguestfs.org/nbd_aio_block_status.3.html
We have two callbacks here, each takes a *error, and I have zero idea of the order the callbacks are supposed to be called in.
The documentation is not great, but I can tell you that they are called in the order:
- extent_cb (usually once, but in theory several times) - completion_cb (once at the end)
The *error parameter is also not well documented, but the behaviour is:
- There is a single error value stored for the asynch block status command (cmd->error).
- *error is initialized from cmd->error on entry to the callback, ie. *error != 0 if an error has been reported by a previous callback.
- If the callback sets *error != 0 _and_ returns -1 then cmd->error is updated.
- The callback can also fail by returning -1 and not setting *error, in which case cmd->error is set to EPROTO (but don't do this as it means your program is buggy - but see below about OCaml).
- Since there's only one error value, updating it overwrites previous values.
- Returning -1 doesn't stop future callbacks, because basically we're pulling data off the wire and there's no way to tell the server to stop. So extent and completion callbacks are still called.
- Don't forget that its not just callbacks that can fail, commands can fail for other reasons, eg. network errors. In this case the whole connection will drop into what we call the %DEAD state (because NBD can't resynchronize). Every call returns -1 and you have to close and reopen the connection. It's possible to explicitly test for this (nbd_aio_is_dead).
https://gitlab.com/nbdkit/libnbd/-/blob/1373d423f5ac42255fe606d2a7ab1caf18a5... https://gitlab.com/nbdkit/libnbd/-/blob/1373d423f5ac42255fe606d2a7ab1caf18a5...
To find out if a command had an error you can do either (but not both):
- Check for *error != 0 in completion_cb. Return 1 from completion_cb which auto-retires the command.
- Call nbd_aio_command_completed which will return -1 and then call nbd_get_errno which will return cmd->error. This also retires the command. (Note in this case you also have to deal with nbd_aio_command_completed returning 0, meaning the command has not completed, or 1, meaning the command completed without error).
The first method is what nbdcopy should be doing but isn't.
The last paragraph of the commit message speaks about this, and the code updates seem to suggest some ordering, but I'm still unsure.
... In fact, I'm now uncertain whether ignoring "err" in virt-v2v's "lib/utils.ml", function "get_disk_allocated", is safe. It's a "sync" (not aio) nbd_block_status() call, but the documentation <https://libguestfs.org/nbd_block_status.3.html> says:
"On entry to the callback, the error parameter contains the errno value of any previously detected error."
The above applies to synch commands as well, with the difference that:
- There's no completion callback (because there's no need for it in a synch call).
- Once the command completes, it is retired and the error status is copied to nbd_get_errno.
- The synch command returns -1 if there was an error or 0 if successful.
See lib/rw.c: wait_for_command and nbd_unlocked_block_status.
So in answer to your question, virt-v2v can ignore !err, unless there is a reason why you want to know if previous callbacks encountered an error (maybe you want to save some work). You'll still get an exception thrown by NBD.block_status.
So what should we do then, in "get_disk_allocated"? Return -1 immediately? And if we do so, what is that going to mean for "NBD.block_status"? Will it raise an exception?
All OCaml wrappers check for the underlying nbd_* call failing and turn that into an exception which carries the errno and error string. See ocaml/nbd-c.c and ocaml/helpers.c:nbd_internal_ocaml_raise_error
In addition, when calling back into OCaml code (ie. into the extent callback), if the callback throws an exception then that is automatically turned into setting *error from the err reference and returning -1.
OCaml code ought to do this when encountering an error inside the callback:
if some_error then ( err := errno; raise AnException )
But there are a number of difficulties with that in practice:
- OCaml functions that you call might raise exceptions which you don't catch, so you don't end up setting !err on all exceptions.
- OCaml code doesn't have access to C errno values!
I wouldn't worry too much about this. What happens in practice is that nbd_internal_ocaml_exception_in_wrapper prints this on stderr:
libnbd: <operation>: uncaught OCaml exception: <exception name>
cmd->error is set to EPROTO and the function as a whole throws some NBD.Error exception. The error doesn't get lost in this case, it's just that the output is not ideal.
See ocaml/nbd-c.c:extent_wrapper_locked and ocaml/helpers.c:nbd_internal_ocaml_exception_in_wrapper
OK -- so end-to-end (IIUC), all I could learn, with reading !err in the OCaml callback, is whether an earlier invocation of the callback raised an exception (possibly indirectly, i.e. by returning -1 to the wrapper, or directly by raising an exception, or even by allowing a deeper exception to escape). Eric said the extents array and the metacontext ID are still valid in this case, so eliminating the loop body would only be a speed-up, and the outermost call will throw an exception anyway. ... What happens "if the server replies with NBD_REPLY_TYPE_ERROR first and then NBD_REPLY_TYPE_BLOCK_STATUS second (a custom server)"? Hmm I don't think it matters much: we already assume that the entries make sense (e.g. we already assert() that the server does not send zero-length entries), so we'd just complete the inner loop (ignoring !err being nonzero), and then the NBD.block_status function would raise an exception. OK! Thanks! Laszlo
Rich.
Thanks, Laszlo
nbdcopy still has a bigger bug in that it is NOT properly checking for *error in the pread/pwrite completion handlers, but fixing that gracefully (rather than just quitting the process immediately, as we can do in the examples), requires more invasive patches which I will be posting separately for review, while the ones posted here are trivial enough that I'm pushing them now (commits 23bcea4a..1373d423).
I also audited other uses of completion callbacks: - in tests, closure-lifetimes, errors, server-death, and shutdown-flags are all safe (either the callback returns 0 and the code later calls nbd_aio_command_completed, or we are explicitly testing aspects of completion callbacks) - in fuse/operations.c, the completion callback returns 0 and calls nbd_aio_command_completed later - in examples/strict-structured-reads.c, the completion callback properly checks *error
I also checked that nbdkit's nbd plugin properly checks *error.
Eric Blake (4): examples: aio-connect-read.c: Fix error handling examples: glib-main-loop: don't strand commands when gsource disappears examples: glib-main-loop: Fix error handling copy: Ignore extents if we encountered earlier error
examples/aio-connect-read.c | 7 +++++++ examples/glib-main-loop.c | 14 ++++++++++++-- copy/nbd-ops.c | 4 ++-- 3 files changed, 21 insertions(+), 4 deletions(-)
On Wed, Feb 02, 2022 at 04:34:09PM +0100, Laszlo Ersek wrote:
OK -- so end-to-end (IIUC), all I could learn, with reading !err in the OCaml callback, is whether an earlier invocation of the callback raised an exception (possibly indirectly, i.e. by returning -1 to the wrapper, or directly by raising an exception, or even by allowing a deeper exception to escape).
If the callback returned -1 it would cause an assert fail. The only way to return an error indication is to raise an exception (which causes the wrapper to return -1 back to libnbd which then sets cmd->error to !err || EPROTO). BTW I noticed that we could be more clever about !err, but the implementation is not as trivial as I thought so I only documented it: https://gitlab.com/nbdkit/libnbd/-/commit/0e2dd4bf5b024889ed9e63dd2cef9a2def...
Eric said the extents array and the metacontext ID are still valid in this case, so eliminating the loop body would only be a speed-up, and the outermost call will throw an exception anyway.
... What happens "if the server replies with NBD_REPLY_TYPE_ERROR first and then NBD_REPLY_TYPE_BLOCK_STATUS second (a custom server)"?
We jump through states %REPLY.STRUCTURED_REPLY.CHECK -> %REPLY.STRUCTURED_REPLY.RECV_ERROR_MESSAGE -> %REPLY.STRUCTURED_REPLY.RECV_ERROR_TAIL. This will set cmd->error (although not if it was already set by a callback). Then it'll jump to %REPLY.FINISHED which calls the completion callback. Then it'll read NBD_REPLY_TYPE_BLOCK_STATUS, get confused and desynchronize (the magic field will be different), resulting in the handle becoming %DEAD. I believe this is all correct behaviour, as far as that's possible with a server that's doing weird stuff.
Hmm I don't think it matters much: we already assume that the entries make sense (e.g. we already assert() that the server does not send zero-length entries), so we'd just complete the inner loop (ignoring !err being nonzero), and then the NBD.block_status function would raise an exception. OK!
Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
participants (3)
-
Eric Blake -
Laszlo Ersek -
Richard W.M. Jones