nbdkit & qemu 2.12: qemu-img: Protocol error: simple reply when structured reply chunk was expected
nbdkit (upstream 5a7a394c699) currently fails with qemu 2.12.0: $ ./nbdkit memory size=64M --run 'qemu-img convert $nbd /var/tmp/out' nbdkit: memory.2: error: invalid request: unknown command (7) ignored qemu-img: Protocol error: simple reply when structured reply chunk was expected This was a bug in qemu which was fixed upstream quite a long time ago by the commit I've attached at the end of this email. Unfortunately RHEL 7.6 & 7.7 (qemu-kvm-rhev-2.12.0-25.el7) doesn't contain the fix. I wonder if we should: (a) Not worry about it - it's Red Hat's problem. (b) Try and get it fixed in RHEL. I filed a BZ already but I guess it won't be fixed any time soon: https://bugzilla.redhat.com/1692018 (c) Add a workaround in nbdkit. I feel that we probably shouldn't be broken out of the box with RHEL and CentOS 7.6 and 7.7, even though it's not an nbdkit (or qemu) bug. Rich. ---------------------------------------------------------------------- 89aa0d87634e2cb98517509dc8bdb876f26ecf8b is the first bad commit commit 89aa0d87634e2cb98517509dc8bdb876f26ecf8b Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Date: Fri Apr 27 17:20:01 2018 +0300 nbd/client: fix nbd_negotiate_simple_meta_context Initialize received variable. Otherwise, is is possible for server to answer without any contexts, but we will set context_id to something random (received_id is not initialized too) and return 1, which is wrong. To solve it, just initialize received to false. Initialize received_id too, just to make all possible checkers happy. Bug was introduced in 78a33ab58782efdb206de14 "nbd: BLOCK_STATUS for standard get_block_status function: client part" with the whole function. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20180427142002.21930-2-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> CC: qemu-stable@nongnu.org Signed-off-by: Eric Blake <eblake@redhat.com> :040000 040000 9993feb118af1a9a59dbc8fe92015a324e93e557 14db90d621d6b7f1ee5ff97a0ca2cb92f6f2f7e9 M nbd -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On 3/23/19 6:42 AM, Richard W.M. Jones wrote:
nbdkit (upstream 5a7a394c699) currently fails with qemu 2.12.0:
$ ./nbdkit memory size=64M --run 'qemu-img convert $nbd /var/tmp/out' nbdkit: memory.2: error: invalid request: unknown command (7) ignored qemu-img: Protocol error: simple reply when structured reply chunk was expected
This was a bug in qemu which was fixed upstream quite a long time ago by the commit I've attached at the end of this email.
Unfortunately RHEL 7.6 & 7.7 (qemu-kvm-rhev-2.12.0-25.el7) doesn't contain the fix. I wonder if we should:
(a) Not worry about it - it's Red Hat's problem.
Upstream qemu has moved on, so from that perspective, yes, it's Red Hat's problem (Red Hat must ensure that any versions of qemu and nbdkit that it ships and which must interoperate do so correctly, whether that be qemu from RHEL 7 and nbdkit from RHEL 8, or the other way around).
(b) Try and get it fixed in RHEL. I filed a BZ already but I guess it won't be fixed any time soon: https://bugzilla.redhat.com/1692018
Several RHEL 7 bugs against NBD are pending - there's hope for RHEL 7.7, but you are also right that guessing timelines is hard.
(c) Add a workaround in nbdkit.
And what workaround would that be? Looking at the qemu patch, the problem is that qemu asked for "base:allocation" and nbdkit replied with 0 contexts. The workaround would either be to reply with NBD_REP_ERR_UNSUP (the behavior before NBD_OPT_SET_META_CONTEXT was added), or to implement "base:allocation" (even if we implement the poor-man's version that always reports that the entire image is data, for all requests, without any feedback from the plugins). The former feels odd, but the latter seems oddly appealing as progress towards our end goal (kind of how our initial implementation of NBD_CMD_WRITE_ZEROES was always advertised even before we finished wiring up plug support).
I feel that we probably shouldn't be broken out of the box with RHEL and CentOS 7.6 and 7.7, even though it's not an nbdkit (or qemu) bug.
Correct that it is not nbdkit's bug; it was a bug in qemu (but one that upstream no longer worries about), and old enough of a qemu release that I don't the the NBD protocol needs to bother mentioning it as a special case.
Rich.
---------------------------------------------------------------------- 89aa0d87634e2cb98517509dc8bdb876f26ecf8b is the first bad commit commit 89aa0d87634e2cb98517509dc8bdb876f26ecf8b Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Date: Fri Apr 27 17:20:01 2018 +0300
nbd/client: fix nbd_negotiate_simple_meta_context
Initialize received variable. Otherwise, is is possible for server to answer without any contexts, but we will set context_id to something random (received_id is not initialized too) and return 1, which is wrong.
To solve it, just initialize received to false. Initialize received_id too, just to make all possible checkers happy.
Bug was introduced in 78a33ab58782efdb206de14 "nbd: BLOCK_STATUS for standard get_block_status function: client part" with the whole function.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20180427142002.21930-2-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> CC: qemu-stable@nongnu.org Signed-off-by: Eric Blake <eblake@redhat.com>
:040000 040000 9993feb118af1a9a59dbc8fe92015a324e93e557 14db90d621d6b7f1ee5ff97a0ca2cb92f6f2f7e9 M nbd
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Sat, Mar 23, 2019 at 06:57:14AM -0500, Eric Blake wrote:
On 3/23/19 6:42 AM, Richard W.M. Jones wrote:
(b) Try and get it fixed in RHEL. I filed a BZ already but I guess it won't be fixed any time soon: https://bugzilla.redhat.com/1692018
Several RHEL 7 bugs against NBD are pending - there's hope for RHEL 7.7, but you are also right that guessing timelines is hard.
Yes, my worry is that if we miss 7.7 then this bug will exist forever since as far as I know we (Red Hat) are not planning any further qemu improvements in 7.8 and above.
(c) Add a workaround in nbdkit.
And what workaround would that be? Looking at the qemu patch, the problem is that qemu asked for "base:allocation" and nbdkit replied with 0 contexts. The workaround would either be to reply with NBD_REP_ERR_UNSUP (the behavior before NBD_OPT_SET_META_CONTEXT was added), or to implement "base:allocation" (even if we implement the poor-man's version that always reports that the entire image is data, for all requests, without any feedback from the plugins). The former feels odd, but the latter seems oddly appealing as progress towards our end goal (kind of how our initial implementation of NBD_CMD_WRITE_ZEROES was always advertised even before we finished wiring up plug support).
A good point here is what happens with the block-status branch[1]. Let's see: $ ./nbdkit memory size=64M --run '/home/rjones/d/qemu/qemu-img convert $nbd /var/tmp/out' qemu-img: Payload too large nbdkit: memory.1: error: write reply: NBD_CMD_BLOCK_STATUS: Broken pipe Oh dear. I believe this is actually a bug in the block status code so let's see if I can narrow this down first ... Rich. [1] https://github.com/rwmjones/nbdkit/tree/block-status -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Sat, Mar 23, 2019 at 12:21:11PM +0000, Richard W.M. Jones wrote:
On Sat, Mar 23, 2019 at 06:57:14AM -0500, Eric Blake wrote:
And what workaround would that be? Looking at the qemu patch, the problem is that qemu asked for "base:allocation" and nbdkit replied with 0 contexts. The workaround would either be to reply with NBD_REP_ERR_UNSUP (the behavior before NBD_OPT_SET_META_CONTEXT was added), or to implement "base:allocation" (even if we implement the poor-man's version that always reports that the entire image is data, for all requests, without any feedback from the plugins). The former feels odd, but the latter seems oddly appealing as progress towards our end goal (kind of how our initial implementation of NBD_CMD_WRITE_ZEROES was always advertised even before we finished wiring up plug support).
A good point here is what happens with the block-status branch[1]. Let's see:
$ ./nbdkit memory size=64M --run '/home/rjones/d/qemu/qemu-img convert $nbd /var/tmp/out' qemu-img: Payload too large nbdkit: memory.1: error: write reply: NBD_CMD_BLOCK_STATUS: Broken pipe
Oh dear. I believe this is actually a bug in the block status code so let's see if I can narrow this down first ...
OK I fixed this bug (in nbdkit). The block-status branch *works* with qemu 2.12.0, so another option here is to do nothing and wait until that work is finished and merged. NB During this I discovered another (but minor) bug in qemu. If you feed qemu a long series of block status replies then it eventually closes the connection. However it does not print an error message. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 3/23/19 7:58 AM, Richard W.M. Jones wrote:
A good point here is what happens with the block-status branch[1]. Let's see:
$ ./nbdkit memory size=64M --run '/home/rjones/d/qemu/qemu-img convert $nbd /var/tmp/out' qemu-img: Payload too large nbdkit: memory.1: error: write reply: NBD_CMD_BLOCK_STATUS: Broken pipe
Oh dear. I believe this is actually a bug in the block status code so let's see if I can narrow this down first ...
OK I fixed this bug (in nbdkit).
Which is why the block-status branch is still in flux :)
The block-status branch *works* with qemu 2.12.0, so another option here is to do nothing and wait until that work is finished and merged.
Well, it works as long as you don't send a simple error reply (our mails crossed, and I submitted that patch for that issue already).
NB During this I discovered another (but minor) bug in qemu. If you feed qemu a long series of block status replies then it eventually closes the connection. However it does not print an error message.
Which version(s) of qemu? Was REQ_ONE in force or not? Was it a long series of status replies that could have been coalesced, or were they alternating status? Did the replies extend beyond the client's original request? I'd like to reproduce the failure, to determine if qemu needs more patches. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Sat, Mar 23, 2019 at 09:52:41AM -0500, Eric Blake wrote:
On 3/23/19 7:58 AM, Richard W.M. Jones wrote:
NB During this I discovered another (but minor) bug in qemu. If you feed qemu a long series of block status replies then it eventually closes the connection. However it does not print an error message.
Which version(s) of qemu? Was REQ_ONE in force or not? Was it a long series of status replies that could have been coalesced, or were they alternating status? Did the replies extend beyond the client's original request?
I'd like to reproduce the failure, to determine if qemu needs more patches.
Here's how to reproduce it. You will need nbdkit block-status https://github.com/rwmjones/nbdkit/tree/block-status @ 8c3b6cbe81bdf with the small patch attached to this email on top. Then run it against qemu (from git) like this: ./nbdkit -fv memory size=64M \ --run '/path/to/qemu-img convert $nbd /var/tmp/out' You should see: nbdkit: memory.0: debug: pread count=512 offset=0 nbdkit: memory.1: debug: extents count=67108864 offset=0 req_one=1 nbdkit: memory.3: debug: client sent NBD_CMD_DISC, closing connection nbdkit: memory.3: debug: exiting worker thread memory.3 and qemu-img exits with code 1, but there is no message printed. (You can use --run '... || echo FAIL' if you don't believe that qemu-img is exiting with a failure.) While preparing this email I noticed that v3.1.0 printed an error message: qemu-img: Protocol error: invalid payload for NBD_REPLY_TYPE_BLOCK_STATUS So I have bisected this to the following commit: d8b4bad846f08ff0f167b46dc156a5310b750484 is the first bad commit commit d8b4bad846f08ff0f167b46dc156a5310b750484 Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Date: Fri Nov 2 18:11:52 2018 +0300 block/nbd-client: use traces instead of noisy error_report_err Reduce extra noise of nbd-client, change 083 correspondingly. In various commits (be41c100 in 2.10, f140e300 in 2.11, 78a33ab in 2.12), we added spots where qemu as an NBD client would report problems communicating with the server to stderr, because there was no where else to send the error to. However, this is racy, particularly since the most common source of these errors is when either the client or the server abruptly hangs up, leaving one coroutine to report the error only if it wins (or loses) the race in attempting the read from the server before another thread completes its cleanup of a protocol error that caused the disconnect in the first place. The race is also apparent in the fact that differences in the flush behavior of the server can alter the frequency of encountering the race in the client (see commit 6d39db96). Rather than polluting stderr, it's better to just trace these situations, for use by developers debugging a flaky connection, particularly since the real error that either triggers the abrupt disconnection in the first place, or that results from the EIO when a request can't receive a reply, DOES make it back to the user in the normal Error propagation channels. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20181102151152.288399-4-vsementsov@virtuozzo.com> [eblake: drop depedence on error hint, enhance commit message] Signed-off-by: Eric Blake <eblake@redhat.com> :040000 040000 8cabc7689a7ea1667850d7d670ad4791873ce0a5 80fcc304538a7c7d47d8cb0d15e69cf5f1472ea9 M block :040000 040000 eb3965069b953ae0a5e7f00169a22c91cc428d81 0dbd45dc86117204869fb8f5bae584be07806d5b M tests Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Adding some cc's; my reply truncates, so if you want to catch up to the full original mail, see https://www.redhat.com/archives/libguestfs/2019-March/msg00124.html On 3/23/19 12:00 PM, Richard W.M. Jones wrote:
On Sat, Mar 23, 2019 at 09:52:41AM -0500, Eric Blake wrote:
On 3/23/19 7:58 AM, Richard W.M. Jones wrote:
NB During this I discovered another (but minor) bug in qemu. If you feed qemu a long series of block status replies then it eventually closes the connection. However it does not print an error message.
Which version(s) of qemu? Was REQ_ONE in force or not? Was it a long series of status replies that could have been coalesced, or were they alternating status? Did the replies extend beyond the client's original request?
I'd like to reproduce the failure, to determine if qemu needs more patches.
Here's how to reproduce it. ... Thanks, that helps.
with the small patch attached to this email on top.
diff --git a/server/protocol.c b/server/protocol.c index ac6eb2c..d406dc8 100644 --- a/server/protocol.c +++ b/server/protocol.c @@ -503,6 +503,9 @@ send_structured_reply_block_status (struct connection *conn, if (blocks == NULL) return connection_set_status (conn, -1);
+ nr_blocks = 32; + blocks[0].length = htobe32 (4096); + reply.magic = htobe32 (NBD_STRUCTURED_REPLY_MAGIC); reply.handle = handle; reply.flags = htobe16 (NBD_REPLY_FLAG_DONE); @@ -526,7 +529,7 @@ send_structured_reply_block_status (struct connection *conn,
/* Send each block descriptor. */ for (i = 0; i < nr_blocks; ++i) { - r = conn->send (conn, &blocks[i], sizeof blocks[i]); + r = conn->send (conn, &blocks[0], sizeof blocks[0]);
So you hack the server into sending adjacent replies that were not coalesced, and sending a reply that covers 128k of the image regardless of how much length the client requested. There are several ways in which that can violate protocol (but hey - testing qemu resiliency is WORTH temporary protocol violations): - If the client requested REQ_ONE, you sent too many extents (sadly, qemu still uses REQ_ONE always, even though I'd like to lift that restriction in the future; the rest of my bullets are applicable when REQ_ONE is not in use) - If the client requested an offset that is less than 128k from end-of-image, you sent a reply that exceeds the advertised image size - If the client requested a length < 127k, you sent too many extents But there are other situations where your reply is valid: - If the client requested a length in [127k, 128k), your reply gave more information than the client asked for, but the protocol allows that because your extra information lies in the final extent when REQ_ONE is not in use - If the client requested a length > 128k, your reply ended early, but the protocol allows that
./nbdkit -fv memory size=64M \ --run '/path/to/qemu-img convert $nbd /var/tmp/out'
You should see:
nbdkit: memory.0: debug: pread count=512 offset=0 nbdkit: memory.1: debug: extents count=67108864 offset=0 req_one=1 nbdkit: memory.3: debug: client sent NBD_CMD_DISC, closing connection nbdkit: memory.3: debug: exiting worker thread memory.3
So the immediate NBD_CMD_DISC is reasonable, implying that the client flagged one of the ways you could have violated protocol (but whether qemu should just hang up, or should just ignore that particular reply but still keep the connection alive, is a QoI question for qemu).
and qemu-img exits with code 1, but there is no message printed. (You can use --run '... || echo FAIL' if you don't believe that qemu-img is exiting with a failure.)
While preparing this email I noticed that v3.1.0 printed an error message:
qemu-img: Protocol error: invalid payload for NBD_REPLY_TYPE_BLOCK_STATUS
Which says that qemu used to be noisy at diagnosing the protocol error, and now it is not. Ideally, qemu as a client shouldn't encounter a server that is violating protocol, but when it does, it makes sense for qemu to call attention to the buggy server. That sounds like a regression in qemu worth fixing, so I'll whip up a patch.
So I have bisected this to the following commit:
d8b4bad846f08ff0f167b46dc156a5310b750484 is the first bad commit commit d8b4bad846f08ff0f167b46dc156a5310b750484 Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Date: Fri Nov 2 18:11:52 2018 +0300
block/nbd-client: use traces instead of noisy error_report_err
That's also helpful, as it pinpoints where we regressed at flagging the protocol error. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Sat, Mar 23, 2019 at 12:48:39PM -0500, Eric Blake wrote:
So you hack the server into sending adjacent replies that were not coalesced, and sending a reply that covers 128k of the image regardless of how much length the client requested. There are several ways in which that can violate protocol (but hey - testing qemu resiliency is WORTH temporary protocol violations): - If the client requested REQ_ONE, you sent too many extents (sadly, qemu still uses REQ_ONE always, even though I'd like to lift that restriction in the future; the rest of my bullets are applicable when REQ_ONE is not in use)
Yes that correct. My original block-status patch had a bug which ignored the REQ_ONE flag by accident, hence it was behaving as you describe. This is now fixed in the latest version, but the patch reintroduces the bug intentionally. [...]
nbdkit: memory.0: debug: pread count=512 offset=0 nbdkit: memory.1: debug: extents count=67108864 offset=0 req_one=1 nbdkit: memory.3: debug: client sent NBD_CMD_DISC, closing connection nbdkit: memory.3: debug: exiting worker thread memory.3
So the immediate NBD_CMD_DISC is reasonable, implying that the client flagged one of the ways you could have violated protocol (but whether qemu should just hang up, or should just ignore that particular reply but still keep the connection alive, is a QoI question for qemu).
I don't know. If the server is out of spec then disconnecting is reasonable. But it depends on whether the qemu community wants to be resilient against bad servers or wants to use its undoubted influence to get them fixed.
and qemu-img exits with code 1, but there is no message printed. (You can use --run '... || echo FAIL' if you don't believe that qemu-img is exiting with a failure.)
While preparing this email I noticed that v3.1.0 printed an error message:
qemu-img: Protocol error: invalid payload for NBD_REPLY_TYPE_BLOCK_STATUS
Which says that qemu used to be noisy at diagnosing the protocol error, and now it is not. Ideally, qemu as a client shouldn't encounter a server that is violating protocol, but when it does, it makes sense for qemu to call attention to the buggy server. That sounds like a regression in qemu worth fixing, so I'll whip up a patch.
Yes this is my opinion too. Obviously it shouldn't happen, but in real life it might happen I can't see any great down-side to qemu/qemu-img printing something useful on stderr. We should be concerned about malicious NBD servers, but I don't think that making the client print an error is a problem (or it's at least arguable). Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 3/23/19 8:21 AM, Richard W.M. Jones wrote:
On Sat, Mar 23, 2019 at 06:57:14AM -0500, Eric Blake wrote:
On 3/23/19 6:42 AM, Richard W.M. Jones wrote:
(b) Try and get it fixed in RHEL. I filed a BZ already but I guess it won't be fixed any time soon: https://bugzilla.redhat.com/1692018
Several RHEL 7 bugs against NBD are pending - there's hope for RHEL 7.7, but you are also right that guessing timelines is hard.
Yes, my worry is that if we miss 7.7 then this bug will exist forever since as far as I know we (Red Hat) are not planning any further qemu improvements in 7.8 and above.
Feel free to add some ACKs to the lengthy backports I've posted in 7.7 recently ... 3.1.0 NBD Roundup is probably particularly of interest to you right now. I'm working on 4.0.0 NBD roundup right now, too.
[Removed public mailing list] On Mon, Mar 25, 2019 at 01:45:32PM -0400, John Snow wrote:
On 3/23/19 8:21 AM, Richard W.M. Jones wrote:
On Sat, Mar 23, 2019 at 06:57:14AM -0500, Eric Blake wrote:
On 3/23/19 6:42 AM, Richard W.M. Jones wrote:
(b) Try and get it fixed in RHEL. I filed a BZ already but I guess it won't be fixed any time soon: https://bugzilla.redhat.com/1692018
Several RHEL 7 bugs against NBD are pending - there's hope for RHEL 7.7, but you are also right that guessing timelines is hard.
Yes, my worry is that if we miss 7.7 then this bug will exist forever since as far as I know we (Red Hat) are not planning any further qemu improvements in 7.8 and above.
Feel free to add some ACKs to the lengthy backports I've posted in 7.7 recently ...
3.1.0 NBD Roundup is probably particularly of interest to you right now.
Is this on rhvirt-patches? I can't see it ... Rich.
I'm working on 4.0.0 NBD roundup right now, too.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
[adding qemu] On 3/23/19 6:57 AM, Eric Blake wrote:
On 3/23/19 6:42 AM, Richard W.M. Jones wrote:
nbdkit (upstream 5a7a394c699) currently fails with qemu 2.12.0:
$ ./nbdkit memory size=64M --run 'qemu-img convert $nbd /var/tmp/out' nbdkit: memory.2: error: invalid request: unknown command (7) ignored qemu-img: Protocol error: simple reply when structured reply chunk was expected
This was a bug in qemu which was fixed upstream quite a long time ago by the commit I've attached at the end of this email.
---------------------------------------------------------------------- 89aa0d87634e2cb98517509dc8bdb876f26ecf8b is the first bad commit commit 89aa0d87634e2cb98517509dc8bdb876f26ecf8b Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Date: Fri Apr 27 17:20:01 2018 +0300
nbd/client: fix nbd_negotiate_simple_meta_context
Initialize received variable. Otherwise, is is possible for server to answer without any contexts, but we will set context_id to something random (received_id is not initialized too) and return 1, which is wrong.
Ouch - upstream qemu still has a latent bug in this area. When qemu as server sends an error to NBD_CMD_BLOCK_STATUS, it uses a structured error reply, so we haven't tickled it before. But the NBD spec permits a simple error reply to any command except NBD_CMD_READ when structured replies are negotiated, and that's what nbdkit currently implements. If I add this hack patch to qemu's server (to intentionally fail NBD_CMD_BLOCK_STATUS with a simple error), then the qemu client _should_ be able to just report that block status failed and keep the connection alive, rather than its current behavior of hanging up. So we still need a patch for qemu 4.0 for the client to accept simple errors. :( With the hack applied, I got: $ ./qemu-io -f raw nbd://localhost:10809 --trace=nbd_\* -c map ... 17218@1553343857.048440:nbd_send_request Sending request to server: { .from = 0, .len = 1049088, .handle = 94796979915216, .flags = 0x8, .type = 7 (block status) } 17218@1553343857.048643:nbd_receive_simple_reply Got simple reply: { .error = 22 (EINVAL), handle = 94796979915216 } 17218@1553343857.048655:nbd_co_request_fail Request failed { .from = 0, .len = 1049088, .handle = 94796979915216, .flags = 0x8, .type = 7 (block status) } ret = -22, err: Protocol error: simple reply when structured reply chunk was expected Failed to get allocation status: Invalid argument diff --git i/nbd/server.c w/nbd/server.c index fd013a2817a..35f549abbf9 100644 --- i/nbd/server.c +++ w/nbd/server.c @@ -2148,16 +2148,16 @@ static int nbd_co_receive_request(NBDRequestData *req, NBDRequest *request, * Returns 0 if connection is still live, -errno on failure to talk to client */ static coroutine_fn int nbd_send_generic_reply(NBDClient *client, - uint64_t handle, + NBDRequest *request, int ret, const char *error_msg, Error **errp) { - if (client->structured_reply && ret < 0) { - return nbd_co_send_structured_error(client, handle, -ret, error_msg, + if (client->structured_reply && ret < 0 && request->type == NBD_CMD_READ) { + return nbd_co_send_structured_error(client, request->handle, -ret, error_msg, errp); } else { - return nbd_co_send_simple_reply(client, handle, ret < 0 ? -ret : 0, + return nbd_co_send_simple_reply(client, request->handle, ret < 0 ? -ret : 0, NULL, 0, errp); } } @@ -2177,7 +2177,7 @@ static coroutine_fn int nbd_do_cmd_read(NBDClient *client, NBDRequest *request, if (request->flags & NBD_CMD_FLAG_FUA) { ret = blk_co_flush(exp->blk); if (ret < 0) { - return nbd_send_generic_reply(client, request->handle, ret, + return nbd_send_generic_reply(client, request, ret, "flush failed", errp); } } @@ -2192,7 +2192,7 @@ static coroutine_fn int nbd_do_cmd_read(NBDClient *client, NBDRequest *request, ret = blk_pread(exp->blk, request->from + exp->dev_offset, data, request->len); if (ret < 0 || request->type == NBD_CMD_CACHE) { - return nbd_send_generic_reply(client, request->handle, ret, + return nbd_send_generic_reply(client, request, ret, "reading from file failed", errp); } @@ -2234,7 +2234,7 @@ static coroutine_fn int nbd_handle_request(NBDClient *client, } ret = blk_pwrite(exp->blk, request->from + exp->dev_offset, data, request->len, flags); - return nbd_send_generic_reply(client, request->handle, ret, + return nbd_send_generic_reply(client, request, ret, "writing to file failed", errp); case NBD_CMD_WRITE_ZEROES: @@ -2247,7 +2247,7 @@ static coroutine_fn int nbd_handle_request(NBDClient *client, } ret = blk_pwrite_zeroes(exp->blk, request->from + exp->dev_offset, request->len, flags); - return nbd_send_generic_reply(client, request->handle, ret, + return nbd_send_generic_reply(client, request, ret, "writing to file failed", errp); case NBD_CMD_DISC: @@ -2256,7 +2256,7 @@ static coroutine_fn int nbd_handle_request(NBDClient *client, case NBD_CMD_FLUSH: ret = blk_co_flush(exp->blk); - return nbd_send_generic_reply(client, request->handle, ret, + return nbd_send_generic_reply(client, request, ret, "flush failed", errp); case NBD_CMD_TRIM: @@ -2265,12 +2265,14 @@ static coroutine_fn int nbd_handle_request(NBDClient *client, if (ret == 0 && request->flags & NBD_CMD_FLAG_FUA) { ret = blk_co_flush(exp->blk); } - return nbd_send_generic_reply(client, request->handle, ret, + return nbd_send_generic_reply(client, request, ret, "discard failed", errp); case NBD_CMD_BLOCK_STATUS: + return nbd_send_generic_reply(client, request, -EINVAL, + "no status for you today", errp); if (!request->len) { - return nbd_send_generic_reply(client, request->handle, -EINVAL, + return nbd_send_generic_reply(client, request, -EINVAL, "need non-zero length", errp); } if (client->export_meta.valid && @@ -2304,7 +2306,7 @@ static coroutine_fn int nbd_handle_request(NBDClient *client, return ret; } else { - return nbd_send_generic_reply(client, request->handle, -EINVAL, + return nbd_send_generic_reply(client, request, -EINVAL, "CMD_BLOCK_STATUS not negotiated", errp); } @@ -2312,7 +2314,7 @@ static coroutine_fn int nbd_handle_request(NBDClient *client, default: msg = g_strdup_printf("invalid request type (%" PRIu32 ") received", request->type); - ret = nbd_send_generic_reply(client, request->handle, -EINVAL, msg, + ret = nbd_send_generic_reply(client, request, -EINVAL, msg, errp); g_free(msg); return ret; @@ -2357,7 +2359,7 @@ static coroutine_fn void nbd_trip(void *opaque) Error *export_err = local_err; local_err = NULL; - ret = nbd_send_generic_reply(client, request.handle, -EINVAL, + ret = nbd_send_generic_reply(client, &request, -EINVAL, error_get_pretty(export_err), &local_err); error_free(export_err); } else { -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
participants (3)
-
Eric Blake -
John Snow -
Richard W.M. Jones