Re: [Libguestfs] [nbdkit PATCH v3 3/5] nbd: Use libnbd 0.1.3+
On 6/12/19 4:00 PM, Eric Blake wrote:
This conversion should be feature compatible with the standalone nbd
code. Note that the use of libnbd makes the binary for this particular
plugin fall under an LGPLv2+ license rather than BSD; but the source
code in nbd.c remains BSD.
A lot of code simply disappears, now that I'm no longer directly
utilizing the NBD protocol files but relying on libnbd. Coordination
between threads from nbdkit and a central state machine thread is
gated by a lock around the pending transition list, using the same
trick as before of using a semaphore to wake up the right thread after
the server's reply, regardless of out-of-order handling from the
server.
Benchmark-wise, using the same setup as in commit e897ed70, I see
a slight slowdown:
Pre-patch, the runs averaged 0.754s, 2.17E+08 bits/s
Post-patch, the runs averaged 0.849s, 1.93E+07 bits/s
With the latest libnbd, the numbers are now:
Pre-patch, the runs averaged 16.259s, 1.03E+10 bits/s
Post-patch, the runs averaged 17.585s, 9.54E+09 bits/s
for about 8% degradation.
The explanation is that pre-patch, one nbdkit thread can be write()ing
the next request to the server at the same time the reader thread is
read()ing the previous reply; post-patch, libnbd's current structure
handles all I/O under a lock such that there is no simultaneous
bidirectional traffic. There are discussions about adding future
libnbd APIs to address this limitation.
My other insight is that since all our calls are aio, they really
shouldn't be blocking. The biggest delay is thus not waiting for a lock,
but the fact that as long as the state machine is blocked on POLLIN, we
are delaying our next request to the server, whether or not the server
would actually be able to read the request at the moment. Parallel
threads for read() and write() are essential for max performance under
blocking I/O, but shouldn't really matter for non-blocking I/O, so our
real problem is that libnbd's state machine doesn't have truly
independent h->rstate and h->wstate.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
Attachments:
- signature.asc (application/pgp-signature — 488 bytes)