[libnbd] Simultaneous read and write
by Richard W.M. Jones
This is a continuation of a discussion we were having on IRC. The
problems with IRC are it's not recorded and it's hard to have deep
technical conversations. I hope this is a decent summary.
Problem simply stated: Certain NBD servers (qemu-nbd in particular)
are able to simultaneously read and write on a socket. ie. They can
be simultaneously reading a request and writing the reply to a
previous request. However libnbd is unable to do this trick.
Although multiple requests can be in flight, libnbd is either writing
on the socket or reading from the socket, but never both.
Visualized it looks like this:
write write write
|===========|______________|=======|_______|===| --> rq to server
_____________|============|_________|=====|_____ <-- rp from server
read read
whereas an ideal libnbd which could write and read simultaneously,
coupled with a server which can do the same:
write
|===========||=======||===| --> requests to server
|============||=====||===== <-- replies from server
read
Eric already observed through testing that the ideal client can be up
to twice as fast as libnbd, as is obvious from the diagram.
The reasons why libnbd can't do this are:
Problem (a) : Only one thread of control can hold the libnbd handle
lock (h->lock), and since there can only be one thread of control
running in each handle at any time, that thread can only be reading or
writing.
Problem (b) : There is only one state machine per handle (h->state),
whereas to handle the write and read sides separately requires two
state machines. In the IRC discussion we gave these the preliminary
names h->wstate and h->rstate.
----------------------------------------------------------------------
It's worth also saying how the current API works, although we might
want to change it.
You grab the underlying file descriptor using nbd_aio_get_fd, which is
what you poll on. You also have to call nbd_aio_get_direction which
returns READ, WRITE or BOTH (== READ|WRITE). You then set up some
events mechanism (eg. poll, epoll, etc.), poll until the file
descriptor is ready, and call one of nbd_aio_notify_read or
nbd_aio_notify_write.
The direction can change any time the handle state changes, which
includes whenever you issue a command (eg. nbd_aio_pread), or whenever
you call nbd_aio_notify_*. You therefore have to call
nbd_aio_get_direction frequently.
A typical loop using poll might look like:
fd = nbd_aio_get_fd (nbd);
for (;;) {
/* <-- If you need to issue more commands, do that here. */
dir = nbd_aio_get_direction (nbd);
pollfd[0].fd = fd;
pollfd[0].events = 0;
if (dir & LIBNBD_AIO_DIRECTION_READ) pollfd[0].events |= POLLIN;
if (dir & LIBNBD_AIO_DIRECTION_WRITE) pollfd[0].events |= POLLOUT;
poll (pollfd, 1, -1);
if (pollfd[0].revents & LIBNBD_AIO_DIRECTION_READ)
nbd_aio_notify_read ();
else if (pollfd[0].revents & LIBNBD_AIO_DIRECTION_WRITE)
nbd_aio_notify_write ();
}
----------------------------------------------------------------------
The above code is of course assuming a single thread. But to
simultaneously read and write we will need at least two threads.
It's hard for me to see a nice way to evolve the API to support
multiple threads, but I guess issues include:
- If one thread is waiting in poll(2) and another thread issues a
command, how do we get the first thread to return from poll and
reevaluate direction? (Eric suggests a pipe-to-self for this)
- If two threads are waiting in poll(2) and both are notified that
the fd is ready, how do we get one of them to read and the other to
write? I think this implies that we don't have two threads doing
poll(2), but if not how do we farm out the read/write work to two
or more threads?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
5 years, 7 months
[libnbd] Some IRC questions
by Richard W.M. Jones
01:58 < eblake_> it would be nice if libnbd had a way to query if nbd support were compiled in
01:58 < eblake_> whether a constant we set in the .h file, or a runtime function
01:58 < eblake_> it would also be nice if libnbd.h declared a macro LIBNBD_VERSION
01:59 < eblake_> although my OCaml skills are too weak to figure out how to reflect something from configure.ac into the generator output
A constant defined by the generator is possible like this (but not
necessarily a good idea, see below).
(1) Create a file called something like generator/config.ml.in which
contains:
let package = "@PACKAGE_NAME@"
let version = "@PACKAGE_VERSION@"
let has_libxml2 = "@HAS_LIBXML2@" = "1"
For an example see:
https://github.com/libguestfs/supermin/blob/master/src/config.ml.in
(2) Generate config.ml using configure.
(3) Use AC_SUBST etc as appropriate.
(4) In generator/generator, add near the top:
#mod_use "config.ml"
(5) The variables can be used from the OCaml code as Config.version, etc.
However I don't think this isn't a good idea. One problem is that you
end up with the version of the library that your program compiled
against, not the version it is linked against (and similarly for other
features like URI support). So as you say above a runtime function
is better, which can be added as an ordinary API.
This is what we did in libguestfs:
http://libguestfs.org/guestfs.3.html#guestfs_version
http://libguestfs.org/guestfs.3.html#guestfs_feature_available
02:00 < eblake_> ^ make that if libxml2 support for uri parsing were compiled in
02:01 < eblake_> and the spec file needs an update: libnbd-0.1.2-1.fc29.x86_64 gives:
02:01 < eblake_> nbdkit: error: failure while creating nbd handle: nbd_connect_uri: libnbd was compiled without libxml2 support, so we do not support NBD URI:
Operation not supported
This is fixed now in the Fedora libnbd-0.1.2-2 packages.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
5 years, 7 months
[nbdkit PATCH v2 0/5] Play with libnbd for nbdkit-nbd
by Eric Blake
libnbd-0.1.2-1 is now available in Fedora 29/30 updates-testing,
although it was not compiled against libxml2 so it lacks uri support
(I ended up testing patch 4 with a self-built libnbd).
Diffs since v1 - rebase to master, bump from libnbd 0.1 to 0.1.2, add
URI support, better timing results
Still not done - patch 5 needs associated tests
Eric Blake (5):
nbd: Check for libnbd
nbd: s/nbd_/nbdplug_/
nbd: Use libnbd 0.1.2
nbd: Add magic uri=... parameter
nbd: Add TLS client support
plugins/nbd/nbdkit-nbd-plugin.pod | 102 ++-
configure.ac | 18 +
plugins/nbd/nbd-standalone.c | 1369 +++++++++++++++++++++++++++++
plugins/nbd/nbd.c | 1338 +++++++++-------------------
TODO | 13 +-
plugins/nbd/Makefile.am | 25 +-
6 files changed, 1891 insertions(+), 974 deletions(-)
create mode 100644 plugins/nbd/nbd-standalone.c
--
2.20.1
5 years, 7 months
