June 2019 - Libguestfs - libguestfs List Archives

[libnbd] Simultaneous read and write
by Richard W.M. Jones 03 Jun '19

03 Jun '19

This is a continuation of a discussion we were having on IRC. The problems with IRC are it's not recorded and it's hard to have deep technical conversations. I hope this is a decent summary. Problem simply stated: Certain NBD servers (qemu-nbd in particular) are able to simultaneously read and write on a socket. ie. They can be simultaneously reading a request and writing the reply to a previous request. However libnbd is unable to do this trick. Although multiple requests can be in flight, libnbd is either writing on the socket or reading from the socket, but never both. Visualized it looks like this: write write write |===========|______________|=======|_______|===| --> rq to server _____________|============|_________|=====|_____ <-- rp from server read read whereas an ideal libnbd which could write and read simultaneously, coupled with a server which can do the same: write |===========||=======||===| --> requests to server |============||=====||===== <-- replies from server read Eric already observed through testing that the ideal client can be up to twice as fast as libnbd, as is obvious from the diagram. The reasons why libnbd can't do this are: Problem (a) : Only one thread of control can hold the libnbd handle lock (h->lock), and since there can only be one thread of control running in each handle at any time, that thread can only be reading or writing. Problem (b) : There is only one state machine per handle (h->state), whereas to handle the write and read sides separately requires two state machines. In the IRC discussion we gave these the preliminary names h->wstate and h->rstate. ---------------------------------------------------------------------- It's worth also saying how the current API works, although we might want to change it. You grab the underlying file descriptor using nbd_aio_get_fd, which is what you poll on. You also have to call nbd_aio_get_direction which returns READ, WRITE or BOTH (== READ|WRITE). You then set up some events mechanism (eg. poll, epoll, etc.), poll until the file descriptor is ready, and call one of nbd_aio_notify_read or nbd_aio_notify_write. The direction can change any time the handle state changes, which includes whenever you issue a command (eg. nbd_aio_pread), or whenever you call nbd_aio_notify_*. You therefore have to call nbd_aio_get_direction frequently. A typical loop using poll might look like: fd = nbd_aio_get_fd (nbd); for (;;) { /* <-- If you need to issue more commands, do that here. */ dir = nbd_aio_get_direction (nbd); pollfd[0].fd = fd; pollfd[0].events = 0; if (dir & LIBNBD_AIO_DIRECTION_READ) pollfd[0].events |= POLLIN; if (dir & LIBNBD_AIO_DIRECTION_WRITE) pollfd[0].events |= POLLOUT; poll (pollfd, 1, -1); if (pollfd[0].revents & LIBNBD_AIO_DIRECTION_READ) nbd_aio_notify_read (); else if (pollfd[0].revents & LIBNBD_AIO_DIRECTION_WRITE) nbd_aio_notify_write (); } ---------------------------------------------------------------------- The above code is of course assuming a single thread. But to simultaneously read and write we will need at least two threads. It's hard for me to see a nice way to evolve the API to support multiple threads, but I guess issues include: - If one thread is waiting in poll(2) and another thread issues a command, how do we get the first thread to return from poll and reevaluate direction? (Eric suggests a pipe-to-self for this) - If two threads are waiting in poll(2) and both are notified that the fd is ready, how do we get one of them to read and the other to write? I think this implies that we don't have two threads doing poll(2), but if not how do we farm out the read/write work to two or more threads? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

2 4

[libnbd] Some IRC questions
by Richard W.M. Jones 02 Jun '19

02 Jun '19

01:58 < eblake_> it would be nice if libnbd had a way to query if nbd support were compiled in 01:58 < eblake_> whether a constant we set in the .h file, or a runtime function 01:58 < eblake_> it would also be nice if libnbd.h declared a macro LIBNBD_VERSION 01:59 < eblake_> although my OCaml skills are too weak to figure out how to reflect something from configure.ac into the generator output A constant defined by the generator is possible like this (but not necessarily a good idea, see below). (1) Create a file called something like generator/config.ml.in which contains: let package = "@PACKAGE_NAME@" let version = "@PACKAGE_VERSION@" let has_libxml2 = "@HAS_LIBXML2@" = "1" For an example see: https://github.com/libguestfs/supermin/blob/master/src/config.ml.in (2) Generate config.ml using configure. (3) Use AC_SUBST etc as appropriate. (4) In generator/generator, add near the top: #mod_use "config.ml" (5) The variables can be used from the OCaml code as Config.version, etc. However I don't think this isn't a good idea. One problem is that you end up with the version of the library that your program compiled against, not the version it is linked against (and similarly for other features like URI support). So as you say above a runtime function is better, which can be added as an ordinary API. This is what we did in libguestfs: http://libguestfs.org/guestfs.3.html#guestfs_version http://libguestfs.org/guestfs.3.html#guestfs_feature_available 02:00 < eblake_> ^ make that if libxml2 support for uri parsing were compiled in 02:01 < eblake_> and the spec file needs an update: libnbd-0.1.2-1.fc29.x86_64 gives: 02:01 < eblake_> nbdkit: error: failure while creating nbd handle: nbd_connect_uri: libnbd was compiled without libxml2 support, so we do not support NBD URI: Operation not supported This is fixed now in the Fedora libnbd-0.1.2-2 packages. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

1 1

[nbdkit PATCH v2 0/5] Play with libnbd for nbdkit-nbd
by Eric Blake 02 Jun '19

02 Jun '19

libnbd-0.1.2-1 is now available in Fedora 29/30 updates-testing, although it was not compiled against libxml2 so it lacks uri support (I ended up testing patch 4 with a self-built libnbd). Diffs since v1 - rebase to master, bump from libnbd 0.1 to 0.1.2, add URI support, better timing results Still not done - patch 5 needs associated tests Eric Blake (5): nbd: Check for libnbd nbd: s/nbd_/nbdplug_/ nbd: Use libnbd 0.1.2 nbd: Add magic uri=... parameter nbd: Add TLS client support plugins/nbd/nbdkit-nbd-plugin.pod | 102 ++- configure.ac | 18 + plugins/nbd/nbd-standalone.c | 1369 +++++++++++++++++++++++++++++ plugins/nbd/nbd.c | 1338 +++++++++------------------- TODO | 13 +- plugins/nbd/Makefile.am | 25 +- 6 files changed, 1891 insertions(+), 974 deletions(-) create mode 100644 plugins/nbd/nbd-standalone.c -- 2.20.1

1 5