May 2019 - Libguestfs - libguestfs List Archives

[PATCH 0/2] libvirt: fix check of custom QEMU
by Pino Toscano 24 May '19

24 May '19

In case you configure libguestfs with a custom QEMU, e.g.: $ ./configure [...] QEMU=/path/to/qemu then the libvirt backend did not use to override it, launching the appliance with the default QEMU for libvirt. This does not change the manual emulator overriding using set-hv. Pino Toscano (2): launch: libvirt: get default QEMU from domcapabilities launch: libvirt: fix custom hypervisor check lib/launch-libvirt.c | 72 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 65 insertions(+), 7 deletions(-) -- 2.21.0

2 3

[PATCH libnbd] api: Get rid of nbd_connection.
by Richard W.M. Jones 23 May '19

23 May '19

This isn't quite finished because not all of the tests or examples have been updated, but it demonstrates an idea: Should we forget about the concept of having multiple connections managed under a single handle? In this patch there is a single ‘struct nbd_handle *’ which manages a single state machine and connection (and therefore no nbd_connection). To connect to a multi-conn server you must do something like: - Connect once, check if the server supports multi-conn. - If it does, either disconnect and connect mulitple times "for real", or else open further handles as required. An example of use of the new API can be found in examples/threaded-reads-and-writes.c after this patch is applied. Rich.

2 2

[PATCH libnbd 0/3] Prevent some misuse of multi-conn.
by Richard W.M. Jones 23 May '19

23 May '19

Per recent discussion here: https://www.redhat.com/archives/libguestfs/2019-May/thread.html#00175

2 5

The Outline of Google Summer of Code
by Hiroyuki Katsura 23 May '19

23 May '19

Hi, I'm Hiroyuki Katsura. I will contribute to libguestfs as a Google Summer of Code project. I'll start to develop from next week. I'm going to work on the project "Create Rust bindings for libguestfs." Project summary is available here. https://summerofcode.withgoogle.com/projects/#6730007286644736 Now, I'll show you the detail of the project. This is the reprint of the proposal(The proposal is not public). ## The detailed description of each part 1. Implement minimal bindings and build it correctly. There are some functions to be implemented manually. Typical examples of this type of functions are the ones which create or close libguestfs handles. 2. Handle errors. The best practice of handling errors varies between languages. For example, error handling is implemented with exceptions in OCaml, while in Lua there is an error stack which is easily pop the last error from it. In order to make bindings easier to use, error handling should be implemented in the way following the custom of each language. In Rust, it is preferable to use Result and error enum, so I adopt this style. 3. Implement the generator of Actions(APIs) and Structs. This is the main part of this project. This can be achieved by creating the "generator" described above. 4. Implement handlers related to Events. Events are a tricky part of the bindings. Users can receive some events like LaunchDone, Progress, SubprocessQuit, etc. from libguestfs by registering a callback. Although not all the languages have this functionality, I will implement this to the Rust bindings. 5. Add documents and examples. In order to make the bindings easier to use for developers, it is necessary to add documents and examples of the bindings. 6. Implement tests. Tests are essential to keep the bindings working in the future and to prevent it from being degraded when someone adds new features to the bindings or APIs. In libguestfs, when you add a new binding, it is preferred to add the same tests as the ones implemented in OCaml or Perl bindings. This means that I will implement 0XX tests defined in tests of OCaml or Perl bindings. ------------------------------------------------------------------------------- I am planning to send the patch 3 times: after implementing the generator of Actions(APIs) and Structs, after implementing the events and miscellaneous, after completion of the project. Regards, Hiroyuki -------------------------- Hiroyuki Katsura Department of Information Science Science University of Tokyo.

1 0

[PATCH libnbd v2 0/6] Test connection states.
by Richard W.M. Jones 23 May '19

23 May '19

Patch 1/6 was posted before and I didn't change it: https://www.redhat.com/archives/libguestfs/2019-May/thread.html#00134 That doesn't necessarily mean I shouldn't change it, I'm posting it again because the other patches depend on it. The main change in this series is we add three new API functions: nbd_aio_is_created - connection has just been created nbd_aio_is_connecting - is connecting or handshaking / option negotiation nbd_aio_is_processing - is issuing or receiving a reply Also in this change I used nbd_aio_is_connecting to simplify the handling of synchronous connects. Rich.

2 10

[libnbd PATCH v3 0/7] Avoid deadlock with in-flight commands
by Eric Blake 22 May '19

22 May '19

Since v2: - rebase to Rich's new API calls - more refactoring in patch 1 (retitled) - new patches 3 and 4 - fix data corruption in patch 6 (was 4) - more tweaks to the reproducer example (including using new API from 3) Eric Blake (7): lib: Refactor command_common() to do more common work commands: Allow for a command queue commands: Expose FIFO ordering of server completions disconnect: Allow shutdown during processing states: Split ISSUE_COMMAND.SEND_REQUEST states: Allow in-flight read while writing next command examples: Add example to demonstrate just-fixed deadlock scenario .gitignore | 1 + examples/Makefile.am | 10 ++ examples/batched-read-write.c | 214 +++++++++++++++++++++++++++++++ generator/generator | 52 ++++++-- generator/states-issue-command.c | 85 ++++++++---- generator/states-reply.c | 13 +- generator/states.c | 5 + lib/aio.c | 13 ++ lib/disconnect.c | 23 +--- lib/internal.h | 14 +- lib/rw.c | 131 ++++++++----------- 11 files changed, 429 insertions(+), 132 deletions(-) create mode 100644 examples/batched-read-write.c -- 2.20.1

2 12

[libnbd PATCH] interop: Don't fail qemu tls testing if nbdkit was not installed
by Eric Blake 22 May '19

22 May '19

On Fedora 29, nbdkit-devel at 1.10.4 is too old for any of the tests/Makefile.am actions to run, but this also means we don't create any tls pki/ or keys.psk files that we then want to reuse during qemu testing, as evidenced by 'make check' failures such as: FAIL: interop-qemu-nbd-tls-certs ================================ qemu-nbd: Unable to access credentials /home/eblake/libnbd/tests/pki/ca-cert.pem: No such file or directory For now, just hack out the qemu tests except under the same conditions as the tls files get created. A better patch would generate the tls files independently of nbdkit and/or qemu availability. Hence I'm just throwing this out for discussion rather than actually pushing it. --- interop/Makefile.am | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/interop/Makefile.am b/interop/Makefile.am index 16438dc..b0ee5a3 100644 --- a/interop/Makefile.am +++ b/interop/Makefile.am @@ -45,9 +45,22 @@ check_PROGRAMS += \ interop-qemu-nbd-tls-certs \ interop-qemu-nbd-tls-psk TESTS += \ - interop-qemu-nbd \ - interop-qemu-nbd-tls-certs \ + interop-qemu-nbd + +# tls tests assume the pre-existence of files created in ../tests/Makefile.am, +# so we can only run them under the same conditions used there +if HAVE_NBDKIT +if HAVE_GNUTLS +if HAVE_CERTTOOL +TESTS += \ + interop-qemu-nbd-tls-certs +endif +if HAVE_PSKTOOL +TESTS += \ interop-qemu-nbd-tls-psk +endif +endif +endif interop_qemu_nbd_SOURCES = interop.c interop_qemu_nbd_CPPFLAGS = \ -- 2.20.1

2 1

[libnbd PATCH v2 0/5] Avoid deadlock with in-flight commands
by Eric Blake 22 May '19

22 May '19

On v1, we discussed whether cmds_to_issue needed to be a list, since it never had more than one element. I played with the idea of making it a list, and allowing the client to queue up new commands regardless of whether the state machine is currently in READY. I also polished up the tmp demo into a bit more full-fledged example file, worth including since it also let me discover a hard-to-hit race with large NBD_CMD_WRITE vs. EAGAIN failures (now fixed). Eric Blake (5): lib: Refactor state event into command_common commands: Allow for a command queue states: Split ISSUE_COMMAND.SEND_REQUEST states: Allow in-flight read while writing next command examples: Add example to demonstrate just-fixed deadlock scenario .gitignore | 1 + examples/Makefile.am | 10 ++ examples/batched-read-write.c | 199 +++++++++++++++++++++++++++++++ generator/generator | 40 +++++-- generator/states-issue-command.c | 68 ++++++++--- generator/states.c | 6 + lib/disconnect.c | 17 +-- lib/internal.h | 10 ++ lib/rw.c | 41 ++++--- 9 files changed, 334 insertions(+), 58 deletions(-) create mode 100644 examples/batched-read-write.c -- 2.20.1

2 12

[PATCH libnbd] api: Synchronous connect waits til all connections are connected.
by Richard W.M. Jones 22 May '19

22 May '19

nbd_connect_unix|tcp had a tricky failure case. This is a consequence of allowing callers to mix synchronous and asynchronous calls, with multi-conn thrown into the mix. I think the new behaviour proposed here is better. We could do with a better way of classifying the state of connections, such as are they connectING. Rich.

2 4

[libnbd PATCH 0/3] Avoid deadlock with in-flight commands
by Eric Blake 21 May '19

21 May '19

This might not be the final solution, but it certainly seems to solve a deadlock for me that I could trigger by using 'nbdkit --filter=noparallel memory 512k' and calling nbd_aio_pread for a request larger than 256k (enough for the Linux kernel to block the server until libnbd read()s), immediately followed by nbd_aio_pwrite for a request larger than 256k (enough to block libnbd until the server read()s, but the serialized server won't read until we parse off the reply). My solution was to allow a notifyread at any time we are in the middle of writing a request, at which point we pause the current write, force the state machine to completely receive the reply, then resume where we left off writing the request. Eric Blake (3): commands: Preserve FIFO ordering states: Split ISSUE_COMMAND.SEND_REQUEST states: Allow in-flight read while writing next command generator/generator | 34 +++++++++++++++- generator/states-issue-command.c | 68 ++++++++++++++++++++++++-------- generator/states-reply.c | 18 +++++++-- lib/internal.h | 1 + lib/rw.c | 13 ++++-- 5 files changed, 108 insertions(+), 26 deletions(-) -- 2.20.1

2 11