On 6/17/19 7:07 PM, Eric Blake wrote:
I've mentioned this topic before (in fact, the idea of adding NBD_CMD_FLAG_DF was first mentioned at [1]), but finally finished enough of an implementation to feel confident in posting it.
I'd still like to add something under examples/ that uses the new API to implement strict checking of a server's structured replies read implementation (ensure that a server never sends data after an error at the same offset, that it does not send overlapping data, and that it does not report success unless all of the range is covered).
For wrapping the synchronous API, this seems simple enough: int wrapped_nbd_pread(struct nbd_handle *h, void *buf, size_t count, uint64_t offset, uint32_t flags) { int r; struct data *data = prep (buf, count, offset, flags); r = nbd_pread_callback (h, buf, count, offset, data, check, flags); return cleanup (data, r); } where prep() mallocs an initial structure to record the entire range of [buf, buf+count) as not yet seen (probably by tracking a sorted linked list of consecutive ranges initially with a single list element), each call to the check() callback updates the list (as long as the callback covers only bytes that were previously unseen, the list is adjusted to remove those bytes, possible by adding or coalescing elements of the linked list if replies come out of order; if the callback includes any range already seen, then it returns -1), then cleanup() checks that the linked list was properly emptied as a final chance to convert success into an error if the server was buggy before cleaning up resources. But for wrapping asynchronous API, I'm wishing that we had a way to associate an arbitrary callback function to be called at the conclusion of any given command. The existing nbd_aio_command_completed() does not provide an easy hook for that. We could either have a single one-shot registration that gets called for all completions (or all completions of a given command type) by storing the callback in struct nbd_handle, as in: nbd_set_read_completion_hook (struct nbd_handle *h, void *opaque, int (*hook) (void *opaque, int64_t handle, int status)); Register a completion hook callback to be called on every command, right before the command will be available in nbd_aio_command_completed(). The hook is passed the command's handle obtained from nbd_aio_pread or similar, as well the status of the command so far (0 if successful, or a positive errno value); if the callback returns -1, the command will fail even if status had been 0. Or we can make the hook a per-handle choice, by adding counterpart aio APIs and storing the callback in struct command_in_flight: nbd_aio_pread_with_notify (struct nbd_handle *h, void *buf, size_t count, uint64_t offset, void *opaque, int (*notify) (void *opaque, int64_t handle, int status), uint32_t flags) where notify() is called for that specific command just before it completes. There's a small question of what to do with APIs that already take a callback (think nbd_aio_pread_callback - do we allow the user two separate opaque arguments, one for the per-chunk callback and one for the final cleanup callback, or do we require the user to tolerate the same callback in both calls?) If we really wanted, we could even copy POSIX aio and take a struct sigevent* parameter with each aio command (the existing nbd_aio_pread would be shorthand for calling a new nbd_aio_pread_event where struct sigevent is initialized to SIGEV_NONE). But that starts to feel a bit complex (SIGEV_SIGNAL may be better if we just tell the user to call pthread_signal from their own function callback). Thoughts? -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org