[Libguestfs] nbdkit background threads

https://github.com/libguestfs/nbdkit/blob/ecef5b16359fb5af7e7abf4fd2fb3ad... Already existing filters (readahead, cache) could be improved if filters could open a background work thread or threads which connect independently to the plugin. A proposed new filter (scan) cannot really be written at all without this capability. First of all the reason this can't be done today is because filters are called with a next_ops structure which is only valid transiently during the filter callback. It cannot safely be saved or passed to another thread. (https://github.com/libguestfs/nbdkit/blob/master/docs/nbdkit-filter.pod#n...) The seemingly obvious implementation - which is what I tried today - would be to let filters create background threads in .config_complete. We would provide a filter API something like: int nbdkit_open_connection (struct nbdkit_next_ops **next_ops, void **nxdata, /* needs a close function */); It would return a next_ops and nxdata that the filter could then use to make data calls from the background thread into the underlying layers. To ensure safe unloading of filters and plugins we would also need a new filter callback (which I called .bg_kill) which must close all background connections opened by the filter before returning. I believe from my rough implementation that this is feasible. However I also thought about another way we might do this: We might open a loopback socket (eg. socketpair) which is passed to the filter and which the filter connects to using libnbd. nbdkit internally would treat this as if it was a regular external connection. Of course this would require libnbd as a dependency, or disable filters / features if not available. Any thoughts on this? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org