[Libguestfs] Thoughts on nbdkit automatic reconnection

We have a running problem with the nbdkit VDDK plugin where the VDDK side apparently disconnects or the network connection is interrupted. During a virt-v2v conversion this causes the entire operation to fail, and since v2v conversions take many hours that's not a happy outcome. (Aside: I should say that we see many cases where it's claimed that the connection was dropped, but often when we examine them in detail the cause is something else. But it seems like this disconnection thing does happen sometimes.) To put this isn't concrete terms which don't involve v2v, let's say you were doing something like: nbdkit ssh host=remote /var/tmp/test.iso \ --run 'qemu-img convert -p -f raw $nbd -O qcow2 test.qcow2' which copies a file over ssh to local. If /var/tmp/test.iso is very large and/or the connection is very slow, and the network connection is interrupted then the whole operation fails. If nbdkit could retry/reconnect on failure then the operation might succeed. There are lots of parameters associated with retrying, eg: - how often should you retry before giving up? - how long should you wait between retries? - which errors should cause a retry, which are a hard failure? So I had an idea we could implement this as a generic "retry" filter, like: nbdkit ssh ... --filter=retry retries=5 retry-delay=5 retry-exponential=yes This cannot be implemented with the current design of filters because a filter would have to call the plugin .close and .open methods, but filters don't have access to those from regular data functions, and in any case this would cause a new plugin handle to be allocated. We could probably do it if we added a special .reopen method to plugins. We could either require plugins which support the concept of retrying to implement this, or we could have a generic implementation in server/backend.c which would call .close, .open and cope with the new handle. Another way to do this would be to modify each plugin to add the feature. nbdkit-nbd-plugin has this for a very limited case, but no others do, and it's quite complex to implement in plugins. As far as I can see it involves checking the return value of any data call that the plugin makes and performing the reconnection logic, while not changing the handle (so just calling self->close, self->open isn't going to work). If anyone has any thoughts about this I'd be happy to hear them. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html