Re: [Libguestfs] Dealing with ImageIO problems
[Adding the mailing list address back] On Thu, Nov 21, 2019 at 12:44:38PM +0200, Nir Soffer wrote:
On Thu, Nov 21, 2019 at 12:16 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Nov 20, 2019 at 06:00:52PM +0200, Nir Soffer wrote:
5. can_zero fails, not sure why (Maybe Richard knows more about this)
nbdkit: python[1]: debug: python: can_zero nbdkit: python[1]: debug: sending error reply: Operation not supported
rhv-upload-plugin always reports that it can zero, I cannot explain this.
I believe this is a bug in nbdkit’s python bindings. Will take a closer look and reply in a new thread. I don't believe it affects anything here however.
6. writing the first sector fails
python[1]: nbdkit: debug: vddk[3]python: pwrite count=65536 offset=0 fua=0: ... nbdkit: python[1]: error: /var/tmp/rhvupload.RShGz0/rhv-upload-plugin.py: pwrite: error: ['Traceback (most recent call last):\n', ' File "/var/tmp/rhvupload.RShGz0/rhv-upload-plugin.py", line 189, in pwrite\n http.endheaders()\n', ' File "/usr/lib64/python3.7/http/client.py", line 1247, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n', ' File "/usr/lib64/python3.7/http/client.py", line 1026, in _send_output\n self.send(msg)\n', ' File "/usr/lib64/python3.7/http/client.py", line 966, in send\n self.connect()\n', ' File "/var/tmp/rhvupload.RShGz0/rhv-upload-plugin.py", line 357, in connect\n self.sock.connect(self.path)\n', 'ConnectionRefusedError: [Errno 111] Connection refused\n']
But it fails in send(), and we don't handle errors proeply around send(), only around getresponse(). So the error is forgotten by the plugin.
That sounds bad?
Indeed, but we never had this issue in real environment.
Basically we should handle any exception in the plugin as failure, but I'm not sure where we should fix it.
We can add error handler around all the entry points (e.g. pwrite()), setting h['failed'] = True. But this makes it harder to write correct plugins; everyone has to ensure correct error handling instead of getting it right by default.
In nbdkit-python-plugin, any exception which escapes will be caught in the C code and turned into an NBD error on the wire. The function which does this is: https://github.com/libguestfs/nbdkit/blob/a52ebb4e8d18a73b9ae009cd768bfab185... Since each plugin method corresponds* (* = mostly) to a single NBD command on the wire, returning an error from the method signals an error back to the NBD client. If the NBD client is ‘qemu-img convert’ then it will exit the first time it sees an error. (The retry filter http://libguestfs.org/nbdkit-retry-filter.1.html changes this behaviour, but we are not using that for output to rhv-upload.)
We can handle this in the python plugin (e.g. h['failed'] = True) but the handle is controlled by the plugin, so this does not sound like a good way.
The close() method is always called, whether or not there was an error. The same connection handle (‘h’) is passed back to the close method. It's quite safe and normal to store state in this handle. nbdkit itself doesn't save the fact that there was an error during the connection (it can't since it doesn't know if the error was recoverable or not). But rhv-upload-plugin needs to handle failed vs successful transfers differently. Thus we save the failure state in the handle the first time we see an error.
Maybe we can change close() to:
def close(failed=True): if failed: # cleanup after some call failed ...
# Cleanup after successful run
The python plugin will remember unhandled errors and will call:
close(failed=True)
To write correct plugin author must not handle fatal errors in the plugin.
This changes how the Python plugin would work versus other plugins, and the C code can't know if the NBD connection failed, because some NBD errors are recoverable. Also the client does not send us any error indication when it disconnects. The answer is probably to wrap every method in a big try...except which sets the failed flag in the handle.
How do we handle errors in other plugins?
It's kind of easier to study this using the sh plugin. Here's an example showing the order in which callbacks happen on error: ---------------------------------------------------------------------- nbdkit sh -fv - --run 'qemu-img convert $nbd /tmp/out' <<'EOF' case "$1" in get_size) echo 512 ;; pread) echo 'EIO error' >&2; exit 1 ;; *) exit 2 ;; esac EOF ---------------------------------------------------------------------- Running this gives the following output (filtered a bit to show the key points): $ bash test.sh nbdkit: debug: sh: load nbdkit: debug: sh: load: tmpdir: /tmp/nbdkitshB6KCs0 nbdkit: debug: sh: config key=script, value=- nbdkit: debug: sh: config_complete nbdkit: sh[1]: debug: sh: open readonly=0 nbdkit: sh[1]: debug: calling: /tmp/nbdkitshB6KCs0/inline-script.sh open false "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitshB6KCs0/inline-script.sh open: status 2 nbdkit: sh[1]: debug: sh: open returned handle 0x7face4002070 nbdkit: sh[1]: debug: sh: get_size nbdkit: sh[1]: debug: calling: /tmp/nbdkitshB6KCs0/inline-script.sh get_size "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitshB6KCs0/inline-script.sh get_size: status 0 nbdkit: sh[1]: debug: sh: can_write nbdkit: sh[1]: debug: calling: /tmp/nbdkitshB6KCs0/inline-script.sh can_write "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitshB6KCs0/inline-script.sh can_write: status 2 ... nbdkit: sh[1]: debug: sh: pread count=512 offset=0 nbdkit: sh[1]: debug: calling: /tmp/nbdkitshB6KCs0/inline-script.sh pread "" 512 0 nbdkit: sh[1]: debug: completed: /tmp/nbdkitshB6KCs0/inline-script.sh pread: status 1 nbdkit: sh[1]: error: /tmp/nbdkitshB6KCs0/inline-script.sh: error nbdkit: sh[1]: debug: sending error reply: Input/output error nbdkit: sh[1]: debug: client sent NBD_CMD_DISC, closing connection qemu-img: Could not open 'nbd:localhost:10809': Could not read image for determining its format: Input/output error nbdkit: sh[1]: debug: sh: close nbdkit: sh[1]: debug: calling: /tmp/nbdkitshB6KCs0/inline-script.sh close "" nbdkit: sh[1]: debug: completed: /tmp/nbdkitshB6KCs0/inline-script.sh close: status 2 nbdkit: debug: sh: unload plugin nbdkit: debug: calling: /tmp/nbdkitshB6KCs0/inline-script.sh unload nbdkit: debug: completed: /tmp/nbdkitshB6KCs0/inline-script.sh unload: status 2 Notice in particular how close() and unload() are both called in the plugin even though qemu-img convert has already exited with an error. Also the same handle (empty string above) is passed back to close().
We probably should continue to the mailing list.
Nir
Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Thu, Nov 21, 2019 at 1:39 PM Richard W.M. Jones <rjones@redhat.com> wrote:
[Adding the mailing list address back]
On Thu, Nov 21, 2019 at 12:44:38PM +0200, Nir Soffer wrote:
On Thu, Nov 21, 2019 at 12:16 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Nov 20, 2019 at 06:00:52PM +0200, Nir Soffer wrote:
5. can_zero fails, not sure why (Maybe Richard knows more about this)
nbdkit: python[1]: debug: python: can_zero nbdkit: python[1]: debug: sending error reply: Operation not supported
rhv-upload-plugin always reports that it can zero, I cannot explain this.
I believe this is a bug in nbdkit’s python bindings. Will take a closer look and reply in a new thread. I don't believe it affects anything here however.
6. writing the first sector fails
python[1]: nbdkit: debug: vddk[3]python: pwrite count=65536 offset=0 fua=0: ... nbdkit: python[1]: error: /var/tmp/rhvupload.RShGz0/rhv-upload-plugin.py: pwrite: error: ['Traceback (most recent call last):\n', ' File "/var/tmp/rhvupload.RShGz0/rhv-upload-plugin.py", line 189, in pwrite\n http.endheaders()\n', ' File "/usr/lib64/python3.7/http/client.py", line 1247, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n', ' File "/usr/lib64/python3.7/http/client.py", line 1026, in _send_output\n self.send(msg)\n', ' File "/usr/lib64/python3.7/http/client.py", line 966, in send\n self.connect()\n', ' File "/var/tmp/rhvupload.RShGz0/rhv-upload-plugin.py", line 357, in connect\n self.sock.connect(self.path)\n', 'ConnectionRefusedError: [Errno 111] Connection refused\n']
But it fails in send(), and we don't handle errors proeply around send(), only around getresponse(). So the error is forgotten by the plugin.
That sounds bad?
Indeed, but we never had this issue in real environment.
Basically we should handle any exception in the plugin as failure, but I'm not sure where we should fix it.
We can add error handler around all the entry points (e.g. pwrite()), setting h['failed'] = True. But this makes it harder to write correct plugins; everyone has to ensure correct error handling instead of getting it right by default.
In nbdkit-python-plugin, any exception which escapes will be caught in the C code and turned into an NBD error on the wire. The function which does this is:
https://github.com/libguestfs/nbdkit/blob/a52ebb4e8d18a73b9ae009cd768bfab185...
Since each plugin method corresponds* (* = mostly) to a single NBD command on the wire, returning an error from the method signals an error back to the NBD client. If the NBD client is ‘qemu-img convert’ then it will exit the first time it sees an error.
(The retry filter http://libguestfs.org/nbdkit-retry-filter.1.html changes this behaviour, but we are not using that for output to rhv-upload.)
We can handle this in the python plugin (e.g. h['failed'] = True) but the handle is controlled by the plugin, so this does not sound like a good way.
The close() method is always called, whether or not there was an error. The same connection handle (‘h’) is passed back to the close method. It's quite safe and normal to store state in this handle.
nbdkit itself doesn't save the fact that there was an error during the connection (it can't since it doesn't know if the error was recoverable or not). But rhv-upload-plugin needs to handle failed vs successful transfers differently. Thus we save the failure state in the handle the first time we see an error.
Maybe we can change close() to:
def close(failed=True): if failed: # cleanup after some call failed ...
# Cleanup after successful run
The python plugin will remember unhandled errors and will call:
close(failed=True)
To write correct plugin author must not handle fatal errors in the plugin.
This changes how the Python plugin would work versus other plugins, and the C code can't know if the NBD connection failed, because some NBD errors are recoverable. Also the client does not send us any error indication when it disconnects.
The answer is probably to wrap every method in a big try...except which sets the failed flag in the handle.
Implemented in https://www.redhat.com/archives/libguestfs/2019-November/msg00147.html This works with current plugin, and can be backported to downstream if needed. However it makes it harder to write plugins in python. The basic idea is that not handling an error in python will crash with a traceback *without* having to do anything. It would be best if the python plugin ensure this. If we want to support recoverable errors, we can define a special RecoverableError exception that signal the python plugin that this request failed, but the error is not fatal. We can also define more specific errors like UnsupportedOperation, for example if plugin's zero() fail and the author wants nbdkit to stop calling zero() and emulate it with write. Nir
On Thu, Nov 21, 2019 at 05:50:18PM +0200, Nir Soffer wrote:
If we want to support recoverable errors, we can define a special RecoverableError exception that signal the python plugin that this request failed, but the error is not fatal.
The problem is only the client knows if the error was recovered from. It's a pretty complicated problem and it seems we've got a good enough way to handle it from the rhv-upload-plugin now. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Thu, Nov 21, 2019 at 6:24 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Nov 21, 2019 at 05:50:18PM +0200, Nir Soffer wrote:
If we want to support recoverable errors, we can define a special RecoverableError exception that signal the python plugin that this request failed, but the error is not fatal.
The problem is only the client knows if the error was recovered from.
Good point.
It's a pretty complicated problem and it seems we've got a good enough way to handle it from the rhv-upload-plugin now.
I agree that what we have is good enough, but it has some limitations: - All operations are considered as fatal error in rhv-upload - Even if client retry an operation and recovery from errors, the plugin will fail the import I think the issue is mixing control path and data path operations. We do: - on open: create disk and transfer - on pwrite/zero/flush - translate request to imageio - on close: finalize or cancel the transfer While creating a disk in open matches the semantics of open() (e.g. O_CREAT), canceling a transfer (deleting a disk) does not match the semantics of close(). The real fix for this would be: - create a disk and transfer *before* running qemu-img and rhv-upload-plugin (after we do the pre-checks) - run qemu-img to transfer the data using the new rhv-upload-plugin - if qemu failed, cancel the transfer and fail the import - otherwise finalize the transfer and create the vm After the recent refactoring in rhv-upload-plugin, this will be an easy change. The relevant code is from: https://github.com/libguestfs/virt-v2v/blob/9fca8345176d88934cf1f351633e5149... to: https://github.com/libguestfs/virt-v2v/blob/9fca8345176d88934cf1f351633e5149... Once we have this, we can easily switch to NBD instead of imageio for better performance, but keep both options to support older systems (4.3). Future (4.4?) version of ovirt will support transport=HTTP/NBD argument when creating a transfer. You will get NBD URL instead of HTTPS URL in transfer_url. v2v will use the NBD URL directly without nbdkit/rhv-upload-plugin. Here is the flow for transport=NBD: - create a disk and transfer with transport=NBD - run qemu-img convert ... overlay.qcow2 nbd://?socket=/path/to/nbd.sock - if qemu fails, cancel the transfer and fail the import - otherwise finalize the transfer and create the vm We can also check if ovirt supports transport=NBD, and fallback to HTTP if not. Later we can add TLS support so we can import from another host using NBD TCP socket. This requires much more work so I'm not sue it can be in 4.4.0, but probably 4.4.z. Nir
participants (2)
-
Nir Soffer -
Richard W.M. Jones