Re: [Libguestfs] [PATCH v2 00/11] rvh-upload: Various fixes and cleanups

On Tue, Nov 19, 2019 at 04:14:45PM +0200, Nir Soffer wrote: >On Tue, Nov 19, 2019 at 3:36 PM Martin Kletzander <mkletzan(a)redhat.com&gt; >wrote: > >> On Tue, Nov 19, 2019 at 01:48:31PM +0100, Martin Kletzander wrote: >> >On Tue, Nov 19, 2019 at 02:24:11PM +0200, Nir Soffer wrote: >> >>On Tue, Nov 19, 2019 at 2:16 PM Martin Kletzander <mkletzan(a)redhat.com&gt; >> wrote: >> >>> On Tue, Nov 19, 2019 at 01:14:23PM +0200, Nir Soffer wrote: >> >>> >On Tue, Nov 19, 2019 at 11:17 AM Martin Kletzander < >> mkletzan(a)redhat.com&gt; wrote: >> >>> >> >> >>> >> Traceback (most recent call last): >> >>> >> File "/var/tmp/rhvupload.Vw0CIU/rhv-upload-plugin.py", line 97, in >> >>> open >> >>> >> transfer_service.cancel() >> >>> >> NameError: name 'transfer_service' is not defined >> >>> > >> >>> >Does this fix the error for you? >> >>> > >> >>> >> https://github.com/nirs/virt-v2v/commit/2f93dbffad81a26445831293ecac213ea... >> >>> > >> >>> >I did not test it yet. >> >>> > >> >>> >> >>> If I remove the stray `def finalize_transfer(...)` it works, yes. >> >>> >> >> >> >>Right, belongs to the next patch. >> >> >> >> >> >>> In the meantime I managed to fix the original issue (which I needed to >> >>> revert to >> >>> test this patch =) ), thankfully it was just caused by my oVirt host >> setup. >> >>> >> >>> Unfortunately I am getting another issue. Now nbdkit fails after all >> the >> >>> data >> >>> is written and it is trying to write the disk id file. I'm getting "No >> >>> such >> >>> file or directory", so because it is writing to that file I'm assuming >> the >> >>> directory does not exist. On the other hand it *must* exists because >> it's >> >>> the >> >>> same one where the python scripts are saved. So I need to debug more. >> >>> >> >> >> >>This works for me, are you running a modified plugin? >> > >> >No, it is a current master of virt-v2v. But don't worry about it now. >> It is >> >most probably something with my setup again. Even though it worked >> before and >> >these issues started showing up after an update to virt-v2v. I'll try >> with >> >current master of nbdkit, libguestfs and virt-v2v first and if that fails >> I'll >> >try different version of virt-v2v and then start debugging it from the >> script >> >itself. It is a bit tedious as I need to modify the script in-tree, >> upload it >> >to the server, build it there, install it, then run virt-v2v-wrapper >> (that's how >> >I'm testing it now) and then inspect the logs on that remote machine. >> >> Actually looking at the whole log what I'm getting is: >> >> nbdkit: python[1]: error: /var/tmp/rhvupload.jngN1W/rhv-upload-plugin.py: >> flush: error: ['Traceback (most recent call last):\n', ' File >> "/var/tmp/rhvupload.jngN1W/rhv-upload-plugin.py", line 298, in flush\n >> http.request("PATCH", h[\'path\'], body=buf, headers=headers)\n', ' File >> "/usr/lib64/python3.7/http/client.py", line 1252, in request\n >> self._send_request(method, url, body, headers, encode_chunked)\n', ' File >> "/usr/lib64/python3.7/http/client.py", line 1263, in _send_request\n >> self.putrequest(method, url, **skips)\n', ' File >> "/usr/lib64/python3.7/http/client.py", line 1108, in putrequest\n raise >> CannotSendRequest(self.__state)\n', 'http.client.CannotSendRequest: >> Request-sent\n'] >> > >Looks like you cannot communicate with imageio daemon - this is fatal error >that should fail the >upload. >... >and then after couple of lines, when it is actually closing I get: > >> >> ... >> >> finalized after 8.060538530349731 seconds >> > >But we tried to finallize the transfer instead of cancel it. > > >> nbdkit: python[1]: error: /var/tmp/rhvupload.jngN1W/rhv-upload-plugin.py: >> close: error: ['Traceback (most recent call last):\n', ' File >> "/var/tmp/rhvupload.jngN1W/rhv-upload-plugin.py", line 362, in close\n', >> "FileNotFoundError: [Errno 2] No such file or directory: >> '/var/tmp/rhvupload.jngN1W/diskid.0'\n"] >> nbdkit: debug: python: unload plugin >> >> So it might be because virt-v2v already removed that directory and did not >> wait >> for nbdkit to completely end. I'm testing with older commit of virt-v2v >> now. >> > >flush() failure probably caused qemu-img to fai, and virt-v2v failed and >cleaned up >which is expected. > >I think the issue is incorrect error handling in the flush() and other >handlers: > >288 def flush(h): >289 http = h['http'] >290 >291 # Construct the JSON request for flushing. >292 buf = json.dumps({'op': "flush"}).encode() >293 >294 headers = {"Content-Type": "application/json", >295 "Content-Length": str(len(buf))} >296 >297 http.request("PATCH", h['path'], body=buf, headers=headers) > >If this raised (like in your traceback, we don't set h['failed'] = True > >299 r = http.getresponse() >300 if r.status != 200: >301 request_failed(h, r, "could not flush") >302 >303 r.read() > >Same here. > >Same issue in all handlers. We should handle all errors in the handlers. > >But the root cause is something else - why you cannot communicate with >imageio daemon? That is something I would love to find out, but I don't know what to try next.