[PATCH v2 0/3] v2v: -o rhv-upload: Add a test.
v1 was here: https://www.redhat.com/archives/libguestfs/2018-September/msg00121.html v2: - Rewrote patch 2 from scratch so it incorporates Nir's suggestions. - Add fake module to EXTRA_DIST. - Retested. Unfortunately I am no longer able to test the ordinary conversion path because ovirtsdk4 is incompatible with Fedora 29 / Python 3.7: https://bugzilla.redhat.com/show_bug.cgi?id=1622043 so I'm unable to verify that the changes (esp patch 2) don't break this. Rich.
--- v2v/output_rhv_upload.ml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml index f03e1ede3..3bb2ca807 100644 --- a/v2v/output_rhv_upload.ml +++ b/v2v/output_rhv_upload.ml @@ -117,6 +117,8 @@ class output_rhv_upload output_alloc output_conn (* Check that nbdkit is available and new enough. *) let error_unless_nbdkit_working () = + let cmd = "nbdkit --version >/dev/null" in + debug "%s" cmd; if 0 <> Sys.command "nbdkit --version >/dev/null" then error (f_"nbdkit is not installed or not working. It is required to use ‘-o rhv-upload’. See \"OUTPUT TO RHV\" in the virt-v2v(1) manual."); @@ -139,6 +141,7 @@ class output_rhv_upload output_alloc output_conn let cmd = sprintf "nbdkit %s %s --dump-plugin >/dev/null" Python_script.python (quote (Python_script.path plugin_script)) in + debug "%s" cmd; if Sys.command cmd <> 0 then error (f_"nbdkit %s plugin is not installed or not working. It is required if you want to use ‘-o rhv-upload’. -- 2.19.0.rc0
For real imageio servers the destination will always be https. This change has no effect there. However when testing we want to use an http server for simplicity. As there is no certificate or cafile in this case the call to create the context will fail. This also simplifies creation of the context object and recognizes the "insecure" flag for connecting to imageio. Thanks: Nir Soffer. --- v2v/rhv-upload-plugin.py | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 5cd6d5cab..1a217b6dc 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -207,14 +207,25 @@ def open(readonly): else: destination_url = urlparse(transfer.proxy_url) - context = ssl.create_default_context() - context.load_verify_locations(cafile = params['rhv_cafile']) - - http = HTTPSConnection( - destination_url.hostname, - destination_url.port, - context = context - ) + if destination_url.scheme == "https": + context = \ + ssl.create_default_context(purpose = ssl.Purpose.SERVER_AUTH, + cafile = cafile) + if params['insecure']: + context.check_hostname = False + context.verify_mode = ssl.CERT_NONE + http = HTTPSConnection( + destination_url.hostname, + destination_url.port, + context = context + ) + elif destination_url.scheme == "http": + http = HTTPConnection( + destination_url.hostname, + destination_url.port, + ) + else: + raise RuntimeError("unknown URL scheme (%s)" % destination_url.scheme) # The first request is to fetch the features of the server. -- 2.19.0.rc0
On Thu, Sep 20, 2018 at 11:51 AM Richard W.M. Jones <rjones@redhat.com> wrote:
For real imageio servers the destination will always be https. This change has no effect there.
However when testing we want to use an http server for simplicity. As there is no certificate or cafile in this case the call to create the context will fail.
This also simplifies creation of the context object and recognizes the "insecure" flag for connecting to imageio.
Thanks: Nir Soffer. --- v2v/rhv-upload-plugin.py | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 5cd6d5cab..1a217b6dc 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -207,14 +207,25 @@ def open(readonly): else: destination_url = urlparse(transfer.proxy_url)
- context = ssl.create_default_context() - context.load_verify_locations(cafile = params['rhv_cafile']) - - http = HTTPSConnection( - destination_url.hostname, - destination_url.port, - context = context - ) + if destination_url.scheme == "https": + context = \ + ssl.create_default_context(purpose = ssl.Purpose.SERVER_AUTH, + cafile = cafile) + if params['insecure']: + context.check_hostname = False + context.verify_mode = ssl.CERT_NONE + http = HTTPSConnection( + destination_url.hostname, + destination_url.port, + context = context + ) + elif destination_url.scheme == "http": + http = HTTPConnection( + destination_url.hostname, + destination_url.port, + ) + else: + raise RuntimeError("unknown URL scheme (%s)" % destination_url.scheme)
I would not change production code to support http. Instead the test server should use HTTPS. This way we may have working tests when the real code path fail during runtime. Nir
# The first request is to fetch the features of the server.
-- 2.19.0.rc0
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
Previously this output method was almost completely untested. This commit adds a fake ovirtsdk4 module so we can test the -o rhv-upload method fairly completely without needing an actual oVirt instance around. --- v2v/Makefile.am | 4 + .../ovirtsdk4/__init__.py | 146 ++++++++++++++++++ .../ovirtsdk4/types.py | 125 +++++++++++++++ v2v/test-v2v-o-rhv-upload.sh | 51 ++++++ 4 files changed, 326 insertions(+) diff --git a/v2v/Makefile.am b/v2v/Makefile.am index 14183572b..aab356637 100644 --- a/v2v/Makefile.am +++ b/v2v/Makefile.am @@ -382,6 +382,7 @@ TESTS += \ test-v2v-o-openstack.sh \ test-v2v-o-qemu.sh \ test-v2v-o-rhv.sh \ + test-v2v-o-rhv-upload.sh \ test-v2v-o-vdsm-options.sh \ test-v2v-oa-option.sh \ test-v2v-of-option.sh \ @@ -539,6 +540,9 @@ EXTRA_DIST += \ test-v2v-o-qemu.sh \ test-v2v-o-rhv.ovf.expected \ test-v2v-o-rhv.sh \ + test-v2v-o-rhv-upload.sh \ + test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py \ + test-v2v-o-rhv-upload-module/ovirtsdk4/types.py \ test-v2v-o-rhv-upload-oo-query.sh \ test-v2v-o-vdsm-oo-query.sh \ test-v2v-o-vdsm-options.ovf.expected \ diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py new file mode 100644 index 000000000..2ddb950ea --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py @@ -0,0 +1,146 @@ +# -*- python -*- +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Fake ovirtsdk4 module used as a test harness. +# See v2v/test-v2v-o-rhv-upload.sh + +class Error(Exception): + pass +class NotFoundError(Error): + pass + +class Connection(object): + def __init__( + self, + url = None, + username = None, + password = None, + ca_file = None, + log = None, + insecure = False, + ): + pass + + def system_service(self): + return SystemService() + +class SystemService(object): + def data_centers_service(self): + return DataCentersService() + + def disks_service(self): + return DisksService() + + def image_transfers_service(self): + return ImageTransfersService() + + def storage_domains_service(self): + return StorageDomainsService() + + def vms_service(self): + return VmsService() + +class DataCentersService(object): + def list(self, search=None, case_sensitive=False): + return [] + +class DiskService(object): + def __init__(self, disk_id): + self._disk_id = disk_id + + def get(self): + return types.Disk() + + def remove(self): + pass + +class DisksService(object): + def add(self, disk=None): + return disk + + def disk_service(self, disk_id): + return DiskService(disk_id) + +class ImageTransferService(object): + def __init__(self): + self._finalized = False + + def get(self): + if self._finalized: + raise NotFoundError + else: + return types.ImageTransfer() + + def finalize(self): + self._finalized = True + +class ImageTransfersService(object): + def add(self, transfer): + return transfer + + def image_transfer_service(self, id): + return ImageTransferService() + +class StorageDomain(object): + id = "ba87af68-b630-4211-a73a-694c1a689405" + +class StorageDomainsService(object): + def list(self, search=None): + return [ StorageDomain() ] + +class VmsService(object): + def add(self, vm): + return vm + + def list(self, search=None): + return [] + +# Create a background thread running a web server which is +# simulating the imageio server. + +from http.server import HTTPServer, BaseHTTPRequestHandler +import random +import threading + +# Choose a random port number in range [50000,59999] +imageio_port = random.randint(50000,60000) + +class RequestHandler(BaseHTTPRequestHandler): + def do_OPTIONS(self): + self.send_response(200) + self.send_header("Content-type", "application/json; charset=UTF-8") + self.end_headers() + # Advertize only zero support. + self.wfile.write(b'''{ "features": [ "zero" ] }''') + + # eg. zero request. Just ignore it. + def do_PATCH(self): + self.send_response(200) + self.end_headers() + + # Flush request. Ignore it. + def do_PUT(self): + self.send_response(200) + self.end_headers() + +def server(): + server_address = ("", imageio_port) + httpd = HTTPServer(server_address, RequestHandler) + httpd.serve_forever() + +thread = threading.Thread(target = server, args = [], daemon = True) +thread.start() diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py new file mode 100644 index 000000000..9b3f557ee --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py @@ -0,0 +1,125 @@ +# -*- python -*- +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Fake ovirtsdk4 module used as a test harness. +# See v2v/test-v2v-o-rhv-upload.sh + +from enum import Enum +from ovirtsdk4 import imageio_port + +class Cluster(object): + def __init__(self, name): + pass + +class Configuration(object): + def __init__(self, type=None, data=None): + pass +class ConfigurationType(Enum): + OVA = 'ova' + OVF = 'ovf' + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class DiskFormat(Enum): + COW = "cow" + RAW = "raw" + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class DiskStatus(Enum): + ILLEGAL = "illegal" + LOCKED = "locked" + OK = "ok" + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class Disk(object): + def __init__( + self, + id = None, + name = None, + description = None, + format = None, + initial_size = None, + provisioned_size = None, + sparse = False, + storage_domains = None + ): + pass + + id = 123 + status = DiskStatus.OK + +class ImageTransferPhase(Enum): + CANCELLED = 'cancelled' + FINALIZING_FAILURE = 'finalizing_failure' + FINALIZING_SUCCESS = 'finalizing_success' + FINISHED_FAILURE = 'finished_failure' + FINISHED_SUCCESS = 'finished_success' + INITIALIZING = 'initializing' + PAUSED_SYSTEM = 'paused_system' + PAUSED_USER = 'paused_user' + RESUMING = 'resuming' + TRANSFERRING = 'transferring' + UNKNOWN = 'unknown' + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class ImageTransfer(object): + def __init__( + self, + disk = None, + host = None, + inactivity_timeout = None, + ): + pass + + id = 456 + phase = ImageTransferPhase.TRANSFERRING + transfer_url = "http://localhost:" + str(imageio_port) + "/" + +class Initialization(object): + def __init__(self, configuration): + pass + +class StorageDomain(object): + def __init__(self, name = None): + pass + +class Vm(object): + def __init__( + self, + cluster = None, + initialization = None + ): + pass diff --git a/v2v/test-v2v-o-rhv-upload.sh b/v2v/test-v2v-o-rhv-upload.sh new file mode 100755 index 000000000..8bda7cc0b --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload.sh @@ -0,0 +1,51 @@ +#!/bin/bash - +# libguestfs virt-v2v test script +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Test -o rhv-upload. +# +# These uses a test harness (see +# v2v/test-v2v-o-rhv-upload-module/ovirtsdk4) to fake responses from +# oVirt. + +set -e +set -x + +$TEST_FUNCTIONS +skip_if_skipped +skip_if_backend uml +skip_unless_phony_guest windows.img + +libvirt_uri="test://$abs_top_builddir/test-data/phony-guests/guests.xml" +f=$top_builddir/test-data/phony-guests/windows.img + +export VIRT_TOOLS_DATA_DIR="$top_srcdir/test-data/fake-virt-tools" +export VIRTIO_WIN="$top_srcdir/test-data/fake-virtio-win" +export PYTHONPATH=$srcdir/test-v2v-o-rhv-upload-module:$PYTHONPATH + +# Run virt-v2v -o rhv-upload. +# +# The fake ovirtsdk4 module doesn't care about most of the options +# like -oc, -oo rhv-cafile, -op etc. Any values may be used. +$VG virt-v2v --debug-gc -v -x \ + -i libvirt -ic "$libvirt_uri" windows \ + -o rhv-upload \ + -oc https://example.com/ovirt-engine/api \ + -oo rhv-cafile=/dev/null \ + -oo rhv-direct \ + -op /dev/null \ + -os . -- 2.19.0.rc0
On Thursday, 20 September 2018 10:50:14 CEST Richard W.M. Jones wrote:
+# Choose a random port number in range [50000,59999] +imageio_port = random.randint(50000,60000) + [...] + +def server(): + server_address = ("", imageio_port) + httpd = HTTPServer(server_address, RequestHandler) + httpd.serve_forever()
The random port can be chosen directly by the OS, using the special port 0:
from http.server import HTTPServer, SimpleHTTPRequestHandler server_address = ("", 0) httpd = HTTPServer(server_address, SimpleHTTPRequestHandler) print(httpd.server_address) ('0.0.0.0', 44725)
-- Pino Toscano
On Wed, Sep 26, 2018 at 07:40:08PM +0200, Pino Toscano wrote:
On Thursday, 20 September 2018 10:50:14 CEST Richard W.M. Jones wrote:
+# Choose a random port number in range [50000,59999] +imageio_port = random.randint(50000,60000) + [...] + +def server(): + server_address = ("", imageio_port) + httpd = HTTPServer(server_address, RequestHandler) + httpd.serve_forever()
The random port can be chosen directly by the OS, using the special port 0:
from http.server import HTTPServer, SimpleHTTPRequestHandler server_address = ("", 0) httpd = HTTPServer(server_address, SimpleHTTPRequestHandler) print(httpd.server_address) ('0.0.0.0', 44725)
Of course :-) Thanks, will fix. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Thu, Sep 20, 2018 at 11:51 AM Richard W.M. Jones <rjones@redhat.com> wrote:
Previously this output method was almost completely untested.
This commit adds a fake ovirtsdk4 module so we can test the -o rhv-upload method fairly completely without needing an actual oVirt instance around. --- v2v/Makefile.am | 4 + .../ovirtsdk4/__init__.py | 146 ++++++++++++++++++ .../ovirtsdk4/types.py | 125 +++++++++++++++ v2v/test-v2v-o-rhv-upload.sh | 51 ++++++ 4 files changed, 326 insertions(+)
diff --git a/v2v/Makefile.am b/v2v/Makefile.am index 14183572b..aab356637 100644 --- a/v2v/Makefile.am +++ b/v2v/Makefile.am @@ -382,6 +382,7 @@ TESTS += \ test-v2v-o-openstack.sh \ test-v2v-o-qemu.sh \ test-v2v-o-rhv.sh \ + test-v2v-o-rhv-upload.sh \ test-v2v-o-vdsm-options.sh \ test-v2v-oa-option.sh \ test-v2v-of-option.sh \ @@ -539,6 +540,9 @@ EXTRA_DIST += \ test-v2v-o-qemu.sh \ test-v2v-o-rhv.ovf.expected \ test-v2v-o-rhv.sh \ + test-v2v-o-rhv-upload.sh \ + test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py \ + test-v2v-o-rhv-upload-module/ovirtsdk4/types.py \ test-v2v-o-rhv-upload-oo-query.sh \ test-v2v-o-vdsm-oo-query.sh \ test-v2v-o-vdsm-options.ovf.expected \ diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py new file mode 100644 index 000000000..2ddb950ea --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py @@ -0,0 +1,146 @@ +# -*- python -*- +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Fake ovirtsdk4 module used as a test harness. +# See v2v/test-v2v-o-rhv-upload.sh + +class Error(Exception): + pass +class NotFoundError(Error): + pass + +class Connection(object): + def __init__( + self, + url = None, + username = None, + password = None, + ca_file = None, + log = None, + insecure = False, + ): + pass + + def system_service(self): + return SystemService() + +class SystemService(object): + def data_centers_service(self): + return DataCentersService() + + def disks_service(self): + return DisksService() + + def image_transfers_service(self): + return ImageTransfersService() + + def storage_domains_service(self): + return StorageDomainsService() + + def vms_service(self): + return VmsService() + +class DataCentersService(object): + def list(self, search=None, case_sensitive=False): + return [] + +class DiskService(object): + def __init__(self, disk_id): + self._disk_id = disk_id + + def get(self): + return types.Disk() + + def remove(self): + pass + +class DisksService(object): + def add(self, disk=None): + return disk + + def disk_service(self, disk_id): + return DiskService(disk_id) + +class ImageTransferService(object): + def __init__(self): + self._finalized = False + + def get(self): + if self._finalized: + raise NotFoundError + else: + return types.ImageTransfer() + + def finalize(self): + self._finalized = True + +class ImageTransfersService(object): + def add(self, transfer): + return transfer + + def image_transfer_service(self, id): + return ImageTransferService() + +class StorageDomain(object): + id = "ba87af68-b630-4211-a73a-694c1a689405" + +class StorageDomainsService(object): + def list(self, search=None): + return [ StorageDomain() ] + +class VmsService(object): + def add(self, vm): + return vm + + def list(self, search=None): + return [] + +# Create a background thread running a web server which is +# simulating the imageio server.
This functionality should be separated from the fake SDK module, since it is not part of the SDK, and may be replaced by real imageio server later.
+ +from http.server import HTTPServer, BaseHTTPRequestHandler +import random +import threading + +# Choose a random port number in range [50000,59999] +imageio_port = random.randint(50000,60000) + +class RequestHandler(BaseHTTPRequestHandler):
This request handler is using HTTP/1.0, and will close the connection after every request. This is not a good implementation of the imageio server, and also hides bugs in this code. Should be fixed by adding: protocol_version = "HTTP/1.1"
+ def do_OPTIONS(self):
+ self.send_response(200)
+ self.send_header("Content-type", "application/json; charset=UTF-8") + self.end_headers() + # Advertize only zero support. + self.wfile.write(b'''{ "features": [ "zero" ] }''') + + # eg. zero request. Just ignore it. + def do_PATCH(self):
This must read content-length bytes from self.rfile.
+ self.send_response(200) + self.end_headers() + + # Flush request. Ignore it. + def do_PUT(self):
This must read content-length bytes from self.rfile. Both bugs are hidden by the fact that this server close the connection at the end of the request.
+ self.send_response(200) + self.end_headers() + +def server(): + server_address = ("", imageio_port) + httpd = HTTPServer(server_address, RequestHandler) + httpd.serve_forever()
Using HTTP instead of HTTPS is not a good idea. We are not testing the same code on the client side. It is easy to run HTTPS server using pre-created certificate, if we disable certificate verification on the client side (e.g. --insecure). Nir
+ +thread = threading.Thread(target = server, args = [], daemon = True) +thread.start() diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py new file mode 100644 index 000000000..9b3f557ee --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py @@ -0,0 +1,125 @@ +# -*- python -*- +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Fake ovirtsdk4 module used as a test harness. +# See v2v/test-v2v-o-rhv-upload.sh + +from enum import Enum +from ovirtsdk4 import imageio_port + +class Cluster(object): + def __init__(self, name): + pass + +class Configuration(object): + def __init__(self, type=None, data=None): + pass +class ConfigurationType(Enum): + OVA = 'ova' + OVF = 'ovf' + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class DiskFormat(Enum): + COW = "cow" + RAW = "raw" + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class DiskStatus(Enum): + ILLEGAL = "illegal" + LOCKED = "locked" + OK = "ok" + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class Disk(object): + def __init__( + self, + id = None, + name = None, + description = None, + format = None, + initial_size = None, + provisioned_size = None, + sparse = False, + storage_domains = None + ): + pass + + id = 123 + status = DiskStatus.OK + +class ImageTransferPhase(Enum): + CANCELLED = 'cancelled' + FINALIZING_FAILURE = 'finalizing_failure' + FINALIZING_SUCCESS = 'finalizing_success' + FINISHED_FAILURE = 'finished_failure' + FINISHED_SUCCESS = 'finished_success' + INITIALIZING = 'initializing' + PAUSED_SYSTEM = 'paused_system' + PAUSED_USER = 'paused_user' + RESUMING = 'resuming' + TRANSFERRING = 'transferring' + UNKNOWN = 'unknown' + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class ImageTransfer(object): + def __init__( + self, + disk = None, + host = None, + inactivity_timeout = None, + ): + pass + + id = 456 + phase = ImageTransferPhase.TRANSFERRING + transfer_url = "http://localhost:" + str(imageio_port) + "/" + +class Initialization(object): + def __init__(self, configuration): + pass + +class StorageDomain(object): + def __init__(self, name = None): + pass + +class Vm(object): + def __init__( + self, + cluster = None, + initialization = None + ): + pass diff --git a/v2v/test-v2v-o-rhv-upload.sh b/v2v/test-v2v-o-rhv-upload.sh new file mode 100755 index 000000000..8bda7cc0b --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload.sh @@ -0,0 +1,51 @@ +#!/bin/bash - +# libguestfs virt-v2v test script +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Test -o rhv-upload. +# +# These uses a test harness (see +# v2v/test-v2v-o-rhv-upload-module/ovirtsdk4) to fake responses from +# oVirt. + +set -e +set -x + +$TEST_FUNCTIONS +skip_if_skipped +skip_if_backend uml +skip_unless_phony_guest windows.img + +libvirt_uri="test://$abs_top_builddir/test-data/phony-guests/guests.xml" +f=$top_builddir/test-data/phony-guests/windows.img + +export VIRT_TOOLS_DATA_DIR="$top_srcdir/test-data/fake-virt-tools" +export VIRTIO_WIN="$top_srcdir/test-data/fake-virtio-win" +export PYTHONPATH=$srcdir/test-v2v-o-rhv-upload-module:$PYTHONPATH + +# Run virt-v2v -o rhv-upload. +# +# The fake ovirtsdk4 module doesn't care about most of the options +# like -oc, -oo rhv-cafile, -op etc. Any values may be used. +$VG virt-v2v --debug-gc -v -x \ + -i libvirt -ic "$libvirt_uri" windows \ + -o rhv-upload \ + -oc https://example.com/ovirt-engine/api \ + -oo rhv-cafile=/dev/null \ + -oo rhv-direct \ + -op /dev/null \ + -os . -- 2.19.0.rc0
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
On Tue, Oct 09, 2018 at 02:28:10PM +0300, Nir Soffer wrote:
+# Create a background thread running a web server which is +# simulating the imageio server.
This functionality should be separated from the fake SDK module, since it is not part of the SDK, and may be replaced by real imageio server later.
Well possibly, but it was very convenient to put it in the class here, and this test is meant for running completely standalone without any other service available.
+class RequestHandler(BaseHTTPRequestHandler):
This request handler is using HTTP/1.0, and will close the connection after every request. This is not a good implementation of the imageio server, and also hides bugs in this code.
Should be fixed by adding:
protocol_version = "HTTP/1.1"
I tried the attached patch, but for some reason it just hangs.
+ def do_OPTIONS(self):
+ self.send_response(200)
+ self.send_header("Content-type", "application/json; charset=UTF-8") + self.end_headers() + # Advertize only zero support. + self.wfile.write(b'''{ "features": [ "zero" ] }''') + + # eg. zero request. Just ignore it. + def do_PATCH(self):
This must read content-length bytes from self.rfile.
+ self.send_response(200) + self.end_headers() + + # Flush request. Ignore it. + def do_PUT(self):
This must read content-length bytes from self.rfile.
Both bugs are hidden by the fact that this server close the connection at the end of the request.
Also this is covered by the attached patch.
+ self.send_response(200) + self.end_headers() + +def server(): + server_address = ("", imageio_port) + httpd = HTTPServer(server_address, RequestHandler) + httpd.serve_forever()
Using HTTP instead of HTTPS is not a good idea. We are not testing the same code on the client side.
It is easy to run HTTPS server using pre-created certificate, if we disable certificate verification on the client side (e.g. --insecure).
This one doesn't seem to be as simple to fix. However point taken, so I added a note in the code. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Mon, Oct 15, 2018 at 6:21 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Oct 09, 2018 at 02:28:10PM +0300, Nir Soffer wrote:
+# Create a background thread running a web server which is +# simulating the imageio server.
This functionality should be separated from the fake SDK module, since it is not part of the SDK, and may be replaced by real imageio server later.
Well possibly, but it was very convenient to put it in the class here, and this test is meant for running completely standalone without any other service available.
+class RequestHandler(BaseHTTPRequestHandler):
This request handler is using HTTP/1.0, and will close the connection after every request. This is not a good implementation of the imageio server, and also hides bugs in this code.
Should be fixed by adding:
protocol_version = "HTTP/1.1"
I tried the attached patch, but for some reason it just hangs.
The issue is probably missing "content-length: 0" header in the response. If we don't close the connection and don't send content-length the client cannot do much but wait :-)
+ def do_OPTIONS(self):
+ self.send_response(200)
+ self.send_header("Content-type", "application/json; charset=UTF-8") + self.end_headers() + # Advertize only zero support. + self.wfile.write(b'''{ "features": [ "zero" ] }''') + + # eg. zero request. Just ignore it. + def do_PATCH(self):
This must read content-length bytes from self.rfile.
+ self.send_response(200) + self.end_headers() + + # Flush request. Ignore it. + def do_PUT(self):
This must read content-length bytes from self.rfile.
Both bugs are hidden by the fact that this server close the connection at the end of the request.
Also this is covered by the attached patch.
+ self.send_response(200) + self.end_headers() + +def server(): + server_address = ("", imageio_port) + httpd = HTTPServer(server_address, RequestHandler) + httpd.serve_forever()
Using HTTP instead of HTTPS is not a good idea. We are not testing the same code on the client side.
It is easy to run HTTPS server using pre-created certificate, if we disable certificate verification on the client side (e.g. --insecure).
This one doesn't seem to be as simple to fix. However point taken, so I added a note in the code.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Mon, Oct 15, 2018 at 06:43:10PM +0300, Nir Soffer wrote:
On Mon, Oct 15, 2018 at 6:21 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Oct 09, 2018 at 02:28:10PM +0300, Nir Soffer wrote:
+# Create a background thread running a web server which is +# simulating the imageio server.
This functionality should be separated from the fake SDK module, since it is not part of the SDK, and may be replaced by real imageio server later.
Well possibly, but it was very convenient to put it in the class here, and this test is meant for running completely standalone without any other service available.
+class RequestHandler(BaseHTTPRequestHandler):
This request handler is using HTTP/1.0, and will close the connection after every request. This is not a good implementation of the imageio server, and also hides bugs in this code.
Should be fixed by adding:
protocol_version = "HTTP/1.1"
I tried the attached patch, but for some reason it just hangs.
The issue is probably missing "content-length: 0" header in the response. If we don't close the connection and don't send content-length the client cannot do much but wait :-)
Ah yes, it was missing from the OPTIONS method. The attached patch does work. What do you think? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Mon, Oct 15, 2018 at 7:13 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Mon, Oct 15, 2018 at 06:43:10PM +0300, Nir Soffer wrote:
On Mon, Oct 15, 2018 at 6:21 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Oct 09, 2018 at 02:28:10PM +0300, Nir Soffer wrote:
+# Create a background thread running a web server which is +# simulating the imageio server.
This functionality should be separated from the fake SDK module, since it is not part of the SDK, and may be replaced by real imageio server later.
Well possibly, but it was very convenient to put it in the class here, and this test is meant for running completely standalone without any other service available.
+class RequestHandler(BaseHTTPRequestHandler):
This request handler is using HTTP/1.0, and will close the connection after every request. This is not a good implementation of the imageio server, and also hides bugs in this code.
Should be fixed by adding:
protocol_version = "HTTP/1.1"
I tried the attached patch, but for some reason it just hangs.
The issue is probably missing "content-length: 0" header in the response. If we don't close the connection and don't send content-length the client cannot do much but wait :-)
Ah yes, it was missing from the OPTIONS method. The attached patch does work. What do you think?
Looks good Nir
On Mon, Oct 15, 2018 at 05:13:31PM +0100, Richard W.M. Jones wrote:
On Mon, Oct 15, 2018 at 06:43:10PM +0300, Nir Soffer wrote:
On Mon, Oct 15, 2018 at 6:21 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Oct 09, 2018 at 02:28:10PM +0300, Nir Soffer wrote:
+# Create a background thread running a web server which is +# simulating the imageio server.
This functionality should be separated from the fake SDK module, since it is not part of the SDK, and may be replaced by real imageio server later.
Well possibly, but it was very convenient to put it in the class here, and this test is meant for running completely standalone without any other service available.
+class RequestHandler(BaseHTTPRequestHandler):
This request handler is using HTTP/1.0, and will close the connection after every request. This is not a good implementation of the imageio server, and also hides bugs in this code.
Should be fixed by adding:
protocol_version = "HTTP/1.1"
I tried the attached patch, but for some reason it just hangs.
The issue is probably missing "content-length: 0" header in the response. If we don't close the connection and don't send content-length the client cannot do much but wait :-)
Ah yes, it was missing from the OPTIONS method. The attached patch does work. What do you think?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
From 6eb473a7e67de82bf08b97f2610ee3364071c741 Mon Sep 17 00:00:00 2001 From: "Richard W.M. Jones" <rjones@redhat.com> Date: Mon, 15 Oct 2018 16:07:52 +0100 Subject: [PATCH] v2v: -o rhv-upload: Test using HTTP/1.1 protocol.
Fixes commit b54b58c44e3f1f54a05117c758eaa21d9f1085f0.
Thanks: Nir Soffer. --- .../ovirtsdk4/__init__.py | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-)
diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py index b5f739471..84b9d56aa 100644 --- a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py @@ -116,24 +116,44 @@ from http.server import HTTPServer, BaseHTTPRequestHandler import threading
class RequestHandler(BaseHTTPRequestHandler): + protocol_version = 'HTTP/1.1' + def do_OPTIONS(self): - self.send_response(200) - self.send_header("Content-type", "application/json; charset=UTF-8") - self.end_headers() + self.discard_request() + # Advertize only zero support. - self.wfile.write(b'''{ "features": [ "zero" ] }''') + content = b'''{ "features": [ "zero" ] }''' + length = len(content) + + self.send_response(200) + self.send_header("Content-type", "application/json; charset=UTF-8") + self.send_header("Content-Length", length) + self.end_headers() + self.wfile.write(content)
# eg. zero request. Just ignore it. def do_PATCH(self): + self.discard_request() self.send_response(200) + self.send_header("Content-Length", "0") self.end_headers()
# Flush request. Ignore it. def do_PUT(self): + self.discard_request() self.send_response(200) + self.send_header("Content-Length", "0") self.end_headers()
+ def discard_request(self): + length = self.headers['Content-Length'] + if length: + length = int(length) + content = self.rfile.read(length)
This commit went upstream: https://github.com/libguestfs/libguestfs/commit/bfd9bee9fc7a91fdca0ca051deca... However it fails on RHEL 7.6 with: Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 295, in _handle_request_nobl ock self.process_request(request, client_address) File "/usr/lib64/python2.7/SocketServer.py", line 321, in process_request self.finish_request(request, client_address) File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__ self.handle() File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle self.handle_one_request() File "/usr/lib64/python2.7/BaseHTTPServer.py", line 328, in handle_one_request method() File "/home/rjones/d/libguestfs-rhel-7.6-lp/v2v/test-v2v-o-rhv-upload-module/o virtsdk4/__init__.py", line 123, in do_OPTIONS self.discard_request() File "/home/rjones/d/libguestfs-rhel-7.6-lp/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py", line 150, in discard_request length = self.headers['Content-Length'] File "/usr/lib64/python2.7/rfc822.py", line 388, in __getitem__ return self.dict[name.lower()] KeyError: 'content-length' I guess that Python won't let us do: length = self.headers['Content-Length'] if length: How about the attached fix? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Tue, Nov 20, 2018 at 05:04:15PM +0000, Richard W.M. Jones wrote:
From 9aca67b9875ab31bea7f3aeec411dad3ed990f17 Mon Sep 17 00:00:00 2001 From: "Richard W.M. Jones" <rjones@redhat.com> Date: Tue, 20 Nov 2018 17:00:33 +0000 Subject: [PATCH] v2v: -o rhv-upload: Fix test so it doesn't fail if Content-Length header missing.
If the Content-Length header was missing from the headers returned by the server then the test would fail with:
KeyError: 'content-length'
This happens on RHEL 7.6 in particular. We can fix this by using an alternate method to fetch the header, which will return None instead of throwing an exception if the header doesn't exist. See: https://stackoverflow.com/a/19255675 --- v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py index 84b9d56aa..8d1058d67 100644 --- a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py @@ -146,7 +146,7 @@ class RequestHandler(BaseHTTPRequestHandler): self.end_headers()
def discard_request(self): - length = self.headers['Content-Length'] + length = self.headers.get('Content-Length') if length: length = int(length) content = self.rfile.read(length) -- 2.19.0.rc0
FYI I have tested this patch on Fedora 29 & RHEL 7 and the test now succeeds on both. Previously the test failed on RHEL 7. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Tue, Nov 20, 2018 at 7:49 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Nov 20, 2018 at 05:04:15PM +0000, Richard W.M. Jones wrote:
From 9aca67b9875ab31bea7f3aeec411dad3ed990f17 Mon Sep 17 00:00:00 2001 From: "Richard W.M. Jones" <rjones@redhat.com> Date: Tue, 20 Nov 2018 17:00:33 +0000 Subject: [PATCH] v2v: -o rhv-upload: Fix test so it doesn't fail if Content-Length header missing.
If the Content-Length header was missing from the headers returned by the server then the test would fail with:
KeyError: 'content-length'
This happens on RHEL 7.6 in particular. We can fix this by using an alternate method to fetch the header, which will return None instead of throwing an exception if the header doesn't exist. See: https://stackoverflow.com/a/19255675 --- v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py index 84b9d56aa..8d1058d67 100644 --- a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py @@ -146,7 +146,7 @@ class RequestHandler(BaseHTTPRequestHandler): self.end_headers()
def discard_request(self): - length = self.headers['Content-Length'] + length = self.headers.get('Content-Length') if length: length = int(length) content = self.rfile.read(length) -- 2.19.0.rc0
Looks fine. Nir
FYI I have tested this patch on Fedora 29 & RHEL 7 and the test now succeeds on both. Previously the test failed on RHEL 7.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Thursday, 20 September 2018 10:50:11 CEST Richard W.M. Jones wrote:
v1 was here:
https://www.redhat.com/archives/libguestfs/2018-September/msg00121.html
v2:
- Rewrote patch 2 from scratch so it incorporates Nir's suggestions.
- Add fake module to EXTRA_DIST.
- Retested.
Except from the note left for patch #3, this series LGTM. -- Pino Toscano
participants (3)
-
Nir Soffer -
Pino Toscano -
Richard W.M. Jones