December 2021 - Libguestfs - libguestfs List Archives

RFC for NBD protocol extension: extended headers
by Eric Blake 04 Oct '22

04 Oct '22

In response to this mail, I will be cross-posting a series of patches to multiple projects as a proof-of-concept implementation and request for comments on a new NBD protocol extension, called NBD_OPT_EXTENDED_HEADERS. With this in place, it will be possible for clients to request 64-bit zero, trim, cache, and block status operations when supported by the server. Not yet complete: an implementation of this in nbdkit. I also plan to tweak libnbd's 'nbdinfo --map' and 'nbdcopy' to take advantage of the larger block status results. Also, with 64-bit commands, we may want to also make it easier to let servers advertise an actual maximum size they are willing to accept for the commands in question (for example, a server may be happy with a full 64-bit block status, but still want to limit non-fast zero and cache to a smaller limit to avoid denial of service). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

4 46

[PATCH 0/5] Fix rhv-upload output
by Nir Soffer 10 Feb '22

10 Feb '22

Fix problems in new rhv-upload implementation: - The plugin does not flush to all connections in flush() - The plugin does not close all connections in cleanup() - Idle connections are closed in imageio server, and we don't have a safe way to recover. - virt-v2v try to get disk allocation using imageio output, but imageio output does not support extents. Even if imageio output will support extents, the call is done after the transfer was finalized so it does not have access to storage. Problems not fixed yet: - Image transfer is finalized *before* closing the connection to imageio - this will always time out with RHV < 4.4.9, and succeeds by mistake with RHV 4.4.9 due to a regression that will be fixed in 4.4.10. This will be a non-issue in next RHV version[1]. To support older RHV versions, virt-v2v must finalize the image transfer *after* closing the output. Tested on RHEL 8.6 with upstream nbdkit and libnbd. [1] https://github.com/oVirt/ovirt-imageio/pull/15 Fixes https://bugzilla.redhat.com/2032324 Nir Soffer (5): output/rhv-upload-plugin: Fix flush and close v2v/lib/util.ml: Get disk allocation from input output/rhv-upload-plugin: Extract send_flush() helper output/rhv-upload-plugin: Track http last request time output/rhv-upload-plugin: Keep connections alive lib/utils.ml | 2 +- output/rhv-upload-plugin.py | 151 ++++++++++++++++++++++++++---------- 2 files changed, 113 insertions(+), 40 deletions(-) -- 2.33.1

3 30

Re: [Libguestfs] libnbd | Failed pipeline for master | 70da51e5
by Richard W.M. Jones 04 Jan '22

04 Jan '22

Hi Martin, On Thu, Sep 02, 2021 at 04:05:56PM +0000, GitLab wrote: > GitLab > ✖ Pipeline #364204388 has failed! > > Project nbdkit / libnbd > Branch ● master > Commit ● 70da51e5 > interop: Link interop-nbd-server-tls with -lgnu... > Commit Author ● Richard W.M. Jones > > Pipeline #364204388 triggered by ● Richard W.M. Jones > had 1 failed job. > Failed jobs > ✖ builds x64-opensuse-tumbleweed > GitLab This is failing on a new test I added, but it's failing because of how a particular package is built in OpenSUSE. The new test is: https://gitlab.com/nbdkit/libnbd/-/commit/c833fa1226092fd51b1211fa195a2a3b7… which tries to test libnbd client with TLS enabled against nbd-server. nbd-server in OpenSUSE gives this error: Error: inetd mode requires syslog Exiting. which means it was compiled without --enable-syslog. I notice that the related test is skipped: SKIP: interop-nbd-server ======================== Test skipped based on ci/skipped_tests file SKIP interop-nbd-server (exit status: 77) The format of ci/skipped_tests is pretty odd. Is this patch OK? diff --git a/ci/skipped_tests b/ci/skipped_tests index e2de9330..c494b9eb 100644 --- a/ci/skipped_tests +++ b/ci/skipped_tests @@ -1,9 +1,9 @@ # Old nbd-server and built without syslog support, tests deadlock, old qemu-img version -^Ubuntu-18\.04$;interop/interop-nbd-server interop/list-exports-nbd-server interop/structured-read.sh -^openSUSE Leap-15;interop/interop-nbd-server interop/list-exports-nbd-server +^Ubuntu-18\.04$;interop/interop-nbd-server interop/interop-nbd-server-tls interop/list-exports-nbd-server interop/structured-read.sh +^openSUSE Leap-15;interop/interop-nbd-server interop/interop-nbd-server-tls interop/list-exports-nbd-server # Similar for Tumbleweed, except tests do not deadlock, only limit to version 2021* for now -^openSUSE Tumbleweed-2021;interop/interop-nbd-server interop/list-exports-nbd-server +^openSUSE Tumbleweed-2021;interop/interop-nbd-server interop/interop-nbd-server-tls interop/list-exports-nbd-server # Debian 10 has weird golang issues (old golang anyway) and old qemu-img ^Debian GNU/Linux-10;golang/run-tests.sh interop/structured-read.sh Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

2 8

[PATCH 0/3] resolve conflict between manual and libvirt-assigned PCI addresses
by Laszlo Ersek 04 Jan '22

04 Jan '22

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2034160 The first patch extends our current <qemu:commandline> hack, moving the virtio-net-pci device to slot 0x1E, where it is very unlikely to conflict with any libvirt-assigned PCI address. The second patch is only refactoring. The third patch resolves the conflict on libvirt >= 3.8.0 in a better way (suggested by Rich): such libvirt versions permit SLIRP network address and prefix specifications right in the <interface> element. Therefore we can have libvirt manage the virtio-net device for us, similarly to virtio-rng, virtio-scsi, and virtio-serial. The (updated) <qemu:commandline> hack is preserved for libvirt < 3.8.0. Gruesomely meticulously tested. (See the Notes sections on the patches.) Sanity-tested both back-ends *without* the "--network" switch as well. Thanks, Laszlo Laszlo Ersek (3): launch-libvirt: place our virtio-net-pci device in slot 0x1e lib: extract NETWORK_ADDRESS and NETWORK_PREFIX as macros launch-libvirt: add virtio-net via the standard <interface> element lib/guestfs-internal.h | 18 ++++++++++++++++++ lib/launch-direct.c | 2 +- lib/launch-libvirt.c | 34 ++++++++++++++++++++++++++++++---- 3 files changed, 49 insertions(+), 5 deletions(-) base-commit: 4af6d68e2d8b856d91fa5527216ea3db04556086 -- 2.19.1.3.g30247aa5d201

3 8

[PATCH libnbd CI] ci: Drop Fedora 33, add Fedora 35
by Richard W.M. Jones 04 Jan '22

04 Jan '22

Hi Martin, The pipeline failed, and I'm assuming this is because Fedora 33 is out of support. The log is not very helpful: https://gitlab.com/nbdkit/libnbd/-/jobs/1874368914 Also we don't seem to have lcitool in Fedora anywhere obvious, so instead I just updated the generated files directly. Also, wouldn't it be nice if we could specify "fedora:n" and "fedora:n-1" instead of having to update this every six months ... Rich.

4 8

[PATCH nbdkit 1/2] tests/test-python-plugin.py: Allow test to use large disks
by Richard W.M. Jones 22 Dec '21

22 Dec '21

The Python test harness uses a plugin which always created a fully allocated disk backed by an in-memory bytearray. This prevented us from testing very large disks (since we could run out of memory easily). Add a feature allowing large all-zero disks to be tested. The disk is not allocated and non-zero writes will fail, but everything else works. --- tests/test-python-plugin.py | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/tests/test-python-plugin.py b/tests/test-python-plugin.py index 70d545db..a30b7f64 100644 --- a/tests/test-python-plugin.py +++ b/tests/test-python-plugin.py @@ -51,13 +51,17 @@ def config_complete(): def open(readonly): + if cfg.get('no_disk', False): + disk = None + else: + disk = bytearray(cfg.get('size', 0)) return { - 'disk': bytearray(cfg.get('size', 0)) + 'disk': disk } def get_size(h): - return len(h['disk']) + return cfg.get('size', 0) def is_rotational(h): @@ -123,6 +127,7 @@ def pwrite(h, buf, offset, flags): actual_fua = bool(flags & nbdkit.FLAG_FUA) assert expect_fua == actual_fua end = offset + len(buf) + assert h['disk'] is not None h['disk'][offset:end] = buf @@ -134,7 +139,8 @@ def trim(h, count, offset, flags): expect_fua = cfg.get('trim_expect_fua', False) actual_fua = bool(flags & nbdkit.FLAG_FUA) assert expect_fua == actual_fua - h['disk'][offset:offset+count] = bytearray(count) + if h['disk'] is not None: + h['disk'][offset:offset+count] = bytearray(count) def zero(h, count, offset, flags): @@ -147,7 +153,8 @@ def zero(h, count, offset, flags): expect_fast_zero = cfg.get('zero_expect_fast_zero', False) actual_fast_zero = bool(flags & nbdkit.FLAG_FAST_ZERO) assert expect_fast_zero == actual_fast_zero - h['disk'][offset:offset+count] = bytearray(count) + if h['disk'] is not None: + h['disk'][offset:offset+count] = bytearray(count) def cache(h, count, offset, flags): -- 2.32.0

2 9

[v2v PATCH] output_rhv: restrict block status collection to the old RHV output
by Laszlo Ersek 22 Dec '21

22 Dec '21

Nir reports that, despite the comment we removed in commit a2a4f7a09996, we generally cannot access the output NBD servers in the finalization stage. In particular, in the "rhv_upload_finalize" function [output/output_rhv_upload.ml], the output disks are disconnected before "create_ovf" is called. Consequently, the "get_disk_allocated" call in the "create_ovf" -> "add_disks" -> "get_disk_allocated" chain fails. Rich suggests that we explicitly restrict the "get_disk_allocated" call with a new optional boolean parameter to the one output plugin that really needs it, namely the old RHV one. Accordingly, revert the VDSM test case to its state at (a2a4f7a09996^). Cc: Nir Soffer <nsoffer(a)redhat.com> Fixes: a2a4f7a09996a5e66d027d0d9692e083eb0a8128 Reported-by: Nir Soffer <nsoffer(a)redhat.com> Suggested-by: Richard W.M. Jones <rjones(a)redhat.com> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2034240 Signed-off-by: Laszlo Ersek <lersek(a)redhat.com> --- lib/create_ovf.mli | 3 +- lib/create_ovf.ml | 35 +++++++++++++--------- output/output_rhv.ml | 4 +-- tests/test-v2v-o-vdsm-options.ovf.expected | 4 +-- 4 files changed, 27 insertions(+), 19 deletions(-) diff --git a/lib/create_ovf.mli b/lib/create_ovf.mli index 0d1cc5a9311a..d6d4e62eeb86 100644 --- a/lib/create_ovf.mli +++ b/lib/create_ovf.mli @@ -46,7 +46,8 @@ val ovf_flavour_to_string : ovf_flavour -> string val create_ovf : Types.source -> Types.inspect -> Types.target_meta -> int64 list -> Types.output_allocation -> string -> string -> string list -> - string list -> string -> string -> ovf_flavour -> DOM.doc + string list -> ?need_actual_sizes:bool -> string -> string -> + ovf_flavour -> DOM.doc (** Create the OVF file. Actually a {!DOM} document is created, not a file. It can be written diff --git a/lib/create_ovf.ml b/lib/create_ovf.ml index dbac3405989b..678d29942abe 100644 --- a/lib/create_ovf.ml +++ b/lib/create_ovf.ml @@ -531,7 +531,8 @@ let rec create_ovf source inspect { output_name; guestcaps; target_firmware; target_nics } sizes output_alloc output_format - sd_uuid image_uuids vol_uuids dir vm_uuid ovf_flavour = + sd_uuid image_uuids vol_uuids ?(need_actual_sizes = false) dir + vm_uuid ovf_flavour = assert (List.length sizes = List.length vol_uuids); let memsize_mb = source.s_memory /^ 1024L /^ 1024L in @@ -745,7 +746,7 @@ let rec create_ovf source inspect (* Add disks to the OVF XML. *) add_disks sizes guestcaps output_alloc output_format - sd_uuid image_uuids vol_uuids dir ovf_flavour ovf; + sd_uuid image_uuids vol_uuids need_actual_sizes dir ovf_flavour ovf; (* Old virt-v2v ignored removable media. XXX *) @@ -791,7 +792,7 @@ and get_flavoured_section ovf ovirt_path rhv_path rhv_path_attr = function (* This modifies the OVF DOM, adding a section for each disk. *) and add_disks sizes guestcaps output_alloc output_format - sd_uuid image_uuids vol_uuids dir ovf_flavour ovf = + sd_uuid image_uuids vol_uuids need_actual_sizes dir ovf_flavour ovf = let references = let nodes = path_to_nodes ovf ["ovf:Envelope"; "References"] in match nodes with @@ -839,7 +840,12 @@ and add_disks sizes guestcaps output_alloc output_format b /^ 1073741824L in let size_gb = bytes_to_gb size in - let actual_size = get_disk_allocated ~dir ~disknr:i in + let actual_size = + if need_actual_sizes then + get_disk_allocated ~dir ~disknr:i + else + None + in let format_for_rhv = match output_format with @@ -891,16 +897,17 @@ and add_disks sizes guestcaps output_alloc output_format "ovf:disk-type", "System"; (* RHBZ#744538 *) "ovf:boot", if is_bootable_drive then "True" else "False"; ] in - (* Ovirt-engine considers the "ovf:actual_size" attribute mandatory. If - * we don't know the actual size, we must create the attribute with - * empty contents. - *) - List.push_back attrs - ("ovf:actual_size", - match actual_size with - | None -> "" - | Some actual_size -> Int64.to_string (bytes_to_gb actual_size) - ); + if (need_actual_sizes) then + (* Ovirt-engine considers the "ovf:actual_size" attribute mandatory. + * If we don't know the actual size, we must create the attribute + * with empty contents. + *) + List.push_back attrs + ("ovf:actual_size", + match actual_size with + | None -> "" + | Some actual_size -> Int64.to_string (bytes_to_gb actual_size) + ); e "Disk" !attrs [] in append_child disk disk_section; diff --git a/output/output_rhv.ml b/output/output_rhv.ml index b902a7ee4619..6a67b7aa152b 100644 --- a/output/output_rhv.ml +++ b/output/output_rhv.ml @@ -183,8 +183,8 @@ and rhv_finalize dir source inspect target_meta (* Create the metadata. *) let ovf = Create_ovf.create_ovf source inspect target_meta sizes - output_alloc output_format esd_uuid image_uuids vol_uuids dir vm_uuid - Create_ovf.RHVExportStorageDomain in + output_alloc output_format esd_uuid image_uuids vol_uuids + ~need_actual_sizes:true dir vm_uuid Create_ovf.RHVExportStorageDomain in (* Write it to the metadata file. *) let dir = esd_mp // esd_uuid // "master" // "vms" // vm_uuid in diff --git a/tests/test-v2v-o-vdsm-options.ovf.expected b/tests/test-v2v-o-vdsm-options.ovf.expected index bd5b5e7d38ec..23ca180f4c2f 100644 --- a/tests/test-v2v-o-vdsm-options.ovf.expected +++ b/tests/test-v2v-o-vdsm-options.ovf.expected @@ -2,7 +2,7 @@ <ovf:Envelope xmlns:rasd='http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSet…' xmlns:vssd='http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingD…' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ovf='http://schemas.dmtf.org/ovf/envelope/1/' xmlns:ovirt='http://www.ovirt.org/ovf' ovf:version='0.9'>  <References> - <File ovf:href='VOL' ovf:id='VOL' ovf:description='generated by virt-v2v' ovf:size='#SIZE#'/> + <File ovf:href='VOL' ovf:id='VOL' ovf:description='generated by virt-v2v'/> </References> <NetworkSection> <Info>List of networks</Info> @@ -10,7 +10,7 @@ </NetworkSection> <DiskSection> <Info>List of Virtual Disks</Info> - <Disk ovf:diskId='IMAGE' ovf:size='1' ovf:capacity='536870912' ovf:fileRef='VOL' ovf:parentRef='' ovf:vm_snapshot_id='#UUID#' ovf:volume-format='COW' ovf:volume-type='Sparse' ovf:format='http://en.wikipedia.org/wiki/Byte' ovf:disk-interface='VirtIO' ovf:disk-type='System' ovf:boot='True' ovf:actual_size='1'/> + <Disk ovf:diskId='IMAGE' ovf:size='1' ovf:capacity='536870912' ovf:fileRef='VOL' ovf:parentRef='' ovf:vm_snapshot_id='#UUID#' ovf:volume-format='COW' ovf:volume-type='Sparse' ovf:format='http://en.wikipedia.org/wiki/Byte' ovf:disk-interface='VirtIO' ovf:disk-type='System' ovf:boot='True'/> </DiskSection> <VirtualSystem ovf:id='VM'> <Name>windows</Name> -- 2.19.1.3.g30247aa5d201

3 6

[PATCH nbdkit] plugins/python: Fix extents() count format string
by Nir Soffer 19 Dec '21

19 Dec '21

The plugin used "i" (int32) instead of "I" (uint32) for the count, so when the client asks for 4294966784 bytes, the python plugin got -512. nbdkit: python.0: debug: python: extents count=4294966784 offset=0 req_one=0 ... nbdkit: python.0: debug: extents: count=-512 offset=0 flags=0 With this fix I can get extents from rhv-upload-plugin using nbdinfo. --- plugins/python/plugin.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/python/plugin.c b/plugins/python/plugin.c index f85512b4..366619f9 100644 --- a/plugins/python/plugin.c +++ b/plugins/python/plugin.c @@ -957,21 +957,21 @@ py_extents (void *handle, uint32_t count, uint64_t offset, ACQUIRE_PYTHON_GIL_FOR_CURRENT_SCOPE; struct handle *h = handle; PyObject *fn; PyObject *r; PyObject *iter, *t; size_t size; if (callback_defined ("extents", &fn)) { PyErr_Clear (); - r = PyObject_CallFunction (fn, "OiLI", h->py_h, count, offset, flags); + r = PyObject_CallFunction (fn, "OILI", h->py_h, count, offset, flags); Py_DECREF (fn); if (check_python_failure ("extents") == -1) return -1; iter = PyObject_GetIter (r); if (iter == NULL) { nbdkit_error ("extents method did not return " "something which is iterable"); Py_DECREF (r); return -1; -- 2.33.1

2 2

[PATCH] output/rhv-upload-plugin: Support extents
by Nir Soffer 19 Dec '21

19 Dec '21

Add extents support using imageio /extents API: https://ovirt.github.io/ovirt-imageio/images.html#extents Imageio does not support yet partial extents, so we cache entire image extents, and search for the requested range. The search can probably be faster using binary search, but it seems good enough as is. Calling pwrite() and zero() invalidates the cached extents. This can be useful for checking if an image is zero before copy, or calculating the allocated size after the copy. There is one caveat - imageio reports holes using qemu:allocation-depth, not using base:allocation. So qcow2 images report holes for unallocated areas, and raw images never reports holes. To get the actual allocation in RHV, oVirt API should be used. Testing is complicated: 1. Start ovirt-imageio server 2. Add ticket for test image using nbd socket 3. Start qemu-nbd serving the nbd socket 4. Run nbdkit with rhv-upload-plugin with and imageio https url I tested various images by getting extents with nbdinfo and qemu-img map and copying image from rhv-upload-plugin with qemu-img convert. To get extents using nbdinfo, we need a fix in nbdkit python plugin: https://listman.redhat.com/archives/libguestfs/2021-December/msg00196.html --- output/rhv-upload-plugin.py | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/output/rhv-upload-plugin.py b/output/rhv-upload-plugin.py index 1cb837dd..4f0c02a5 100644 --- a/output/rhv-upload-plugin.py +++ b/output/rhv-upload-plugin.py @@ -42,20 +42,25 @@ url = None cafile = None insecure = False is_ovirt_host = False # List of options read from imageio server. options = None # Pool of HTTP connections. pool = None +# Cached extents for entire image. The imageio server does not support yet +# getting partial extents, and getting all extents may be expensive, so we +# cache the entire image extents. +cached_extents = None + # Parse parameters. def config(key, value): global cafile, url, is_ovirt_host, insecure, size if key == "cafile": cafile = value elif key == "insecure": insecure = value.lower() in ['true', '1'] elif key == "is_ovirt_host": @@ -123,20 +128,24 @@ def can_fua(h): return options['can_flush'] def can_multi_conn(h): # We can always handle multiple connections, and the number of NBD # connections is independent of the number of HTTP clients in the # pool. return True +def can_extents(h): + return options["can_extents"] + + def get_size(h): return size # Any unexpected HTTP response status from the server will end up calling this # function which logs the full error, and raises a RuntimeError exception. def request_failed(r, msg): status = r.status reason = r.reason try: @@ -184,48 +193,54 @@ def pread(h, buf, offset, flags): while got < count: n = r.readinto(view[got:]) if n == 0: request_failed(r, "short read offset %d size %d got %d" % (offset, count, got)) got += n def pwrite(h, buf, offset, flags): + global cached_extents + count = len(buf) flush = "y" if (options['can_flush'] and (flags & nbdkit.FLAG_FUA)) else "n" with http_context(pool) as http: http.putrequest("PUT", url.path + "?flush=" + flush) # The oVirt server only uses the first part of the range, and the # content-length. http.putheader("Content-Range", "bytes %d-%d/*" % (offset, offset + count - 1)) http.putheader("Content-Length", str(count)) http.endheaders() try: http.send(buf) except BrokenPipeError: pass + cached_extents = None + r = http.getresponse() if r.status != 200: request_failed(r, "could not write sector offset %d size %d" % (offset, count)) r.read() def zero(h, count, offset, flags): + global cached_extents + # Unlike the trim and flush calls, there is no 'can_zero' method # so nbdkit could call this even if the server doesn't support # zeroing. If this is the case we must emulate. if not options['can_zero']: emulate_zero(h, count, offset, flags) return flush = bool(options['can_flush'] and (flags & nbdkit.FLAG_FUA)) # Construct the JSON request for zeroing. @@ -233,39 +248,45 @@ def zero(h, count, offset, flags): 'offset': offset, 'size': count, 'flush': flush}).encode() headers = {"Content-Type": "application/json", "Content-Length": str(len(buf))} with http_context(pool) as http: http.request("PATCH", url.path, body=buf, headers=headers) + cached_extents = None + r = http.getresponse() if r.status != 200: request_failed(r, "could not zero sector offset %d size %d" % (offset, count)) r.read() def emulate_zero(h, count, offset, flags): + global cached_extents + flush = "y" if (options['can_flush'] and (flags & nbdkit.FLAG_FUA)) else "n" with http_context(pool) as http: http.putrequest("PUT", url.path + "?flush=" + flush) http.putheader("Content-Range", "bytes %d-%d/*" % (offset, offset + count - 1)) http.putheader("Content-Length", str(count)) http.endheaders() + cached_extents = None + try: buf = bytearray(128 * 1024) while count > len(buf): http.send(buf) count -= len(buf) http.send(memoryview(buf)[:count]) except BrokenPipeError: pass r = http.getresponse() @@ -289,20 +310,76 @@ def flush(h, flags): for http in iter_http_pool(pool): http.request("PATCH", url.path, body=buf, headers=headers) r = http.getresponse() if r.status != 200: request_failed(r, "could not flush") r.read() +def extents(h, count, offset, flags): + global cached_extents + + if cached_extents is None: + cached_extents = get_all_extents() + + end = offset + count + + for ext in cached_extents: + start = ext["start"] + length = ext["length"] + + # Stop when extent exceeds the requested range. + if start >= end: + return + + # Skip over extents before the requested range. + if start + length <= offset: + continue + + extent_type = 0 + + if ext["zero"]: + extent_type |= nbdkit.EXTENT_ZERO + + # Old imageio server did not report holes. Note that imageio reports + # holes only for unallocated area in qcow2 image, so this flag may not + # be very useful. + if ext.get("hole"): + extent_type |= nbdkit.EXTENT_HOLE + + # The first extent may start before the requested range. + if start < offset: + length -= offset - start + start = offset + + # The last extent may end after the requested range. + if start + length > end: + length = end - start + + yield start, length, extent_type + + +def get_all_extents(): + with http_context(pool) as http: + http.request("GET", url.path + "/extents") + + r = http.getresponse() + if r.status != 200: + request_failed(r, "could not get extents") + + data = r.read() + + return json.loads(data) + + # Modify http.client.HTTPConnection to work over a Unix domain socket. # Derived from uhttplib written by Erik van Zijst under an MIT license. # (https://pypi.org/project/uhttplib/) # Ported to Python 3 by Irit Goihman. class UnixHTTPConnection(HTTPConnection): def __init__(self, path, timeout=socket._GLOBAL_DEFAULT_TIMEOUT): self.path = path HTTPConnection.__init__(self, "localhost", timeout=timeout) def connect(self): @@ -425,28 +502,30 @@ def get_options(http, url): http.request("OPTIONS", url.path) r = http.getresponse() data = r.read() if r.status == 200: j = json.loads(data) features = j["features"] return { "can_flush": "flush" in features, "can_zero": "zero" in features, + "can_extents": "extents" in features, "unix_socket": j.get('unix_socket'), "max_readers": j.get("max_readers", 1), "max_writers": j.get("max_writers", 1), } elif r.status == 405 or r.status == 204: # Old imageio servers returned either 405 Method Not Allowed or # 204 No Content (with an empty body). return { "can_flush": False, "can_zero": False, + "can_extents": False, "unix_socket": None, "max_readers": 1, "max_writers": 1, } else: raise RuntimeError("could not use OPTIONS request: %d: %s" % (r.status, r.reason)) -- 2.33.1

1 0

[v2v PATCH v2] convert_linux: translate the first CD-ROM's references in boot conf files
by Laszlo Ersek 17 Dec '21

17 Dec '21

If the only CD-ROM in "s_removables" is on an IDE controller, and the guest kernel represents it with a /dev/hdX device node, then convert references to this device node, in the boot config files, to /dev/cdrom. On the destination (after conversion), /dev/cdrom will point to whataver node we converted the CD-ROM to, masking a potential i440fx -> q35 (IDE -> SATA) board change. If the only CD-ROM is not on an IDE controller, or the guest is modern enough to represent the IDE CD-ROM as /dev/sr0, then perform no translation. Namely, /dev/sr0 survives a potential i440fx -> q35 (IDE -> SATA) board change intact. When multiple CD-ROMs exist, emit a warning, and attempt the conversion on the first CD-ROM, as a guess. This may be inexact, but we can't do better, because: - SATA, SCSI, and (on modern guests) IDE CD-ROMs are lumped together in the /dev/sr* namespace, on the source side, and "s_removable_slot" is useless for telling them apart, as we don't know the exact controller topology (and OS enumeration order); - after conversion: some OSes don't create /dev/cdrom* symlinks to all CD-ROMs, and even if multiple such symlinks are created, their order is basically random. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1637857 Signed-off-by: Laszlo Ersek <lersek(a)redhat.com> --- Notes: v2: - Use List.length >= 2 rather than matching with three cons operations. [Rich] - At the same time, eliminate wrong precedence between "->" in a match, and ";". [Rich] - Match the first element of the cdrom list, and the contents of that element, in a single "match" expression. [Rich] - Not tested, due to the issue I described in <8b3ce08b-ea47-dc1c-f441-c8b91708bd6f(a)redhat.com> (cannot provide a mailing list URL because the archive seems to have stopped refreshing itself?) convert/convert_linux.ml | 19 +++++++++++++++++++ tests/test-v2v-cdrom.expected | 2 +- tests/test-v2v-cdrom.xml.in | 4 +++- tests/test-v2v-i-ova.xml | 2 +- 4 files changed, 24 insertions(+), 3 deletions(-) diff --git a/convert/convert_linux.ml b/convert/convert_linux.ml index 8dc648169dcb..d49ecec03aeb 100644 --- a/convert/convert_linux.ml +++ b/convert/convert_linux.ml @@ -1020,6 +1020,25 @@ let convert (g : G.guestfs) source inspect keep_serial_console _ = "xvd" ^ drive_name i, block_prefix_after_conversion ^ drive_name i ) source.s_disks in + (* Check the first CD-ROM. If its controller is IDE, and the OS is RHEL<=5, + * then translate the CD-ROM from "/dev/hd[SLOT]" to "/dev/cdrom". See + * RHBZ#1637857 for details. + *) + let cdroms = List.filter + (fun removable -> removable.s_removable_type = CDROM) + source.s_removables in + if List.length cdroms >= 2 then + warning (f_"multiple CD-ROMs found; translation of CD-ROM references \ + may be inexact"); + let map = map @ + (match cdroms with + | { s_removable_controller = Some Source_IDE; + s_removable_slot = Some slot } :: _ + when family = `RHEL_family && inspect.i_major_version <= 5 -> + [("hd" ^ drive_name slot, "cdrom")] + | _ -> [] + ) in + if verbose () then ( eprintf "block device map:\n"; List.iter ( diff --git a/tests/test-v2v-cdrom.expected b/tests/test-v2v-cdrom.expected index 34d2bf5961b0..17bd152d8e64 100644 --- a/tests/test-v2v-cdrom.expected +++ b/tests/test-v2v-cdrom.expected @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> diff --git a/tests/test-v2v-cdrom.xml.in b/tests/test-v2v-cdrom.xml.in index 6bad5eab1cd4..a6e1e3f514d5 100644 --- a/tests/test-v2v-cdrom.xml.in +++ b/tests/test-v2v-cdrom.xml.in @@ -35,7 +35,9 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='@abs_top_builddir@/test-data/phony-guests/blank-disk.img'/> -  +  <target dev='hdc' bus='ide'/> </disk> </devices> diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml index d7383905fdc0..9f3c1974243f 100644 --- a/tests/test-v2v-i-ova.xml +++ b/tests/test-v2v-i-ova.xml @@ -28,7 +28,7 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hda' bus='ide'/> + <target dev='sda' bus='sata'/> </disk> <disk device='floppy' type='file'> <driver name='qemu' type='raw'/> -- 2.19.1.3.g30247aa5d201

3 12