October 2018 - Libguestfs - libguestfs List Archives

Re: [Libguestfs] Question about mounting QCOW2 files....
by Richard W.M. Jones 16 Oct '18

16 Oct '18

[Adding libguestfs mailing list] On Sun, Oct 14, 2018 at 11:53:17AM -0400, Raghuram Devarakonda wrote: > Hi, > > My name is Raghuram Devarakonda and I am a big fan of "nbdkit". I have > successfully used it to prototype (using Python plugin) a complex > project in our company and since then, I have also been trying to > understand and learn "guestfs-tools" as well. I have a question in > this regard and I hope you don't mind my sending mail gratuitously. > > In our project, we deal with quite large sparse files whose total size > runs into several hundred terabytes or even more, though actual > allocated size on disk is much smaller. The problem is that such large > files present issues for copying around, compression, or even for > computing checksums. I am wondering of I can use guestfs-tools to > mount a QCOW2 image and then use the image as sparse file. The idea is > that actual file on the disk would be compact though to our code, > sparse file interface is preserved. It's not very clear to me exactly what you want, but in general yes qcow2 is a good way to handle very large, sparse disk images. If you can be clearer about exactly what you mean by "mount" then I could answer the question better. For example, do you mean "mount a filesystem in the qcow2 image"? In which case use guestmount. If you mean "attach the qcow2 disk as a local device" then qemu-nbd can do this. Rich. > Can you please let me know if this is possible? Can I mount a sparse > file using guestfs-tools even though it is not really a disk? > > Thanks in advance, > Raghu -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

2 3

Bug Report, .vhdx file not attaching
by Antoine KILZI 15 Oct '18

15 Oct '18

I am trying to mount a windows 10 backup .vhdx file. I installed libguestfs through ```sudo apt-get install libguestfs-tools``` I am running Ubuntu 18.04.1 LTS and this is the installed version: ```1:1.36.13-1ubuntu3.2``` This is the output of the command I ran: ``` ~$ guestmount --add Documents/8be29c38-0000-0000-0000-602200000000.vhdx --inspector --ro /media/Windows10/ libguestfs: trace: set_verbose true libguestfs: trace: set_verbose = 0 libguestfs: create: flags = 0, handle = 0x562d44e7c570, program = guestmount libguestfs: trace: set_recovery_proc false libguestfs: trace: set_recovery_proc = 0 libguestfs: trace: add_drive "Documents/8be29c38-0000-0000-0000-602200000000.vhdx" "readonly:true" libguestfs: creating COW overlay to protect original drive content libguestfs: trace: get_tmpdir libguestfs: trace: get_tmpdir = "/tmp" libguestfs: trace: disk_create "/tmp/libguestfsOz0P6y/overlay1.qcow2" "qcow2" -1 "backingfile:/home/a17kilzi/Documents/8be29c38-0000-0000-0000-602200000000.vhdx" libguestfs: command: run: qemu-img libguestfs: command: run: \ create libguestfs: command: run: \ -f qcow2 libguestfs: command: run: \ -o backing_file=/home/a17kilzi/Documents/8be29c38-0000-0000-0000-602200000000.vhdx libguestfs: command: run: \ /tmp/libguestfsOz0P6y/overlay1.qcow2 Formatting '/tmp/libguestfsOz0P6y/overlay1.qcow2', fmt=qcow2 size=142622064640 backing_file=/home/a17kilzi/Documents/8be29c38-0000-0000-0000-602200000000.vhdx cluster_size=65536 lazy_refcounts=off refcount_bits=16 libguestfs: trace: disk_create = 0 libguestfs: trace: add_drive = 0 libguestfs: trace: launch libguestfs: trace: version libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 36, release: 13, extra: , > libguestfs: trace: get_backend libguestfs: trace: get_backend = "direct" libguestfs: launch: program=guestmount libguestfs: launch: version=1.36.13 libguestfs: launch: backend registered: unix libguestfs: launch: backend registered: uml libguestfs: launch: backend registered: libvirt libguestfs: launch: backend registered: direct libguestfs: launch: backend=direct libguestfs: launch: tmpdir=/tmp/libguestfsOz0P6y libguestfs: launch: umask=0022 libguestfs: launch: euid=1000 libguestfs: is_openable: /dev/kvm: Permission denied libguestfs: trace: get_backend_setting "force_tcg" libguestfs: trace: get_backend_setting = NULL (error) libguestfs: warning: current user is not a member of the KVM group (group ID 132). This user cannot access /dev/kvm, so libguestfs may run very slowly. It is recommended that you 'chmod 0666 /dev/kvm' or add the current user to the KVM group (you might need to log out and log in again). libguestfs: trace: get_cachedir libguestfs: trace: get_cachedir = "/var/tmp" libguestfs: begin building supermin appliance libguestfs: run supermin libguestfs: command: run: /usr/bin/supermin libguestfs: command: run: \ --build libguestfs: command: run: \ --verbose libguestfs: command: run: \ --if-newer libguestfs: command: run: \ --lock /var/tmp/.guestfs-1000/lock libguestfs: command: run: \ --copy-kernel libguestfs: command: run: \ -f ext2 libguestfs: command: run: \ --host-cpu x86_64 libguestfs: command: run: \ /usr/lib/x86_64-linux-gnu/guestfs/supermin.d libguestfs: command: run: \ -o /var/tmp/.guestfs-1000/appliance.d supermin: version: 5.1.19 supermin: package handler: debian/dpkg supermin: acquiring lock on /var/tmp/.guestfs-1000/lock supermin: build: /usr/lib/x86_64-linux-gnu/guestfs/supermin.d supermin: reading the supermin appliance supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/base.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/excludefiles type uncompressed excludefiles supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/hostfiles type uncompressed hostfiles supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/init.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages type uncompressed packages supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-hfsplus type uncompressed packages supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-reiserfs type uncompressed packages supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-xfs type uncompressed packages supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar) supermin: mapping package names to installed packages supermin: resolving full list of package dependencies supermin: build: 215 packages, including dependencies supermin: build: 9953 files supermin: build: 6727 files, after matching excludefiles supermin: build: 6730 files, after adding hostfiles supermin: build: 6727 files, after removing unreadable files supermin: build: 6730 files, after munging supermin: kernel: looking for kernel using environment variables ... supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ... supermin: kernel: looking for kernels in /boot ... supermin: kernel: kernel version of /boot/vmlinuz-4.15.0-36-generic = 4.15.0-36-generic (from filename) supermin: kernel: picked modules path /lib/modules/4.15.0-36-generic supermin: kernel: picked vmlinuz /boot/vmlinuz-4.15.0-36-generic supermin: kernel: kernel_version 4.15.0-36-generic supermin: kernel: modpath /lib/modules/4.15.0-36-generic cp: cannot open '/boot/vmlinuz-4.15.0-36-generic' for reading: Permission denied supermin: cp -p '/boot/vmlinuz-4.15.0-36-generic' '/var/tmp/.guestfs-1000/appliance.d.ouzjasdp/kernel': command failed, see earlier errors libguestfs: error: /usr/bin/supermin exited with error status 1, see debug messages above libguestfs: trace: launch = -1 (error) libguestfs: trace: close libguestfs: closing guestfs handle 0x562d44e7c570 (state 0) libguestfs: command: run: rm libguestfs: command: run: \ -rf /tmp/libguestfsOz0P6y ``` And this is the output of the libguestfs-test-tool: ``` $ libguestfs-test-tool ************************************************************ * IMPORTANT NOTICE * * When reporting bugs, include the COMPLETE, UNEDITED * output below in your bug report. * ************************************************************ libguestfs: trace: set_verbose true libguestfs: trace: set_verbose = 0 libguestfs: trace: set_verbose true libguestfs: trace: set_verbose = 0 LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 PATH=/home/a17kilzi/bin:/home/a17kilzi/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin XDG_RUNTIME_DIR=/run/user/1000 SELinux: sh: 1: getenforce: not found libguestfs: trace: add_drive_scratch 104857600 libguestfs: trace: get_tmpdir libguestfs: trace: get_tmpdir = "/tmp" libguestfs: trace: disk_create "/tmp/libguestfsS9Wa6T/scratch1.img" "raw" 104857600 libguestfs: trace: disk_create = 0 libguestfs: trace: add_drive "/tmp/libguestfsS9Wa6T/scratch1.img" "format:raw" "cachemode:unsafe" libguestfs: trace: add_drive = 0 libguestfs: trace: add_drive_scratch = 0 libguestfs: trace: get_append libguestfs: trace: get_append = "NULL" guestfs_get_append: (null) libguestfs: trace: get_autosync libguestfs: trace: get_autosync = 1 guestfs_get_autosync: 1 libguestfs: trace: get_backend libguestfs: trace: get_backend = "direct" guestfs_get_backend: direct libguestfs: trace: get_backend_settings libguestfs: trace: get_backend_settings = [] guestfs_get_backend_settings: [] libguestfs: trace: get_cachedir libguestfs: trace: get_cachedir = "/var/tmp" guestfs_get_cachedir: /var/tmp libguestfs: trace: get_direct libguestfs: trace: get_direct = 0 guestfs_get_direct: 0 libguestfs: trace: get_hv libguestfs: trace: get_hv = "/usr/bin/qemu-system-x86_64" guestfs_get_hv: /usr/bin/qemu-system-x86_64 libguestfs: trace: get_memsize libguestfs: trace: get_memsize = 500 guestfs_get_memsize: 500 libguestfs: trace: get_network libguestfs: trace: get_network = 0 guestfs_get_network: 0 libguestfs: trace: get_path libguestfs: trace: get_path = "/usr/lib/x86_64-linux-gnu/guestfs" guestfs_get_path: /usr/lib/x86_64-linux-gnu/guestfs libguestfs: trace: get_pgroup libguestfs: trace: get_pgroup = 0 guestfs_get_pgroup: 0 libguestfs: trace: get_program libguestfs: trace: get_program = "libguestfs-test-tool" guestfs_get_program: libguestfs-test-tool libguestfs: trace: get_recovery_proc libguestfs: trace: get_recovery_proc = 1 guestfs_get_recovery_proc: 1 libguestfs: trace: get_smp libguestfs: trace: get_smp = 1 guestfs_get_smp: 1 libguestfs: trace: get_sockdir libguestfs: trace: get_sockdir = "/run/user/1000" guestfs_get_sockdir: /run/user/1000 libguestfs: trace: get_tmpdir libguestfs: trace: get_tmpdir = "/tmp" guestfs_get_tmpdir: /tmp libguestfs: trace: get_trace libguestfs: trace: get_trace = 1 guestfs_get_trace: 1 libguestfs: trace: get_verbose libguestfs: trace: get_verbose = 1 guestfs_get_verbose: 1 host_cpu: x86_64 Launching appliance, timeout set to 600 seconds. libguestfs: trace: launch libguestfs: trace: version libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 36, release: 13, extra: , > libguestfs: trace: get_backend libguestfs: trace: get_backend = "direct" libguestfs: launch: program=libguestfs-test-tool libguestfs: launch: version=1.36.13 libguestfs: launch: backend registered: unix libguestfs: launch: backend registered: uml libguestfs: launch: backend registered: libvirt libguestfs: launch: backend registered: direct libguestfs: launch: backend=direct libguestfs: launch: tmpdir=/tmp/libguestfsS9Wa6T libguestfs: launch: umask=0022 libguestfs: launch: euid=1000 libguestfs: is_openable: /dev/kvm: Permission denied libguestfs: trace: get_backend_setting "force_tcg" libguestfs: trace: get_backend_setting = NULL (error) libguestfs: warning: current user is not a member of the KVM group (group ID 132). This user cannot access /dev/kvm, so libguestfs may run very slowly. It is recommended that you 'chmod 0666 /dev/kvm' or add the current user to the KVM group (you might need to log out and log in again). libguestfs: trace: get_cachedir libguestfs: trace: get_cachedir = "/var/tmp" libguestfs: begin building supermin appliance libguestfs: run supermin libguestfs: command: run: /usr/bin/supermin libguestfs: command: run: \ --build libguestfs: command: run: \ --verbose libguestfs: command: run: \ --if-newer libguestfs: command: run: \ --lock /var/tmp/.guestfs-1000/lock libguestfs: command: run: \ --copy-kernel libguestfs: command: run: \ -f ext2 libguestfs: command: run: \ --host-cpu x86_64 libguestfs: command: run: \ /usr/lib/x86_64-linux-gnu/guestfs/supermin.d libguestfs: command: run: \ -o /var/tmp/.guestfs-1000/appliance.d supermin: version: 5.1.19 supermin: package handler: debian/dpkg supermin: acquiring lock on /var/tmp/.guestfs-1000/lock supermin: build: /usr/lib/x86_64-linux-gnu/guestfs/supermin.d supermin: reading the supermin appliance supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/base.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/excludefiles type uncompressed excludefiles supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/hostfiles type uncompressed hostfiles supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/init.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages type uncompressed packages supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-hfsplus type uncompressed packages supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-reiserfs type uncompressed packages supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-xfs type uncompressed packages supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar) supermin: mapping package names to installed packages supermin: resolving full list of package dependencies supermin: build: 215 packages, including dependencies supermin: build: 9953 files supermin: build: 6727 files, after matching excludefiles supermin: build: 6730 files, after adding hostfiles supermin: build: 6727 files, after removing unreadable files supermin: build: 6730 files, after munging supermin: kernel: looking for kernel using environment variables ... supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ... supermin: kernel: looking for kernels in /boot ... supermin: kernel: kernel version of /boot/vmlinuz-4.15.0-36-generic = 4.15.0-36-generic (from filename) supermin: kernel: picked modules path /lib/modules/4.15.0-36-generic supermin: kernel: picked vmlinuz /boot/vmlinuz-4.15.0-36-generic supermin: kernel: kernel_version 4.15.0-36-generic supermin: kernel: modpath /lib/modules/4.15.0-36-generic cp: cannot open '/boot/vmlinuz-4.15.0-36-generic' for reading: Permission denied supermin: cp -p '/boot/vmlinuz-4.15.0-36-generic' '/var/tmp/.guestfs-1000/appliance.d.mv7vaue2/kernel': command failed, see earlier errors libguestfs: error: /usr/bin/supermin exited with error status 1, see debug messages above libguestfs: trace: launch = -1 (error) libguestfs: trace: close libguestfs: closing guestfs handle 0x562c6e1c50d0 (state 0) libguestfs: command: run: rm libguestfs: command: run: \ -rf /tmp/libguestfsS9Wa6T ```

3 2

Re: [Libguestfs] Provide NBD via Browser over Websockets
by Eric Blake 13 Oct '18

13 Oct '18

[adding nbdkit readers] On 10/13/18 1:39 PM, Eric Wheeler wrote: > Hello all, > > It might be neat to attach ISOs to KVM guests via websockets. Basically > the browser would be the NBD "server" and an NBD client would run on the > hypervisor, then use `virsh change-media vm1 hdc --insert /dev/nbd0` could > use an ISO from my desk to boot from. Are you using qemu as the hypervisor? If you are using something else, like vmware, then using the kernel NBD module as the NBD client to expose the browser-as-server via /dev/nbd0 is reasonable since practically any hypervisor will attach to a raw block device. But if you are using qemu, I'd suggest that you use qemu directly as the NBD client, rather than indirecting through the kernel module (for that matter, qemu's NBD client knows how to efficiently handle sparse files if the server is capable, while to date the kernel nbd module does not). Since you are using virsh, the libvirt list is a better resource for figuring out the command line and/or XML changes to direct qemu to connect as an NBD client. > > Here's an HTML5 open file example: > https://stackoverflow.com/questions/3582671/how-to-open-a-local-disk-file-w… > > and the NBD protocol looks simple enough to implement in javascript: > https://stackoverflow.com/questions/17295140/where-is-the-network-block-dev… > > What do you think? Does anyone have an interest in doing this? The nbdkit project (under the libguestfs umbrella) has already implemented an NBD server that can bridge to an image served over http, if you want to use that as a starting point. Then the obvious question - do you really need to implement a solution using websockets through your browser in order to serve up a file on your desktop, or can you reuse existing solutions of a userspace NBD server to directly serve the file without involving the browser in the middle. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

1 0

virt-builder fail with "bridge ‘virbr0’ not found" (fixed by LIBGUESTFS_BACKEND=direct ?!)
by Nir Soffer 12 Oct '18

12 Oct '18

I'm trying to build fedora-27 image for testing uploads: $ virt-builder fedora-27 -o /var/tmp/fedora-27.img ... [ 25.2] Opening the new disk virt-builder: error: libguestfs error: bridge ‘virbr0’ not found. Try running: brctl show to get a list of bridges on the host, and then selecting the bridge you wish the appliance network to connect to using: export _SETTINGS=network_bridge=<bridge name> I don't want to add bridges to my new fresh oVirt test host, and remember that this used to work previously without the bridge. So I tried the usual "fix" (I know it should not be used in production): $ export LIBGUESTFS_BACKEND=direct And for some reason this works now: $ virt-builder fedora-27 -o /var/tmp/fedora-27.img ... [ 41.2] Finishing off Output file: /var/tmp/fedora-27.img Output size: 6.0G Output format: raw Total usable space: 5.3G Free space: 4.4G (81%) How LIBGUESTFS_BACKEND is related to the bridge? why do we need the bridge for building an image? Nir

4 5

OpenStack output workflow
by Fabien Dupont 12 Oct '18

12 Oct '18

Hi, There has been discussion about the OpenStack output and Richard asked for a public thread on this list, so here it is. For v2v from VMware to RHV, there is a Python script that does some extra steps to create the virtual machine after the disks have been converted. We want to have the same behavior for OpenStack, i.e. have virt-v2v create the instance once the volumes have been created. For that, I've written a Python script [1] that takes a JSON file (sample here [2]) as input. I expect this JSON input to be generated by virt-v2v openstack output module, from the command line options and the volumes ids generated during conversion. Here are the options I think we should have for the OpenStack output: -o openstack -oo os-auth-url='http://controller.example.com:5000/v3' -oo os-user-domain-name='Default' -oo os-project-name='v2v-project' -oo os-username='admin' -oo os-password='secret' -oo server-id='01234567-89ab-cdef-0123-456789abcdef' -oo destination_project_id='01234567-89ab-cdef-0123-456789abcdef' -oo volume_type_id='01234567-89ab-cdef-0123-456789abcdef' -oo flavor_id='01234567-89ab-cdef-0123-456789abcdef' -oo security_groups_ids='01234567-89ab-cdef-0123-456789abcdef,01234567-89ab-cdef-0123-456789abcdef' --mac 01:23:45:67:89:ab:network:01234567-89ab-cdef-0123-456789abcdef You'll see that the --mac option is not specific to OpenStack, but it shows how it would look like with a network id. And it should be passed to the post-conversion script. The translation to JSON is pretty straight forward and should not be difficult. We simply have to agree on the JSON keys we expect and the where the new -oo keys go. Also, the script is quite simple and relies on OpenStack Python SDK, which is also used by the OpenStack CLI, so no additional dependencies are required and it should be easy to maintain. [1] https://gist.github.com/fdupont-redhat/934b3efb6d66a991a80149235066d7d7#fil… [2] https://gist.github.com/fdupont-redhat/934b3efb6d66a991a80149235066d7d7#fil… -- *Fabien Dupont* PRINCIPAL SOFTWARE ENGINEER Red Hat - Solutions Engineering fabien(a)redhat.com M: +33 (0) 662 784 971 <+33662784971> <http://redhat.com> *TRIED. TESTED. TRUSTED.* Twitter: @redhatway <https://twitter.com/redhatway> | Instagram: @redhatinc <https://www.instagram.com/redhatinc/> | Snapchat: @redhatsnaps

2 10

[PATCH] v2v: ovf: add firmware and machine type element
by Tomáš Golembiovský 10 Oct '18

10 Oct '18

Add oVirt specific elemnt to OVF. It represents the combination of machine type (i440fx/q35) and firmware (BIOS/UEFI). Signed-off-by: Tomáš Golembiovský <tgolembi(a)redhat.com> --- v2v/create_ovf.ml | 20 +++++++++++++++++++- v2v/create_ovf.mli | 2 +- v2v/output_rhv.ml | 6 ++---- v2v/output_rhv_upload.ml | 4 ++-- v2v/output_vdsm.ml | 6 ++---- v2v/test-v2v-o-rhv.ovf.expected | 1 + v2v/test-v2v-o-vdsm-options.ovf.expected | 1 + 7 files changed, 28 insertions(+), 12 deletions(-) diff --git a/v2v/create_ovf.ml b/v2v/create_ovf.ml index 2cf610333..1cab11dfd 100644 --- a/v2v/create_ovf.ml +++ b/v2v/create_ovf.ml @@ -462,6 +462,22 @@ let origin_of_source_hypervisor = function *) | _ -> None +let get_ovirt_biostype guestcaps target_firmware = + let uefi_firmware = + match target_firmware with + | TargetBIOS -> None + | TargetUEFI -> Some (find_uefi_firmware guestcaps.gcaps_arch) in + let secure_boot_required = + match uefi_firmware with + | Some { Uefi.flags = flags } + when List.mem Uefi.UEFI_FLAG_SECURE_BOOT_REQUIRED flags -> true + | _ -> false in + match target_firmware, secure_boot_required with + | TargetUEFI, true -> 3 (* q35 + UEFI + secure boot *) + | TargetUEFI, _ -> 2 (* q35 + UEFI *) + (* 1 is q35 + SeaBIOS *) + | _, _ -> 0 (* i440fx + SeaBIOS *) + (* Generate the .meta file associated with each volume. *) let create_meta_files output_alloc sd_uuid image_uuids targets = (* Note: Upper case in the .meta, mixed case in the OVF. *) @@ -506,7 +522,7 @@ let create_meta_files output_alloc sd_uuid image_uuids targets = ) (List.combine targets image_uuids) (* Create the OVF file. *) -let rec create_ovf source targets guestcaps inspect +let rec create_ovf source targets guestcaps inspect target_firmware output_alloc sd_uuid image_uuids vol_uuids vm_uuid ovf_flavour = assert (List.length targets = List.length vol_uuids); @@ -515,6 +531,7 @@ let rec create_ovf source targets guestcaps inspect let vmtype = get_vmtype inspect in let vmtype = match vmtype with `Desktop -> "0" | `Server -> "1" in let ostype = get_ostype inspect in + let biostype = get_ovirt_biostype guestcaps target_firmware in let ovf : doc = doc "ovf:Envelope" [ @@ -562,6 +579,7 @@ let rec create_ovf source targets guestcaps inspect e "VmType" [] [PCData vmtype]; (* See https://bugzilla.redhat.com/show_bug.cgi?id=1260590#c17 *) e "DefaultDisplayType" [] [PCData "1"]; + e "BiosType" [] [PCData (string_of_int biostype)]; ] in (match source.s_cpu_model with diff --git a/v2v/create_ovf.mli b/v2v/create_ovf.mli index 8200b76f9..cb6c12690 100644 --- a/v2v/create_ovf.mli +++ b/v2v/create_ovf.mli @@ -43,7 +43,7 @@ val ovf_flavour_to_string : ovf_flavour -> string create OVF for another target management system then we would need to heavily modify or even duplicate this code. *) -val create_ovf : Types.source -> Types.target list -> Types.guestcaps -> Types.inspect -> Types.output_allocation -> string -> string list -> string list -> string -> ovf_flavour -> DOM.doc +val create_ovf : Types.source -> Types.target list -> Types.guestcaps -> Types.inspect -> Types.target_firmware -> Types.output_allocation -> string -> string list -> string list -> string -> ovf_flavour -> DOM.doc (** Create the OVF file. Actually a {!DOM} document is created, not a file. It can be written diff --git a/v2v/output_rhv.ml b/v2v/output_rhv.ml index 5260ab030..ce57f0309 100644 --- a/v2v/output_rhv.ml +++ b/v2v/output_rhv.ml @@ -115,7 +115,7 @@ object method as_options = sprintf "-o rhv -os %s" os - method supported_firmware = [ TargetBIOS ] + method supported_firmware = [ TargetBIOS; TargetUEFI ] (* RHV doesn't support serial consoles. This causes the conversion * step to remove it. @@ -270,12 +270,10 @@ object (* This is called after conversion to write the OVF metadata. *) method create_metadata source targets _ guestcaps inspect target_firmware = - (* See #supported_firmware above. *) - assert (target_firmware = TargetBIOS); (* Create the metadata. *) let ovf = Create_ovf.create_ovf source targets guestcaps inspect - output_alloc esd_uuid image_uuids vol_uuids vm_uuid + target_firmware output_alloc esd_uuid image_uuids vol_uuids vm_uuid Create_ovf.RHVExportStorageDomain in (* Write it to the metadata file. *) diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml index 63fa2411a..3dcd6d952 100644 --- a/v2v/output_rhv_upload.ml +++ b/v2v/output_rhv_upload.ml @@ -253,7 +253,7 @@ object sprintf " -oc %s -op %s -os %s" output_conn output_password output_storage - method supported_firmware = [ TargetBIOS ] + method supported_firmware = [ TargetBIOS; TargetUEFI ] (* rhev-apt.exe will be installed (if available). *) method install_rhev_apt = true @@ -407,7 +407,7 @@ If the messages above are not sufficient to diagnose the problem then add the (* Create the metadata. *) let ovf = Create_ovf.create_ovf source targets guestcaps inspect - output_alloc + target_firmware output_alloc sd_uuid image_uuids vol_uuids vm_uuid OVirt in let ovf = DOM.doc_to_string ovf in diff --git a/v2v/output_vdsm.ml b/v2v/output_vdsm.ml index 9a1b748bc..f32acedae 100644 --- a/v2v/output_vdsm.ml +++ b/v2v/output_vdsm.ml @@ -123,7 +123,7 @@ object | flav -> sprintf "-oo vdsm-ovf-flavour=%s" (Create_ovf.ovf_flavour_to_string flav)) - method supported_firmware = [ TargetBIOS ] + method supported_firmware = [ TargetBIOS; TargetUEFI ] (* RHV doesn't support serial consoles. This causes the conversion * step to remove it. @@ -240,11 +240,9 @@ object (* This is called after conversion to write the OVF metadata. *) method create_metadata source targets _ guestcaps inspect target_firmware = - (* See #supported_firmware above. *) - assert (target_firmware = TargetBIOS); - (* Create the metadata. *) let ovf = Create_ovf.create_ovf source targets guestcaps inspect + target_firmware output_alloc dd_uuid vdsm_options.image_uuids vdsm_options.vol_uuids diff --git a/v2v/test-v2v-o-rhv.ovf.expected b/v2v/test-v2v-o-rhv.ovf.expected index 7bcc456c5..c6bd05c56 100644 --- a/v2v/test-v2v-o-rhv.ovf.expected +++ b/v2v/test-v2v-o-rhv.ovf.expected @@ -25,6 +25,7 @@ <IsStateless>False</IsStateless> <VmType>0</VmType> <DefaultDisplayType>1</DefaultDisplayType> + <BiosType>0</BiosType> <Section ovf:id='#VM_ID#' ovf:required='false' xsi:type='ovf:OperatingSystemSection_Type'> <Info>Microsoft Windows 7 Phony Edition</Info> <Description>Windows7</Description> diff --git a/v2v/test-v2v-o-vdsm-options.ovf.expected b/v2v/test-v2v-o-vdsm-options.ovf.expected index abaf37e54..f0d418b46 100644 --- a/v2v/test-v2v-o-vdsm-options.ovf.expected +++ b/v2v/test-v2v-o-vdsm-options.ovf.expected @@ -25,6 +25,7 @@ <IsStateless>False</IsStateless> <VmType>0</VmType> <DefaultDisplayType>1</DefaultDisplayType> + <BiosType>0</BiosType> <OperatingSystemSection ovf:id='VM' ovf:required='false' ovirt:id='11'> <Info>Microsoft Windows 7 Phony Edition</Info> <Description>Windows7</Description> -- 2.18.0

3 5

Re: [Libguestfs] LZO compression for NBD ?
by Eric Blake 04 Oct '18

04 Oct '18

[adding libguestfs list, for nbdkit reference below] On 10/4/18 8:39 AM, Stefan Fröberg wrote: > Hello. > Is it possible to improve NBD throughtput with LZO compression ? As in, have a way for the client and server to negotiate that both understand an LZO extension, at which point the client can request a read or write with a flag set to mark that the data is LZO-compressed, for less network traffic but higher CPU usage between the client and server? The idea might have merit, you're certainly welcome to propose it as an extension (we can help with word-smithing the extension into the NBD specification and ideas of the best way to negotiate the feature); it is also helpful if you also plan on showing a reference implementation of both a server and client implementing the extension. Would it always have to be LZO compression, or would it be smarter to have the negotiation phase pass a list of names of compression algorithms, where clients and servers can both advertise their list of algorithms they support, and then settle on one that is common to both sides? Is it better to request that ALL read/write transactions automatically be compressed by the negotiated algorithm, or only the specific transactions where the client requests compression (after all, the overhead of compressing a small transaction might be worse than just sending the data directly; while the benefits of increased CPU times for compression/decompression are more likely to matter for large transactions where the transmission time becomes the bottleneck). On a different track, you may want to try some experiments with LZO compression by using nbdkit. Assuming that the real bottleneck that you are trying to speed up is TCP/IP traffic, while local NBD connections over a Unix socket are much faster, you could replace: client => uncompressed over TCP/IP => server with client => uncompressed over Unix => nbdkit with custom plugin => compressed over TCP/IP => custom wrapper => server where you write a wrapper on the server side to perform the LZO compression (nbdkit does not yet provide a plugin library for easy writing of a client, but maybe it should), then a plugin on the client side that can interpret your LZO-compressed stream on the client side. nbdkit already comes with a plugin that can read from .xz files (slightly different from compressing individual transactions). But the point remains that if what you are trying to design is an alternative network stream with compressed data while still having a traditional NBD interface once the data is local, then using nbdkit or other intermediaries as boundary points can prove useful. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

1 0

[PATCH v2] lib: Use qemu-img info -U option to avoid locking error.
by Richard W.M. Jones 04 Oct '18

04 Oct '18

https://bugs.launchpad.net/qemu/+bug/1740364 --- lib/guestfs-internal.h | 3 +++ lib/handle.c | 2 ++ lib/info.c | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+) diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h index adeb9478a..c66c55e70 100644 --- a/lib/guestfs-internal.h +++ b/lib/guestfs-internal.h @@ -510,6 +510,9 @@ struct guestfs_h { /* Cached features. */ struct cached_feature *features; size_t nr_features; + + /* Used by lib/info.c. -1 = not tested or error; else 0 or 1. */ + int qemu_img_supports_U_option; }; /** diff --git a/lib/handle.c b/lib/handle.c index a47aaafab..297ff6d67 100644 --- a/lib/handle.c +++ b/lib/handle.c @@ -101,6 +101,8 @@ guestfs_create_flags (unsigned flags, ...) g->memsize = DEFAULT_MEMSIZE; + g->qemu_img_supports_U_option = -1; /* not tested, see lib/info.c */ + /* Start with large serial numbers so they are easy to spot * inside the protocol. */ diff --git a/lib/info.c b/lib/info.c index 2eadc1c11..74e4424b8 100644 --- a/lib/info.c +++ b/lib/info.c @@ -57,6 +57,7 @@ cleanup_json_t_decref (void *ptr) #endif static json_t *get_json_output (guestfs_h *g, const char *filename); +static int qemu_img_supports_U_option (guestfs_h *g); static void set_child_rlimits (struct command *); char * @@ -149,6 +150,11 @@ get_json_output (guestfs_h *g, const char *filename) guestfs_int_cmd_add_arg (cmd, "qemu-img"); guestfs_int_cmd_add_arg (cmd, "info"); + switch (qemu_img_supports_U_option (g)) { + case -1: return NULL; + case 0: break; + default: guestfs_int_cmd_add_arg (cmd, "-U"); + } guestfs_int_cmd_add_arg (cmd, "--output"); guestfs_int_cmd_add_arg (cmd, "json"); if (filename[0] == '/') @@ -218,3 +224,36 @@ set_child_rlimits (struct command *cmd) guestfs_int_cmd_set_child_rlimit (cmd, RLIMIT_CPU, 10 /* seconds */); #endif } + +/** + * Test if the qemu-img info command supports the C<-U> option to + * disable locking. The result is memoized in the handle. + * + * Note this option was added in qemu 2.11. We can remove this test + * when we can assume everyone is using qemu >= 2.11. + */ +static int +qemu_img_supports_U_option (guestfs_h *g) +{ + if (g->qemu_img_supports_U_option >= 0) + return g->qemu_img_supports_U_option; + + CLEANUP_CMD_CLOSE struct command *cmd = guestfs_int_new_command (g); + int r; + + guestfs_int_cmd_add_string_unquoted (cmd, + "qemu-img info --help | " + "grep -sq -- 'info.*-U'"); + r = guestfs_int_cmd_run (cmd); + if (r == -1) + return -1; + if (!WIFEXITED (r)) { + guestfs_int_external_command_failed (g, r, + "qemu-img info -U option test", + NULL); + return -1; + } + + g->qemu_img_supports_U_option = WEXITSTATUS (r) == 0; + return g->qemu_img_supports_U_option; +} -- 2.19.0.rc0

3 7

[PATCH] v2v: -it vddk: Add a warning about using vddk* options (RHBZ#1527334).
by Richard W.M. Jones 04 Oct '18

04 Oct '18

Stress these options shouldn't be used unless you know what you're doing. --- v2v/virt-v2v.pod | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/v2v/virt-v2v.pod b/v2v/virt-v2v.pod index fbc5733a6..18db7206b 100644 --- a/v2v/virt-v2v.pod +++ b/v2v/virt-v2v.pod @@ -439,7 +439,8 @@ See L</INPUT FROM VDDK> below for details. When using VDDK mode, these options are passed unmodified to the L<nbdkit(1)> VDDK plugin. Please refer to L<nbdkit-vddk-plugin(1)>. -These are all optional. +Do not use these options unless you know what you are doing. These +are all optional. =item B<-ip> filename @@ -1894,6 +1895,7 @@ Other options that you might need to add in rare circumstances include I<-io vddk-config>, I<-io vddk-cookie>, I<-io vddk-nfchostport>, I<-io vddk-port>, I<-io vddk-snapshot>, and I<-io vddk-transports>, which are all explained in the L<nbdkit-vddk-plugin(1)> documentation. +Do not use these options unless you know what you are doing. =head2 VDDK: DEBUGGING VDDK FAILURES -- 2.19.0.rc0

2 1

[PATCH] v2v: Remove ‘-io vimapiver’ option (RHBZ#1527334).
by Richard W.M. Jones 04 Oct '18

04 Oct '18

This option was added in error. It never had any effect and now the nbdkit VDDK plugin ignores it. Virt-v2v users shouldn't have been using it. This removes the option completely (so if anyone was using it they will now get an error). --- v2v/cmdline.ml | 2 -- v2v/input_libvirt_vddk.ml | 4 +--- v2v/test-v2v-docs.sh | 1 - v2v/virt-v2v.pod | 7 ++----- 4 files changed, 3 insertions(+), 11 deletions(-) diff --git a/v2v/cmdline.ml b/v2v/cmdline.ml index cd2d279e9..6ec076151 100644 --- a/v2v/cmdline.ml +++ b/v2v/cmdline.ml @@ -259,8 +259,6 @@ let parse_cmdline () = s_"Same as ‘-io vddk-thumbprint=thumbprint’"; [ L"vddk-transports" ], Getopt.String ("transports", set_input_option_compat "vddk-transports"), s_"Same as ‘-io vddk-transports=transports’"; - [ L"vddk-vimapiver" ], Getopt.String ("apiver", set_input_option_compat "vddk-vimapiver"), - s_"Same as ‘-io vddk-vimapiver=apiver’"; [ L"vdsm-compat" ], Getopt.String ("0.10|1.1", set_output_option_compat "vdsm-compat"), s_"Same as ‘-oo vdsm-compat=0.10|1.1’"; [ L"vdsm-image-uuid" ], Getopt.String ("uuid", set_output_option_compat "vdsm-image-uuid"), diff --git a/v2v/input_libvirt_vddk.ml b/v2v/input_libvirt_vddk.ml index f74ca05af..2ca0ef1e0 100644 --- a/v2v/input_libvirt_vddk.ml +++ b/v2v/input_libvirt_vddk.ml @@ -44,8 +44,7 @@ let vddk_option_keys = "port"; "snapshot"; "thumbprint"; - "transports"; - "vimapiver" ] + "transports" ] let print_input_options () = printf (f_"Input options (-io) which can be used with -it vddk: @@ -64,7 +63,6 @@ All other settings are optional: VDDK snapshot moref -io vddk-transports=MODE:MODE:.. VDDK transports - -io vddk-vimapiver=APIVER VDDK vimapiver Refer to nbdkit-vddk-plugin(1) and the VDDK documentation for further information on these settings. diff --git a/v2v/test-v2v-docs.sh b/v2v/test-v2v-docs.sh index 64dfe5492..dfb12bb14 100755 --- a/v2v/test-v2v-docs.sh +++ b/v2v/test-v2v-docs.sh @@ -46,7 +46,6 @@ $top_srcdir/podcheck.pl virt-v2v.pod virt-v2v \ --vddk-snapshot,\ --vddk-thumbprint,\ --vddk-transports,\ ---vddk-vimapiver,\ --vdsm-compat,\ --vdsm-image-uuid,\ --vdsm-ovf-flavour,\ diff --git a/v2v/virt-v2v.pod b/v2v/virt-v2v.pod index e028540e2..fbc5733a6 100644 --- a/v2v/virt-v2v.pod +++ b/v2v/virt-v2v.pod @@ -437,8 +437,6 @@ See L</INPUT FROM VDDK> below for details. =item B<-io vddk-transports=>MODE:MODE:... -=item B<-io vddk-vimapiver=>APIVER - When using VDDK mode, these options are passed unmodified to the L<nbdkit(1)> VDDK plugin. Please refer to L<nbdkit-vddk-plugin(1)>. These are all optional. @@ -1894,9 +1892,8 @@ SSL thumbprint: Other options that you might need to add in rare circumstances include I<-io vddk-config>, I<-io vddk-cookie>, I<-io vddk-nfchostport>, -I<-io vddk-port>, I<-io vddk-snapshot>, I<-io vddk-transports> and -I<-io vddk-vimapiver>, which are all explained in the -L<nbdkit-vddk-plugin(1)> documentation. +I<-io vddk-port>, I<-io vddk-snapshot>, and I<-io vddk-transports>, +which are all explained in the L<nbdkit-vddk-plugin(1)> documentation. =head2 VDDK: DEBUGGING VDDK FAILURES -- 2.19.0.rc0

2 1