July 2020 - Libguestfs - libguestfs List Archives

[PATCH] daemon: inspect_fs_windows: Handle parted errors
by Sam Eiderman 22 Sep '20

22 Sep '20

By creating an empty disk and using it as the first disk of the vm (i.e. /dev/sda, /dev/sdb{1,2} contains the windows fses) we change the iteration order of the disks. This causes inspect_os() to fail since Parted returns a Unix_error if the device does not contain any partitions - fix this by handling this Unix_error. Signed-off-by: Sam Eiderman <sameid(a)google.com> --- daemon/inspect_fs_windows.ml | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/daemon/inspect_fs_windows.ml b/daemon/inspect_fs_windows.ml index c4a05bc38..bc6b98b60 100644 --- a/daemon/inspect_fs_windows.ml +++ b/daemon/inspect_fs_windows.ml @@ -365,8 +365,10 @@ and map_registry_disk_blob_mbr devices blob = let device = List.find ( fun dev -> - Parted.part_get_parttype dev = "msdos" && - pread dev 4 0x01b8 = diskid + try + Parted.part_get_parttype dev = "msdos" && + pread dev 4 0x01b8 = diskid + with Unix.Unix_error (Unix.EINVAL, _, _) -> false ) devices in (* Next 8 bytes are the offset of the partition in bytes(!) given as @@ -402,7 +404,10 @@ and map_registry_disk_blob_gpt partitions blob = fun part -> let partnum = Devsparts.part_to_partnum part in let device = Devsparts.part_to_dev part in - let typ = Parted.part_get_parttype device in + let typ = + try + Parted.part_get_parttype device + with Unix.Unix_error (Unix.EINVAL, _, _) -> "unknown" in if typ <> "gpt" then false else ( let guid = Parted.part_get_gpt_guid device partnum in -- 2.27.0.212.ge8ba1cc988-goog

2 3

[PATCH v2] v2v: fix UEFI bootloader for linux guests
by Denis Plotnikov 26 Aug '20

26 Aug '20

v2: Rich, I hope I've done all modifications according to your comments, namely: * moved the code from linux_bootloaders to convert_linux * made minor code modifications --- Not all UEFI guests can survive conversion, because of lost bootloader information in UEFI NVRAM. But some guest can cope with this because they have a fallback bootloader and use UEFI Removable Media Boot Behavior to load (see https://uefi.org/sites/default/files/resources/UEFI_Spec_2_8_A_Feb14.pdf 3.5.1.1 Removable Media Boot Behavior). If UEFI firmware can't find a bootloader in its settings it uses the removable media boot behavior to find a bootloader. We can fix the guests which don't have such a fallback loader by providing a temporary one. This bootloader is used for the first boot only, then the conversion script restores the initial bootloader settings and removes the temporary loader. Signed-off-by: Denis Plotnikov <dplotnikov(a)virtuozzo.com> --- v2v/convert_linux.ml | 123 ++++++++++++++++++++++++++++++++++++++ v2v/linux_bootloaders.ml | 7 +++ v2v/linux_bootloaders.mli | 4 ++ 3 files changed, 134 insertions(+) diff --git a/v2v/convert_linux.ml b/v2v/convert_linux.ml index e91ae120f..045afa98b 100644 --- a/v2v/convert_linux.ml +++ b/v2v/convert_linux.ml @@ -1122,6 +1122,129 @@ shutdown -h -P +0 Linux.augeas_reload g ); + (* Some linux uefi setups can't boot after conversion because of + lost uefi boot entries. The uefi boot entries are stored in uefi + NVRAM. The NVRAM content isn't a part of vm disk content and + usualy can't be converted alongside the vm. + If a vm doesn't have uefi fallback path (/EFI/BOOT/BOOT<arch>.efi) + the vm is unbootable after conversion. + The following code tries to make an uefi fallback path for + a uefi linux vm. + *) + match inspect.i_firmware with + | I_UEFI _ -> ( + (* Standard uefi fallback path *) + let uefi_fallback_path = "/boot/efi/EFI/BOOT/" in + + (* Helper function checks if 'source' contains 's' *) + let contains source s = ( + let re = Str.regexp_string s in + try + ignore (Str.search_forward re source 0); + true + with Not_found -> false + ) in + + let cant_fix_uefi () = + info (f_"Can't fix UEFI bootloader. VM may not boot.") + in + + let get_uefi_arch_suffix = function + | "x86_64" -> Some "X64" + | "x86_32" -> Some "X32" + | _ -> None + in + + match get_uefi_arch_suffix inspect.i_arch with + | None -> cant_fix_uefi () + | Some suffix -> ( + let uefi_fallback_name = + sprintf "%sBOOT%s.EFI" uefi_fallback_path suffix in + + let file_exists file = + if g#exists file then + true + else ( + info (f_"Can't find file: '%s' needed for UEFI fixing") + file; + false ) + in + + let grub_config = bootloader#get_config_file () in + + let grub_path = + String.sub grub_config 0 (String.rindex grub_config '/') in + + if g#exists uefi_fallback_name then + (* don't do anything if uefi fallback exists *) + () + else ( + info (f_"Fixing UEFI bootloader."); + match inspect.i_distro, inspect.i_major_version with + | "centos", 6 -> + (* to make a bootable uefi centos 6 we need to + * copy grub.efi and grub.conf to UEFI fallback path + * and rename them to BOOT<arch>.efi and BOOT<arch>.conf + * correspondingly *) + let uefi_grub_name = String.concat "" [grub_path; "/grub.efi"] in + let uefi_grub_conf = String.concat "" [ + String.sub uefi_fallback_name 0 + (String.rindex uefi_fallback_name '.'); + ".conf" ] in + if file_exists uefi_grub_name && file_exists grub_config then ( + g#mkdir_p uefi_fallback_path; + g#cp uefi_grub_name uefi_fallback_name; + g#cp grub_config uefi_grub_conf; + let fix_script = sprintf +"#!/bin/bash +efibootmgr -c -L \"CentOS 6\" +rm -rf %s" uefi_fallback_path in + Firstboot.add_firstboot_script + g inspect.i_root "fix uefi boot" fix_script) + else + cant_fix_uefi () + | "ubuntu", 14 -> + (* to make a bootable uefi ubuntu 14 we need to + * copy shim<arch>.efi to UEFI fallback path + * and rename it to BOOT<arch>.efi, also we copy + * grub.efi and grub.cfg to UEFI fallback path without renaming *) + let arch_suffix = String.lowercase_ascii suffix in + + let shim = + String.concat "" [grub_path; "/shim"; arch_suffix; ".efi"] in + let uefi_grub_name = + String.concat "" [grub_path; "/grub"; arch_suffix; ".efi"] in + + if file_exists shim && file_exists uefi_grub_name + && file_exists grub_config then ( + g#mkdir_p uefi_fallback_path; + g#cp shim uefi_fallback_name; + g#cp uefi_grub_name uefi_fallback_path; + g#cp grub_config uefi_fallback_path; + (* if the shim is at the standard path, clean up uefi fixing + * if not, then just don't clean up and leave the temp loader + * at UEFI fallback path for simplicity + *) + if contains shim "/boot/efi/EFI/ubuntu/shim" then + let fix_script = sprintf +"#!/bin/bash +sudo efibootmgr -c -L ubuntu -l \\\\EFI\\\\ubuntu\\\\shim%s.efi +rm -rf %s" arch_suffix uefi_fallback_path in + Firstboot.add_firstboot_script + g inspect.i_root "fix uefi boot" fix_script + else + ()) + else + cant_fix_uefi () + | _, _ -> + info (f_"No UEFI fix rule for %s %d") + inspect.i_distro inspect.i_major_version; + cant_fix_uefi () + ) + ) + ) + | I_BIOS -> (); + (* Delete blkid caches if they exist, since they will refer to the old * device names. blkid will rebuild these on demand. * diff --git a/v2v/linux_bootloaders.ml b/v2v/linux_bootloaders.ml index de3d107e9..4ca28782b 100644 --- a/v2v/linux_bootloaders.ml +++ b/v2v/linux_bootloaders.ml @@ -36,6 +36,7 @@ class virtual bootloader = object method virtual configure_console : unit -> unit method virtual remove_console : unit -> unit method update () = () + method virtual get_config_file : unit -> string end (* Helper function for SUSE: remove (hdX,X) prefix from a path. *) @@ -184,6 +185,9 @@ object loop paths; g#aug_save () + + method get_config_file () = + grub_config end (** The method used to get and set the default kernel in Grub2. *) @@ -342,6 +346,9 @@ object (self) method update () = ignore (g#command [| grub2_mkconfig_cmd; "-o"; grub_config |]) + + method get_config_file () = + grub_config end (* Helper type used in detect_bootloader. *) diff --git a/v2v/linux_bootloaders.mli b/v2v/linux_bootloaders.mli index 30cdfe3c7..d9f0be8e1 100644 --- a/v2v/linux_bootloaders.mli +++ b/v2v/linux_bootloaders.mli @@ -44,6 +44,10 @@ class virtual bootloader : object (** Update the bootloader: For grub2 only this runs the [grub2-mkconfig] command to rebuild the configuration. This is not necessary for grub-legacy. *) + + method virtual get_config_file : unit -> string + (** Returns the path to the bootloader config file, + e.g /boot/grub/grub.cfg *) end (** Encapsulates a Linux boot loader as object. *) -- 2.17.0

3 4

nbdkit build failure in Koji
by Richard W.M. Jones 04 Aug '20

04 Aug '20

Hi Eric, I wonder if you have any thoughts about this build failure in tests/test-nozero.sh? https://koji.fedoraproject.org/koji/taskinfo?taskID=48259627 log: https://kojipkgs.fedoraproject.org//work/tasks/9762/48259762/build.log The error is “nozero6.img was trimmed by mistake”. I added “set -x” to the script earlier today so we can see exactly what's wrong, and it is that: ++ stat -c %b nozero2.img ++ stat -c %b nozero6.img + test 4096 '!=' 2048 + echo 'nozero6.img was trimmed by mistake' AFAICT what this means is that nozero2.img is growing during the test (from 2048 to 4096 blocks). When I run the test locally this file stays at 2048 blocks the whole time, and the test does not fail. One other unfortunate problem is that Fedora is having lots of toolchain problems right now (see Fedora devel list passim) so we cannot really be sure that *any* other tool we are using has been built correctly :-( I've already disabled LTO in qemu and libguestfs, but possibly there are other toolchain bugs. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

4 10

[RFC nbdkit PATCH 0/4] Progress towards .list_exports
by Eric Blake 03 Aug '20

03 Aug '20

This takes Rich's API proposal and starts fleshing it out with enough meat that I was able to test 'nbdkit eval' advertising multiple exports with descriptions paired with 'qemu-nbd --list'. Eric Blake (3): server: Add exports list functions server: Prepare to use export list from plugin sh, eval: Add .list_exports support Richard W.M. Jones (1): server: Implement list_exports. docs/nbdkit-plugin.pod | 57 +++++++++- docs/nbdkit-protocol.pod | 7 +- plugins/eval/nbdkit-eval-plugin.pod | 2 + plugins/sh/nbdkit-sh-plugin.pod | 31 ++++++ include/nbdkit-common.h | 4 + include/nbdkit-filter.h | 12 +++ include/nbdkit-plugin.h | 3 + server/Makefile.am | 2 + tests/Makefile.am | 2 + server/internal.h | 6 ++ common/utils/cleanup.h | 3 + server/backend.c | 22 ++++ server/exports.c | 150 +++++++++++++++++++++++++++ server/filters.c | 9 ++ server/nbdkit.syms | 5 + server/plugins.c | 15 +++ server/protocol-handshake-newstyle.c | 80 ++++++++------ server/protocol-handshake.c | 24 +++++ common/utils/cleanup-nbdkit.c | 6 ++ plugins/sh/methods.h | 4 +- plugins/eval/eval.c | 2 + plugins/sh/methods.c | 91 ++++++++++++++++ plugins/sh/sh.c | 1 + plugins/sh/example.sh | 8 ++ tests/test-eval-exports.sh | 65 ++++++++++++ 25 files changed, 570 insertions(+), 41 deletions(-) create mode 100644 server/exports.c create mode 100755 tests/test-eval-exports.sh -- 2.28.0

2 7

[libnbd PATCH] generator: Trace return even when in wrong state
by Eric Blake 03 Aug '20

03 Aug '20

Place the tracing of return values after any generated 'out:' label, as it is important to see even the errors caused by calling a function in the wrong handle state. Fixes: 9df088bfa9 --- Multiple bugs found today, but only time to submit one patch so far. I found them all while trying to implement .list_exports in nbdkit. Bug 1: 'nbdinfo --list --json $uri' produces invalid JSON if there are 0 or more than 1 exports. We need to move the '"exports":[' and ']' into list_all_exports(), not list_one_export(), and manage correct trailing comma output. (We only tested nbdinfo with qemu-nbd, which only lists one export, explaining how we missed this) Bug 2: nbd_get_nr_list_exports() fails if the handle is still in READY state, even though there were exports listed and available in that case. I see no reason to forbid this during READY; but I've also mentioned that there are probably more changes coming to the way we do list export handling in libnbd to allow the reuse of a single handle rather than our current hacks of one handle per export. (We only tested nbdinfo with qemu-nbd, which generally does NOT allow NBD_OPT_GO on the '' export if it was advertising some other export; nbdkit is more forgiving, explaining why I hit a case where nbdinfo was calling this function in the wrong state) Bug 3: this patch. When a function fails due to use in the wrong state, the debug output was useless in telling me why. generator/C.ml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/generator/C.ml b/generator/C.ml index 4a0b45d..2c62d4e 100644 --- a/generator/C.ml +++ b/generator/C.ml @@ -542,13 +542,13 @@ let generate_lib_api_c () = pr " ret = nbd_unlocked_%s " name; print_arg_list ~wrap:true ~types:false ~handle:true args optargs; pr ";\n"; - if may_set_error then ( - pr "\n"; - print_trace_leave ret; - pr "\n" - ); + pr "\n"; if !need_out_label then pr " out:\n"; + if may_set_error then ( + print_trace_leave ret; + pr "\n" + ); if is_locked then ( pr " if (h->public_state != get_next_state (h))\n"; pr " h->public_state = get_next_state (h);\n"; -- 2.28.0

2 1

[PATCH nbdkit v2] PROPOSED: server: Implement list_exports.
by Richard W.M. Jones 31 Jul '20

31 Jul '20

Updated proposal, taking into account the default export. Instead of adding a second call, I made a couple of changes to list_exports: (1) If the plugin has a concept of a default export, it should add it as the first element in the exports list. (2) There is a new default_only flag which tells the plugin that the client is trying to request the name of the default export, so the plugin may return only a single element. However it's fine for plugins to ignore this flag. What about plugins that don't have any concept of a default export? This makes the default export be whatever happens to be first in the list. Not sure if this is a bad thing or not. Rich.

2 6

[libnbd PATCH 0/2] Expose export description
by Eric Blake 29 Jul '20

29 Jul '20

An incremental improvement on top of listing exports. I still think it's worth experimenting with revisiting how our API for list mode should actually work [1] (so that we can reuse a single connection for both grabbing the list and finally using NBD_OPT_GO), but this change was easier to whip together while still thinking about that. [1] https://www.redhat.com/archives/libguestfs/2020-July/msg00092.html Eric Blake (2): api: Get description alongside name from NBD_REP_SERVER info: Expose description in list mode lib/internal.h | 7 +++++- generator/API.ml | 21 ++++++++++++++++-- generator/states-newstyle-opt-list.c | 29 +++++++++++++++++-------- lib/handle.c | 32 ++++++++++++++++++++++++---- examples/list-exports.c | 6 +++++- interop/Makefile.am | 6 ++++-- interop/list-exports.c | 12 +++++++++-- info/info-list-json.sh | 3 ++- info/info-list.sh | 3 ++- info/nbdinfo.c | 21 +++++++++++++----- 10 files changed, 112 insertions(+), 28 deletions(-) -- 2.27.0

2 3

Latest Stable version
by Liepert, Ben 28 Jul '20

28 Jul '20

To whom it may concern, What is the latest stable version of libguestfs? On libguestfs.org, the latest stable is listed as 1.40.x, however, in the downloads page linked on this website (http://download.libguestfs.org/) 1.42 is listed as the latest stable. Thanks, Ben Liepert Stratus Technologies

2 1

[PATCH 0/7] Support Windows BitLocker (RHBZ#1808977).
by Richard W.M. Jones 27 Jul '20

27 Jul '20

These commits, along with the associated changes to common: https://www.redhat.com/archives/libguestfs/2020-March/msg00286.html support the transparent decryption and inspection of Windows guests encrypted with BitLocker encryption. To do the BitLocker decryption requires cryptsetup 2.3.0 (although cryptsetup 2.3 is not required for existing LUKS use). It also requires a new-ish Linux kernel, I believe 5.3+ This starts by adding new APIs cryptsetup-open and cryptsetup-close. As well as dropping the "luks-" prefix, this allows specifying an optional type parameter, allowing you to select BitLocker encryption. Although also the new API does not require this parameter, and will autodetect the correct type of decryption to apply in every known case. The main work is then in updating list-filesystems and ensuring that inspection still works. This really needs but also lacks tests. See commit message of patch 1/7 for why. I believe it's impossible for us to prove unit testing right now, but hopefully this situation can improve in future. Rich.

2 9

[PATCH common 0/4] options: Support Windows BitLocker (RHBZ#1808977).
by Richard W.M. Jones 27 Jul '20

27 Jul '20

Support transparent decryption/inspection of Windows guests encrypted with BitLocker encryption. This won't make much sense without the associated libguestfs patches which I will post momentarily. (Submodules, ho hum) Rich.

2 6