June 2020 - Libguestfs - libguestfs List Archives

[PATCH] daemon: inspect_fs_windows: Handle parted errors
by Sam Eiderman 22 Sep '20

22 Sep '20

By creating an empty disk and using it as the first disk of the vm (i.e. /dev/sda, /dev/sdb{1,2} contains the windows fses) we change the iteration order of the disks. This causes inspect_os() to fail since Parted returns a Unix_error if the device does not contain any partitions - fix this by handling this Unix_error. Signed-off-by: Sam Eiderman <sameid(a)google.com> --- daemon/inspect_fs_windows.ml | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/daemon/inspect_fs_windows.ml b/daemon/inspect_fs_windows.ml index c4a05bc38..bc6b98b60 100644 --- a/daemon/inspect_fs_windows.ml +++ b/daemon/inspect_fs_windows.ml @@ -365,8 +365,10 @@ and map_registry_disk_blob_mbr devices blob = let device = List.find ( fun dev -> - Parted.part_get_parttype dev = "msdos" && - pread dev 4 0x01b8 = diskid + try + Parted.part_get_parttype dev = "msdos" && + pread dev 4 0x01b8 = diskid + with Unix.Unix_error (Unix.EINVAL, _, _) -> false ) devices in (* Next 8 bytes are the offset of the partition in bytes(!) given as @@ -402,7 +404,10 @@ and map_registry_disk_blob_gpt partitions blob = fun part -> let partnum = Devsparts.part_to_partnum part in let device = Devsparts.part_to_dev part in - let typ = Parted.part_get_parttype device in + let typ = + try + Parted.part_get_parttype device + with Unix.Unix_error (Unix.EINVAL, _, _) -> "unknown" in if typ <> "gpt" then false else ( let guid = Parted.part_get_gpt_guid device partnum in -- 2.27.0.212.ge8ba1cc988-goog

2 3

[PATCH 0/7] Support Windows BitLocker (RHBZ#1808977).
by Richard W.M. Jones 27 Jul '20

27 Jul '20

These commits, along with the associated changes to common: https://www.redhat.com/archives/libguestfs/2020-March/msg00286.html support the transparent decryption and inspection of Windows guests encrypted with BitLocker encryption. To do the BitLocker decryption requires cryptsetup 2.3.0 (although cryptsetup 2.3 is not required for existing LUKS use). It also requires a new-ish Linux kernel, I believe 5.3+ This starts by adding new APIs cryptsetup-open and cryptsetup-close. As well as dropping the "luks-" prefix, this allows specifying an optional type parameter, allowing you to select BitLocker encryption. Although also the new API does not require this parameter, and will autodetect the correct type of decryption to apply in every known case. The main work is then in updating list-filesystems and ensuring that inspection still works. This really needs but also lacks tests. See commit message of patch 1/7 for why. I believe it's impossible for us to prove unit testing right now, but hopefully this situation can improve in future. Rich.

2 9

[PATCH common 0/4] options: Support Windows BitLocker (RHBZ#1808977).
by Richard W.M. Jones 27 Jul '20

27 Jul '20

Support transparent decryption/inspection of Windows guests encrypted with BitLocker encryption. This won't make much sense without the associated libguestfs patches which I will post momentarily. (Submodules, ho hum) Rich.

2 6

[PATCH nbdkit 0/2] tar: Rewrite the tar plugin (again), this time in C.
by Richard W.M. Jones 06 Jul '20

06 Jul '20

For context see these threads: https://lists.gnu.org/archive/html/qemu-discuss/2020-06/threads.html#00053 https://lists.gnu.org/archive/html/qemu-block/2020-06/threads.html#01496 Rich.

3 10

[PATCH v3] python: Fix UnicodeError in inspect_list_applications2() (RHBZ#1684004)
by Sam Eiderman 06 Jul '20

06 Jul '20

The python3 bindings create PyUnicode objects from application strings on the guest (i.e. installed rpm, deb packages). It is documented that rpm package fields such as description should be utf8 encoded - however in some cases they are not a valid unicode string, on SLES11 SP4 the encoding of the description of the following packages is latin1 and they fail to be converted to unicode using guestfs_int_py_fromstring() (which invokes PyUnicode_FromString()): PackageKit aaa_base coreutils dejavu desktop-data-SLED gnome-utils hunspell hunspell-32bit hunspell-tools libblocxx6 libexif libgphoto2 libgtksourceview-2_0-0 libmpfr1 libopensc2 libopensc2-32bit liborc-0_4-0 libpackagekit-glib10 libpixman-1-0 libpixman-1-0-32bit libpoppler-glib4 libpoppler5 libsensors3 libtelepathy-glib0 m4 opensc opensc-32bit permissions pinentry poppler-tools python-gtksourceview splashy syslog-ng tar tightvnc xorg-x11 xorg-x11-xauth yast2-mouse Fix this by globally changing guestfs_int_py_fromstring() and guestfs_int_py_fromstringsize() to fallback to latin1 decoding if utf-8 decoding fails. Using the "strict" error handler doesn't matter in the case of latin1 and has the same effect of "replace": https://docs.python.org/3/library/codecs.html#error-handlers Signed-off-by: Sam Eiderman <sameid(a)google.com> --- python/handle.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/python/handle.c b/python/handle.c index 2fb8c18f0..fe89dc58a 100644 --- a/python/handle.c +++ b/python/handle.c @@ -387,7 +387,7 @@ guestfs_int_py_fromstring (const char *str) #if PY_MAJOR_VERSION < 3 return PyString_FromString (str); #else - return PyUnicode_FromString (str); + return guestfs_int_py_fromstringsize (str, strlen (str)); #endif } @@ -397,7 +397,12 @@ guestfs_int_py_fromstringsize (const char *str, size_t size) #if PY_MAJOR_VERSION < 3 return PyString_FromStringAndSize (str, size); #else - return PyUnicode_FromStringAndSize (str, size); + PyObject *s = PyUnicode_FromString (str); + if (s == NULL) { + PyErr_Clear (); + s = PyUnicode_Decode (str, strlen(str), "latin1", "strict"); + } + return s; #endif } -- 2.26.2.303.gf8c07b1a785-goog

3 11

[PATCH] lib: Increase default memsize to 1280 (RHBZ#1837765).
by Richard W.M. Jones 30 Jun '20

30 Jun '20

Argon2 is the default LUKS Password-Based Key Derivation Function (PBKDF) for some new guests such as RHEL 8.2 and Fedora. It is designed to be "memory hard", meaning that by design it requires large amounts of memory, making it expensive to brute-force. Unfortunately the default for guests which had more than a few GB of RAM at install time is to require about 1 GB of RAM to decrypt the block device, which is considerably larger than the default available in the libguestfs appliance. To make it possible to open these encrypted disks we need to make the appliance larger. This could be done as a one-off, and the current workaround is simply to set LIBGUESTFS_MEMSIZE=2048 or a similar amount. However since we don't know in advance whether we could be dealing with an encrypted disk, partition, etc. or what PBKDF it uses, the only way to deal with this in all circumstances is to increase the default memsize. This commit increases it quite a lot (768 -> 1280) which is unfortunate. Note as there is some confusion on this point: Since libguestfs does not attempt to decrypt disks in parallel, you only need ~ 1GB in total, not per encrypted disk. For a reproducer, see: https://bugzilla.redhat.com/show_bug.cgi?id=1837765#c14 --- lib/guestfs-internal.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h index ce6eb68e8..d7ec7215d 100644 --- a/lib/guestfs-internal.h +++ b/lib/guestfs-internal.h @@ -94,7 +94,7 @@ * creating device nodes. */ #ifdef __powerpc__ -# define DEFAULT_MEMSIZE 1024 +# define DEFAULT_MEMSIZE 1280 #endif /* Kernel 3.19 is unable to uncompress the initramfs on aarch64 unless @@ -103,12 +103,12 @@ * common on aarch64, treat this like the ppc case above. */ #ifdef __aarch64__ -# define DEFAULT_MEMSIZE 1024 +# define DEFAULT_MEMSIZE 1280 #endif /* The default and minimum memory size for most users. */ #ifndef DEFAULT_MEMSIZE -# define DEFAULT_MEMSIZE 768 +# define DEFAULT_MEMSIZE 1280 #endif #ifndef MIN_MEMSIZE # define MIN_MEMSIZE 256 -- 2.26.2

2 1

[PATCH nbdkit 0/5 NOT WORKING] nbd: Implement command= and socket-fd= parameters.
by Richard W.M. Jones 30 Jun '20

30 Jun '20

The first four patches are fairly routine clean up and can be reviewed/applied on their own. The fifth patch is problematic as described below. At the moment if you want to proxy through to qemu-nbd (eg. for handling qcow2 files) it's rather complicated and you end up having to manage the sockets and clean up yourself. However the library we use for the proxying supports a perfectly good way to run qemu-nbd as a subprocess: http://libguestfs.org/nbd_connect_systemd_socket_activation.3.html#EXAMPLE Why not use that instead? Then you'd be able to write: nbdkit nbd command=' qemu-nbd -f qcow2 disk.qcow2 ' and place filters in front. nbdkit would deal with clean up of the qemu-nbd process. However I've been staring at this most of the day and for reasons I don't understand the nbd plugin deadlocks. Specifically it waits forever on the semaphore in nbdplug_reply. A second problem with this patch (maybe related to the first??) is that the command is specified as a string, but the libnbd API wants an argv[]. There's no really good way to specify an argv as parameters of an nbdkit plugin. You'd end up having to write something pretty clunky such as one of these: nbdkit nbd command='qemu-nbd,-f,qcow2,disk.qcow2' nbdkit nbd command=qemu-nbd command=-f command=qcow2 command=disk.qcow2 Rich.

1 5

virt-resize - guest unbootable after update-initramfs
by Stewart Middleton 26 Jun '20

26 Jun '20

Hi, I am having a hard time rationalizing why virt-resize would be the cause of this issue, however it appears to definitely be the trigger, hence trying this list. My environment comprises Debian 10 hosts and guests, although this issue has also been witnessed on Debian 9 hosts running Debian9/10 guests. In all cases, stable versions of libvirt, qemu-kvm, libguestfs- tools, etc running on AMD64 arch. The issue is that for guests whose disks have been populated by virt- resize, when they first run `update-initramfs` (e.g. after a kernel update), the system will become unbootable. In this state it fails to the initramfs debug shell where it can be seen that none of the IO modules have loaded, and so it is unable to mount root: "mount: mounting /dev/vda1 on /root failed: No such device" My workflow is as follows: * Prereq: I have a 'base' guest, backed by a small LVM volume containing / & swap (xfs, but also tried ext4). This is a vanilla Debian 10 install, with some very minor mods (a couple of packages added, a user created, IPV6 disabled) * For a new guest, I then have a script: * Creates an LVM snapshot of the base guest volume * `kpartx -a /path/to/snapshot` to access the partitions * mount / and then small edits to /etc/hostname and /etc/network/interfaces to configure networking * `kpartx -d /path/to/snapshot` * Create a new LVM volume at the size required for the final guest * `virt-resize` to copy the partitions from the snapshot to the new volume, resizing swap to a fixed size and then growing / to fill the remainder * Generate an XML config file to define the new host, based on the XML used to create the base host (making the hosts identical, other than their unqiue differences, UUID, MAC, disks, etc) At this point the guest will boot OK, and continue to run without issue, including any number of reboots, until the first time update- initramfs is run. At this point the guest fails to boot. By contrast, similar guests that have disks that have not been mananged by `virt-resize` do not experience this issue. I have performed a wide array of swap tests between source volumes, snapshots, volumes created by a `dd` copy and `virt-resize` each connected to different hosts, including the original base host and those derived from it. The only common thread so far, is the problem occurs with disks populated by `virt-resize`. Booting a guest pre/post having update-initramfs run and passing "break=premount" to the kernel to gain access to the initramfs debug shell and then checking the output of `dmesg`, show identical output, right up until the end when you see the following differences: Host prior to update-initramfs: " [ 0.736291] Run /init as init process [ 0.795060] lpc_ich 0000:00:1f.0: I/O space for GPIO uninitialized [ 0.800259] PCI Interrupt Link [GSIA] enabled at IRQ 16 [ 0.800363] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt [ 0.811428] cryptd: max_cpu_qlen set to 1000 [ 0.821407] AVX2 version of gcm_enc/dec engaged. [ 0.821407] AES CTR mode by8 optimization enabled [ 0.823702] ACPI: bus type USB registered [ 0.823712] usbcore: registered new interface driver usbfs [ 0.823717] usbcore: registered new interface driver hub [ 0.823726] usbcore: registered new device driver usb [ 0.829766] SCSI subsystem initialized [ 0.830240] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.830430] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.844882] virtio_blk virtio2: [vda] 41943040 512-byte logical blocks (21.5 GB/20.0 GiB) [ 0.854231] vda: vda1 vda2 [ 0.857633] xhci_hcd 0000:02:00.0: xHCI Host Controller [ 0.857638] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 1 [ 0.857900] xhci_hcd 0000:02:00.0: hcc params 0x00087001 hci version 0x100 quirks 0x0000000000000010 [ 0.859088] virtio_net virtio0 enp1s0: renamed from eth0 [ 0.860055] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19 [ 0.860056] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.860056] usb usb1: Product: xHCI Host Controller [ 0.860057] usb usb1: Manufacturer: Linux 4.19.0-9-amd64 xhci-hcd [ 0.860057] usb usb1: SerialNumber: 0000:02:00.0 " Host after to update-initramfs: " [ 0.743940] Run /init as init process [ 0.801833] lpc_ich 0000:00:1f.0: I/O space for GPIO uninitialized [ 0.814263] cryptd: max_cpu_qlen set to 1000 [ 0.817594] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.817785] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.820087] PCI Interrupt Link [GSIA] enabled at IRQ 16 [ 0.820187] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt [ 0.829008] AVX2 version of gcm_enc/dec engaged. [ 0.829008] AES CTR mode by8 optimization enabled [ 0.839844] virtio_blk virtio2: [vda] 41943040 512-byte logical blocks (21.5 GB/20.0 GiB) [ 0.843734] vda: vda1 vda2 [ 0.846319] virtio_net virtio0 enp1s0: renamed from eth0 " (I have trimmed the output in the case of the host prior as it just goes on to load all the expected modules, however in the after case, that is the end of the dmesg output) Does anybody have any suggestions as to what might cause virt-resize to directly/indirectly trigger this condition? I can supply any necessary diag information or run any tests, but don't want to make this email even longer with redundant/distracting info. Many thanks in advance, Stewart

1 0

[PATCH libguestfs-common 1/2] mlcustomize: Refactor SELinux_relabel code.
by Richard W.M. Jones 24 Jun '20

24 Jun '20

This shouldn't change the effect of this code. --- mlcustomize/SELinux_relabel.ml | 121 ++++++++++++++++++--------------- 1 file changed, 65 insertions(+), 56 deletions(-) diff --git a/mlcustomize/SELinux_relabel.ml b/mlcustomize/SELinux_relabel.ml index 44995df..5df1f08 100644 --- a/mlcustomize/SELinux_relabel.ml +++ b/mlcustomize/SELinux_relabel.ml @@ -28,65 +28,74 @@ module G = Guestfs let array_find a l = List.mem a (Array.to_list l) -let relabel (g : G.guestfs) = - (* Is the guest using SELinux? *) - if g#is_file ~followsymlinks:true "/usr/sbin/load_policy" && - g#is_file ~followsymlinks:true "/etc/selinux/config" then ( - (* Is setfiles / SELinux relabelling functionality available? *) - if g#feature_available [| "selinuxrelabel" |] then ( - (* Use Augeas to parse /etc/selinux/config. *) - g#aug_init "/" (16+32) (* AUG_SAVE_NOOP | AUG_NO_LOAD *); - (* See: https://bugzilla.redhat.com/show_bug.cgi?id=975412#c0 *) - ignore (g#aug_rm "/augeas/load/*[\"/etc/selinux/config/\" !~ regexp('^') + glob(incl) + regexp('/.*')]"); - g#aug_load (); - debug_augeas_errors g; - - (* Get the SELinux policy name, eg. "targeted", "minimum". - * Use "targeted" if not specified, just like libselinux does. +let rec relabel (g : G.guestfs) = + (* Is the guest using SELinux? (Otherwise this is a no-op). *) + if is_selinux_guest g then ( + try + use_setfiles g; + (* That worked, so we don't need to autorelabel. *) + g#rm_f "/.autorelabel" + with Failure _ -> + (* This is the fallback in case something in the setfiles + * method didn't work. That includes the case where a non-SELinux + * host is processing an SELinux guest, and other things. *) - let policy = - let config_path = "/files/etc/selinux/config" in - let selinuxtype_path = config_path ^ "/SELINUXTYPE" in - let keys = g#aug_ls config_path in - if array_find selinuxtype_path keys then - g#aug_get selinuxtype_path - else - "targeted" in + g#touch "/.autorelabel" + ) - g#aug_close (); +and is_selinux_guest g = + g#is_file ~followsymlinks:true "/usr/sbin/load_policy" && + g#is_file ~followsymlinks:true "/etc/selinux/config" - (* Get the spec file name. *) - let specfile = - sprintf "/etc/selinux/%s/contexts/files/file_contexts" policy in +and use_setfiles g = + (* Is setfiles / SELinux relabelling functionality available? *) + if not (g#feature_available [| "selinuxrelabel" |]) then + failwith "no selinux relabel feature"; - (* RHEL 6.2 - 6.5 had a malformed specfile that contained the - * invalid regular expression "/var/run/spice-vdagentd.\pid" - * (instead of "\.p"). This stops setfiles from working on - * the guest. - * - * Because an SELinux relabel writes all over the filesystem, - * it seems reasonable to fix this problem in the specfile - * at the same time. (RHBZ#1374232) - *) - if g#grep ~fixed:true "vdagentd.\\pid" specfile <> [||] then ( - debug "fixing invalid regular expression in %s" specfile; - let old_specfile = specfile ^ "~" in - g#mv specfile old_specfile; - let content = g#read_file old_specfile in - let content = - String.replace content "vdagentd.\\pid" "vdagentd\\.pid" in - g#write specfile content; - g#copy_attributes ~all:true old_specfile specfile - ); + (* Use Augeas to parse /etc/selinux/config. *) + g#aug_init "/" (16+32) (* AUG_SAVE_NOOP | AUG_NO_LOAD *); + (* See: https://bugzilla.redhat.com/show_bug.cgi?id=975412#c0 *) + ignore (g#aug_rm "/augeas/load/*[\"/etc/selinux/config/\" !~ regexp('^') + glob(incl) + regexp('/.*')]"); + g#aug_load (); + debug_augeas_errors g; - (* Relabel everything. *) - g#selinux_relabel ~force:true specfile "/"; + (* Get the SELinux policy name, eg. "targeted", "minimum". + * Use "targeted" if not specified, just like libselinux does. + *) + let policy = + let config_path = "/files/etc/selinux/config" in + let selinuxtype_path = config_path ^ "/SELINUXTYPE" in + let keys = g#aug_ls config_path in + if array_find selinuxtype_path keys then + g#aug_get selinuxtype_path + else + "targeted" in - (* If that worked, we don't need to autorelabel. *) - g#rm_f "/.autorelabel" - ) - else ( - (* SELinux guest, but not SELinux host. Fallback to this. *) - g#touch "/.autorelabel" - ) - ) + g#aug_close (); + + (* Get the spec file name. *) + let specfile = + sprintf "/etc/selinux/%s/contexts/files/file_contexts" policy in + + (* RHEL 6.2 - 6.5 had a malformed specfile that contained the + * invalid regular expression "/var/run/spice-vdagentd.\pid" + * (instead of "\.p"). This stops setfiles from working on + * the guest. + * + * Because an SELinux relabel writes all over the filesystem, + * it seems reasonable to fix this problem in the specfile + * at the same time. (RHBZ#1374232) + *) + if g#grep ~fixed:true "vdagentd.\\pid" specfile <> [||] then ( + debug "fixing invalid regular expression in %s" specfile; + let old_specfile = specfile ^ "~" in + g#mv specfile old_specfile; + let content = g#read_file old_specfile in + let content = + String.replace content "vdagentd.\\pid" "vdagentd\\.pid" in + g#write specfile content; + g#copy_attributes ~all:true old_specfile specfile + ); + + (* Relabel everything. *) + g#selinux_relabel ~force:true specfile "/" -- 2.24.1

2 5

Fwd: 回复： virt-builder problem
by Richard W.M. Jones 24 Jun '20

24 Jun '20

----- Forwarded message from 雁渡寒潭 <chnlcq(a)qq.com> ----- Date: Wed, 24 Jun 2020 11:38:41 +0800 From: 雁渡寒潭 <chnlcq(a)qq.com> To: rjones <rjones(a)redhat.com> Subject: 回复： [Libguestfs] 回复： virt-builder problem thank you vrey much! problem is Resolved！ http://archive.libguestfs.org/builder/index.asc   is blocked! I'M disables it! But i can  accesss  http://builder.libguestfs.org/index.asc    they is one domain libguesfs.org! I'm Replay process； I didn't pay attention to the repo! according to CURL error,  i'm know is network problem! but virt-builder -x -v  i don't know which website link is not access! ------------------ 原始邮件 ------------------ 发件人: "rjones"<rjones(a)redhat.com>; 发送时间: 2020年6月24日(星期三) 凌晨0:51 收件人: "雁渡寒潭"<chnlcq(a)qq.com>; 抄送: "rjones"<libguestfs(a)redhat.com>; 主题: Re: [Libguestfs] 回复： virt-builder problem On Tue, Jun 23, 2020 at 11:58:44PM +0800, 雁渡寒潭 wrote: > Please check the attachment. Thanks for collecting that. As far as I can see the error is coming from one of the downloads. These is the full list of downloads that virt-builder makes in the default configuration: http://download.opensuse.org/repositories/Virtualization:/virt-builder-imag… http://builder.libguestfs.org/index.asc http://archive.libguestfs.org/builder/index.asc I wonder if the problem is with archive.libguestfs.org?  Can you access all of those URLs? Also you can disable particular archives by editing the files in /etc/virt-builder/repos.d/*.conf.  Just comment out the sections that you don't want to download to see if that improves things. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines.  Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ----- End forwarded message ----- -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

1 0