January 2023 - Libguestfs - libguestfs List Archives

[PATCH v2v] -o rhv-upload: Give a nicer error if the storage domain
by Richard W.M. Jones 27 Jan '23

27 Jan '23

https://bugzilla.redhat.com/show_bug.cgi?id=1986386 My RHV instance is dead at the moment so I didn't do much more than check this compiles and passes the one test we have. Also I want to spend as little time as possible on RHV outputs for virt-v2v since the RHV product will be discontinued soon. I did want to point out some things: - The preceeding code is probably wrong. https://github.com/libguestfs/virt-v2v/blob/master/output/rhv-upload-transf… It attempts to search for the output storage using: storage_domains = system_service.storage_domains_service().list( search='name=%s' % params['output_storage'], case_sensitive=True, ) I couldn't find any documentation about what can go into that search string, but it's clearly a lot more complicated than just pasting in the literal name after "name=". At the very least, spaces are not permitted, see: https://github.com/libguestfs/virt-v2v/blob/master/output/rhv-upload-transf… - The bug reporter used "data*" as the name and I suspect that is parsed in some way (wildcard? regexp? I've no idea). - Probably for the same reason, the preceeding code ought to fail with an error if the output storage domain doesn't exist. The fact we reach the code patched here at all also indicates some bug, maybe in the search string. As I say above, I don't especially care about any of this. Rich.

2 4

libnbd | Failed pipeline for master | dd4ec1b8
by GitLab 21 Jan '23

21 Jan '23

Pipeline #754009111 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: dd4ec1b8 ( https://gitlab.com/nbdkit/libnbd/-/commit/dd4ec1b8c4c85fd587a6bbe853b93d17e… ) Commit Message: Version 1.15.9. Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #754009111 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/754009111 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #3640476499 ( https://gitlab.com/nbdkit/libnbd/-/jobs/3640476499/raw ) Stage: builds Name: x86_64-macos-11 -- You're receiving this email because of your account on gitlab.com.

1 0

FYI: libguestfs development version now depends on ocaml-augeas
by Richard W.M. Jones 21 Jan '23

21 Jan '23

Hi Hillu, Stephane, This bug ... https://bugzilla.redhat.com/show_bug.cgi?id=2162845 ... possibly - I'm not sure if this person is using Debian, but it'll affect Debian too. Libguestfs used to bundle a copy of the OCaml bindings for Augeas: http://augeas.net/ https://people.redhat.com/rjones/augeas/ I have removed the bundled copy in the development version: https://github.com/libguestfs/libguestfs/commit/fbf7fe87933ac24dd78362a1611… so they'll have to be packaged separately now (or rebundled, but I guess you won't want that). The ocaml-augeas package is very stable. It hasn't changed since around 2019. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v

1 0

[p2v PATCH 0/4] RHEL9 build host tweaks
by Laszlo Ersek 20 Jan '23

20 Jan '23

In preparation for some GUI investigation / work, I've run "make run-virt-p2v-in-a-vm" for the first time on my new RHEL-9.1 laptop. Some tweaks appear to be necessary (or at least useful). Laszlo Laszlo Ersek (4): ssh: shield virt-v2v from bash RC files make-disk: restrict kernel update bug work-around make-disk: replace here-docs with printfs for better indentation make-disk: inject EPEL9 repo in RHEL9 disk image ssh.c | 2 +- test-virt-p2v-ssh.sh | 2 +- virt-p2v-make-disk.in | 57 +++++++++++++++++++++++++++++++------------ 3 files changed, 44 insertions(+), 17 deletions(-)

2 9

[PATCH nbdkit] New plugin: blkio [incomplete]
by Richard W.M. Jones 17 Jan '23

17 Jan '23

This is an incomplete outline implementation for a libblkio plugin for nbdkit. At the moment it only supports reading the same ("capacity") of the device, and not even reading or writing. I have some questions about the libblkio API before I can complete the plugin (see below). The idea here is that by connecting libblkio to NBD we can use the existing set of scripting tools to script access to devices. For example you could use Python to read or modify a device: ---------------------------------------------------------------------- $ nbdsh -c - <<'EOF' from subprocess import * # Run nbdkit-blkio-plugin. h.connect_systemd_socket_activation ("nbdkit" "blkio", "virtio-blk-vhost-user", "path=unix.sock") print("device size=%d", h.get_size()) # Dump the boot sector. bootsect = h.pread(512, 0) p = Popen("hexdump -C", shell=True, stdin=PIPE) p.stdin.write(bootsect) EOF ---------------------------------------------------------------------- So my questions and comments about libblkio: (1) There is no way to know which properties are readable, writable, and those which need to be set before or after blkio_connect (see is_preconnect_property in the plugin). It should be possible to introspect this information. Also might be nice to be able read a list of all available properties. (2) It would be nice if libblkio had a way to enable debugging and call the caller back for debug messages. We could then redirect the callbacks into the nbdkit debug API (nbdkit_debug()) where they would eventually end up on stderr or syslog. However don't send debug messages to stderr, or at least allow that behaviour to be overridden. (3) It seems like some drivers require pre-allocated memory regions, and since some do that means we might as well implement this. It also seems like some drivers require file-backed pre-allocated memory regions, and so we might as well implement that too. However what is not clear: does memfd_create(2) always allocate sufficiently aligned memory regions, such that we never need to bother reading the mem-region-alignment property? I notice that the example: https://gitlab.com/libblkio/libblkio/-/blob/main/examples/blkio-copy.c just passes on this and calls blkio_alloc_mem_region(). Is that the safest and easiest thing to do which will always work? (4) As a first pass, I only want to bother implementing blocking mode. It'll be slow, but it doesn't matter for this application. Also I've chosen nbdkit's NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS so nbdkit will serialise all requests (again, only for a very simple first pass). Looking at: https://libblkio.gitlab.io/libblkio/blkio.html#performing-i-o seems simple enough but: (4a) What is the queue number? Always 0? Is it affected by num-entries? (4b) It's unclear how completions work. If I set min=max=1, will it return after the whole operation has completed? Do I need to call it again? What about if the request is very large, can it get split? Reading the example https://gitlab.com/libblkio/libblkio/-/blob/main/examples/blkio-copy.c it appears that requests cannot ever be split? Rich.

4 20

[nbdkit PATCH] data: don't ignore SIGPIPE
by Thomas Weißschuh 16 Jan '23

16 Jan '23

If all of the requested data has been read, the call to popen_close will close the scripts output pipe and on the next write a SIGPIPE is delivered. When the scripts inherited signalhandler is ignoring the singal, it won't be aborted by default. This happens in the test suite, as make 4.4 seems to ignore this signal. Therefore the executed scripts never stop and the testsuite never finishes. --- plugins/data/format.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/plugins/data/format.c b/plugins/data/format.c index 3667ead6790f..2319ea09475c 100644 --- a/plugins/data/format.c +++ b/plugins/data/format.c @@ -40,6 +40,7 @@ #include <stdarg.h> #include <string.h> #include <assert.h> +#include <signal.h> #define NBDKIT_API_VERSION 2 #include <nbdkit-plugin.h> @@ -1951,6 +1952,9 @@ store_script_len (struct allocator *a, char buf[BUFSIZ]; size_t n; + /* Restore SIGPIPE back to SIG_DFL, since shell can't undo SIG_IGN */ + signal (SIGPIPE, SIG_DFL); + pp = popen (script, "r"); if (pp == NULL) { nbdkit_error ("popen: %m"); base-commit: 09a61a0a00d3036ebfbc2fbadb0ba3e10a778ff3 -- 2.39.0

3 3

Cargo edition problem with Debian 11
by Richard W.M. Jones 16 Jan '23

16 Jan '23

https://gitlab.com/nbdkit/nbdkit/-/jobs/3598390121 ---------------------------------------------------------------------- cargo build --release --example ramdisk Downloading crates ... Downloaded float-cmp v0.9.0 Downloaded downcast v0.11.0 Downloaded mockall_derive v0.11.3 Downloaded itertools v0.10.5 Downloaded aho-corasick v0.7.20 Downloaded predicates-tree v1.0.7 error: failed to parse manifest at `/root/.cargo/registry/src/github.com-1ecc6299db9ec823/predicates-tree-1.0.7/Cargo.toml` Caused by: failed to parse the `edition` key Caused by: this version of Cargo is older than the `2021` edition, and only supports `2015` and `2018` editions. ---------------------------------------------------------------------- nbdkit itself uses edition = 2018, but this seems to affects one of the dependencies. I'm not sure how to solve this, but one ideas I had is in ./configure to check if the cargo/rust we're trying to use doesn't support some base edition (eg. latest edition supported < 2021) then we would disable rust bindings. Unfortunately actually getting the latest supported edition seems hard. The best I could find is parsing this which doesn't seem ideal: $ rustc --help |& grep -- --edition --edition 2015|2018|2021|2024 What do you think? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 21

[v2v PATCH 0/2] tests: spell out btrfs and ntfs feature group dependencies
by Laszlo Ersek 13 Jan '23

13 Jan '23

A few test cases fail if the libguestfs appliance lacks the btrfs or the ntfs feature group. Let us skip these tests in those situations instead. Laszlo Ersek (2): test-v2v-fedora-btrfs-conversion: spell out btrfs feature group dependency test-v2v-i-ova: spell out ntfs feature group dependency tests/test-v2v-fedora-btrfs-conversion.sh | 1 + tests/test-v2v-i-ova-directory.sh | 4 +++- tests/test-v2v-i-ova.sh | 4 +++- 3 files changed, 7 insertions(+), 2 deletions(-)

2 4

Re: [Libguestfs] [PATCH nbdkit] New plugin: blkio [incomplete]
by Alberto Faria 09 Jan '23

09 Jan '23

> This is upstream in nbdkit now: > https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/blkio > > Another question: > > (6) vhost-user + "read-only" property acts a bit strangely. After > opening virtio-blk-vhost-user it throws an EROFS error if you try to > "blkio_start" it. However if you set the "read-only" property to true > then it's OK to start it, and subsequent read operations work too. > > I would expect that any device can be started without needing to set > this property, and in general I don't understand why vhost-user is r/o > since everything I read about it seems to indicate it's a general > purpose writable protocol. read-only has POSIX file open-like semantics: if the device is itself read only but the read-only property is false, blkio_start() fails. With qemu-storage-daemon, you must set writable=on on the vhost-user-blk export for the device to be writable. Alberto

2 1

[PATCH] inspect: check presence of BIOS boot partition to handle BIOS+GPT setup
by Andrey Drobyshev 09 Jan '23

09 Jan '23

If the guest uses BIOS firmware but GPT partitioned disk, and in addition has "bios, esp" flag enabled on its "/boot" partition, then inspection wrongly detects UEFI firmware and v2v ends up with the error: "error: no bootloader detected". Let's fix this by checking for the presence of BIOS boot partition (0xef02 type for gdisk, "bios_grub" flag for parted), which is used to store a bootloader code in BIOS+GPT configurations. If such a partition is present, then it's likely a BIOS+GPT setup. Signed-off-by: Andrey Drobyshev <andrey.drobyshev(a)virtuozzo.com> --- convert/inspect_source.ml | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/convert/inspect_source.ml b/convert/inspect_source.ml index 056d0bca..6650e136 100644 --- a/convert/inspect_source.ml +++ b/convert/inspect_source.ml @@ -219,6 +219,9 @@ and list_applications g root = function (* See if this guest could use UEFI to boot. It should use GPT and * it should have an EFI System Partition (ESP). * + * If it has a BIOS boot partition (BIOS+GPT setup), then [BIOS] is + * returned. + * * If it has ESP(s), then [UEFI devs] is returned where [devs] is the * list of at least one ESP. * @@ -226,9 +229,13 @@ and list_applications g root = function *) and get_firmware_bootable_device g = let rec uefi_ESP_guid = "C12A7328-F81F-11D2-BA4B-00A0C93EC93B" + and bios_boot_guid = "21686148-6449-6E6F-744E-656564454649" and is_uefi_ESP dev part = let partnum = g#part_to_partnum part in g#part_get_gpt_type dev partnum = uefi_ESP_guid + and is_bios_boot dev part = + let partnum = g#part_to_partnum part in + g#part_get_gpt_type dev partnum = bios_boot_guid and parttype_is_gpt dev = try g#part_get_parttype dev = "gpt" with G.Error msg as exn -> @@ -239,14 +246,26 @@ and get_firmware_bootable_device g = and is_uefi_bootable_part part = let dev = g#part_to_dev part in parttype_is_gpt dev && is_uefi_ESP dev part + and is_bios_gpt_part part = + let dev = g#part_to_dev part in + parttype_is_gpt dev && is_bios_boot dev part in let partitions = Array.to_list (g#list_partitions ()) in - let partitions = List.filter is_uefi_bootable_part partitions in + let esp_partitions = List.filter is_uefi_bootable_part partitions in - match partitions with - | [] -> I_BIOS - | partitions -> I_UEFI partitions + (* If there's a BIOS boot partition present (0xef02 type for gdisk, + * "bios_grub" flag for parted), then this is likely a BIOS+GPT setup. + * Note that if a source VM is using UEFI firmware and has a secondary + * non-bootable disk attached which contains such a partition, the + * firmware detection will detect I_BIOS wrongly. But this can only be + * done manually, and we assume that there's no point doing it on purpose. + *) + if List.exists is_bios_gpt_part partitions then I_BIOS + else + match esp_partitions with + | [] -> I_BIOS + | esp_partitions -> I_UEFI esp_partitions (* If some inspection fields are "unknown", then that indicates a * failure in inspection, and we shouldn't continue. For an example -- 2.31.1

3 11