December 2021 - Libguestfs - libguestfs List Archives

[v2v PATCH] test-v2v-o-libvirt: undefine temporary guest when exiting
by Laszlo Ersek 17 Dec '21

17 Dec '21

I've noticed that my libvirtd host that I use for v2v development is littered with tmp-* guests. Re-enable the "virsh undefine" command. Signed-off-by: Laszlo Ersek <lersek(a)redhat.com> --- tests/test-v2v-o-libvirt.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/test-v2v-o-libvirt.sh b/tests/test-v2v-o-libvirt.sh index 7396b2ad69b2..d44f55dd6feb 100755 --- a/tests/test-v2v-o-libvirt.sh +++ b/tests/test-v2v-o-libvirt.sh @@ -55,8 +55,7 @@ rm -rf $d # Clean up. cleanup_fn rm -r $d cleanup_fn virsh pool-destroy $poolname -# I think because the pool is transient, this is not necessary: -#cleanup_fn virsh undefine $guestname +cleanup_fn virsh undefine $guestname mkdir $d -- 2.19.1.3.g30247aa5d201

2 2

[PATCH nbdkit BROKEN 0/2] vddk: Support multi-conn, but only open one VDDK connection
by Richard W.M. Jones 16 Dec '21

16 Dec '21

This is broken as noted in the first commit message. I'm only posting it to document and archive the approach. We would like to enable multi-conn in the VDDK plugin for its potential performance benefits, and also to avoid the hack that we had to add to virt-v2v: https://github.com/libguestfs/virt-v2v/commit/bb0e698360470cb4ff5992e8e01a3… One way I thought of doing this would be to advertise multi-conn, but internally in the plugin only ever open a single VDDK connection. Multiple calls to .open would open the VDDK connection on the first call and increment a counter in other cases. Multiple calls to .close would close the connection only on the last close. However it doesn't actually work. For some reason our old enemy QueryAllocatedBlocks (QAB) suddenly becomes many times (about 18-20 times) slower when you do this. Of course it's closed source so we can never work out why, but my speculation is that QAB has a mutex that prevents it from overlapping with any other call, and that it only works well when called linearly across the disk. When nbdcopy sees multi-conn it will launch 4 threads reading in parallel across different offsets which would break both of these assumptions. I observed similar huge slowdowns (7x) in QAB when opening multiple read-only connections to the same disk using the current code. Anyway, for whatever reason it doesn't work, so this is just for archiving. Rich.

1 2

[PATCH] convert_linux: translate the first CD-ROM's references in boot conf files
by Laszlo Ersek 16 Dec '21

16 Dec '21

If the only CD-ROM in "s_removables" is on an IDE controller, and the guest kernel represents it with a /dev/hdX device node, then convert references to this device node, in the boot config files, to /dev/cdrom. On the destination (after conversion), /dev/cdrom will point to whataver node we converted the CD-ROM to, masking a potential i440fx -> q35 (IDE -> SATA) board change. If the only CD-ROM is not on an IDE controller, or the guest is modern enough to represent the IDE CD-ROM as /dev/sr0, then perform no translation. Namely, /dev/sr0 survives a potential i440fx -> q35 (IDE -> SATA) board change intact. When multiple CD-ROMs exist, emit a warning, and attempt the conversion on the first CD-ROM, as a guess. This may be inexact, but we can't do better, because: - SATA, SCSI, and (on modern guests) IDE CD-ROMs are lumped together in the /dev/sr* namespace, on the source side, and "s_removable_slot" is useless for telling them apart, as we don't know the exact controller topology (and OS enumeration order); - after conversion: some OSes don't create /dev/cdrom* symlinks to all CD-ROMs, and even if multiple such symlinks are created, their order is basically random. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1637857 Signed-off-by: Laszlo Ersek <lersek(a)redhat.com> --- convert/convert_linux.ml | 25 +++++++++++++++++++++++++ tests/test-v2v-cdrom.expected | 2 +- tests/test-v2v-cdrom.xml.in | 4 +++- tests/test-v2v-i-ova.xml | 2 +- 4 files changed, 30 insertions(+), 3 deletions(-) diff --git a/convert/convert_linux.ml b/convert/convert_linux.ml index 8dc648169dcb..9ea94b75bfc9 100644 --- a/convert/convert_linux.ml +++ b/convert/convert_linux.ml @@ -1020,6 +1020,31 @@ let convert (g : G.guestfs) source inspect keep_serial_console _ = "xvd" ^ drive_name i, block_prefix_after_conversion ^ drive_name i ) source.s_disks in + (* Check the first CD-ROM. If its controller is IDE, and the OS is RHEL<=5, + * then translate the CD-ROM from "/dev/hd[SLOT]" to "/dev/cdrom". See + * RHBZ#1637857 for details. + *) + let cdroms = List.filter + (fun removable -> removable.s_removable_type = CDROM) + source.s_removables in + + match cdroms with + | _ :: _ :: _ -> warning (f_"multiple CD-ROMs found; translation of \ + CD-ROM references may be inexact") + | _ -> (); + + let map = map @ + (match cdroms with + | cdrom :: _ -> + (match (cdrom.s_removable_controller, cdrom.s_removable_slot, + family, inspect.i_major_version) with + | Some Source_IDE, Some slot, `RHEL_family, v when v <= 5 -> + [("hd" ^ drive_name slot, "cdrom")] + | _ -> [] + ) + | [] -> [] + ) in + if verbose () then ( eprintf "block device map:\n"; List.iter ( diff --git a/tests/test-v2v-cdrom.expected b/tests/test-v2v-cdrom.expected index 34d2bf5961b0..17bd152d8e64 100644 --- a/tests/test-v2v-cdrom.expected +++ b/tests/test-v2v-cdrom.expected @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> diff --git a/tests/test-v2v-cdrom.xml.in b/tests/test-v2v-cdrom.xml.in index 6bad5eab1cd4..a6e1e3f514d5 100644 --- a/tests/test-v2v-cdrom.xml.in +++ b/tests/test-v2v-cdrom.xml.in @@ -35,7 +35,9 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='@abs_top_builddir@/test-data/phony-guests/blank-disk.img'/> -  +  <target dev='hdc' bus='ide'/> </disk> </devices> diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml index d7383905fdc0..9f3c1974243f 100644 --- a/tests/test-v2v-i-ova.xml +++ b/tests/test-v2v-i-ova.xml @@ -28,7 +28,7 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hda' bus='ide'/> + <target dev='sda' bus='sata'/> </disk> <disk device='floppy' type='file'> <driver name='qemu' type='raw'/> -- 2.19.1.3.g30247aa5d201

2 14

[PATCH libnbd 1/2] ocaml/examples: Fix extents example
by Richard W.M. Jones 13 Dec '21

13 Dec '21

Laszlo Ersek noted that the original example we had did not work properly if the size of the disk was larger than around 2G. The nbd_block_status API is really difficult to use correctly! In particular it is not guaranteed that the server will return extents covering the size requested. It's also not guaranteed that a bad server will return any base:allocation extents at all (although such a server would not be conforming - the protocol says that servers must always make forward progress). This commit attempts a fix, although it is not complete especially if the server is badly behaved. It also makes the output look a bit better by aligning the columns. Also we use nbdkit-sparse-random- plugin with a larger size to test the > 2G case. --- ocaml/examples/extents.ml | 53 ++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 26 deletions(-) diff --git a/ocaml/examples/extents.ml b/ocaml/examples/extents.ml index 6fa70e087..e4422b270 100644 --- a/ocaml/examples/extents.ml +++ b/ocaml/examples/extents.ml @@ -4,32 +4,33 @@ let () = let nbd = NBD.create () in NBD.add_meta_context nbd "base:allocation"; NBD.connect_command nbd - ["nbdkit"; "-s"; "--exit-with-parent"; "memory"; "size=128K"]; - - (* Write some sectors. *) - let data_sector = Bytes.make 512 'a' in - let zero_sector = Bytes.make 512 '\000' in - NBD.pwrite nbd data_sector 0_L; - NBD.pwrite nbd zero_sector 32768_L; - NBD.pwrite nbd data_sector 65536_L; + ["nbdkit"; "-s"; "--exit-with-parent"; "-r"; + "sparse-random"; "8G"]; (* Read the extents and print them. *) let size = NBD.get_size nbd in - NBD.block_status nbd size 0_L ( - fun meta _ entries err -> - printf "err=%d\n" !err; - if meta = "base:allocation" then ( - printf "index\tlength\tflags\n"; - for i = 0 to Array.length entries / 2 - 1 do - let flags = - match entries.(i*2+1) with - | 0_l -> "data" - | 1_l -> "hole" - | 2_l -> "zero" - | 3_l -> "hole+zero" - | i -> sprintf "unknown (%ld)" i in - printf "%d:\t%ld\t%s\n" i entries.(i*2) flags - done - ); - 0 - ) + let fetch_offset = ref 0_L in + while !fetch_offset < size do + let remaining = Int64.sub size !fetch_offset in + let fetch_size = min remaining 0x8000_0000_L in + NBD.block_status nbd fetch_size !fetch_offset ( + fun meta _ entries err -> + printf "nbd_block_status callback: meta=%s err=%d\n" meta !err; + if meta = "base:allocation" then ( + printf "index\t%-20s %-20s %s\n" "offset" "length" "flags"; + for i = 0 to Array.length entries / 2 - 1 do + let len = Int64.of_int32 entries.(i*2) + and flags = + match entries.(i*2+1) with + | 0_l -> "data" + | 1_l -> "hole" + | 2_l -> "zero" + | 3_l -> "hole+zero" + | i -> sprintf "unknown (%ld)" i in + printf "%d:\t%-20Ld %-20Ld %s\n" i !fetch_offset len flags; + fetch_offset := Int64.add !fetch_offset len + done; + ); + 0 + ) (* NBD.block_status *) + done -- 2.32.0

2 4

[PATCH] v2v: skip decryption of VM disks
by Maxim Davydov 13 Dec '21

13 Dec '21

In some situations, we want to be able to skip an encrypted disk automatically, without waiting for a passphrase from the user for this disk. For instance, it maybe be required by hosting service providers that use automatic guest reconfiguration but cannot have passphrases for encrypted guest disks. So this option can be part of "--in-place" because it is usually used in automatic mode. But this feature can also be implemented as a separate command line option, for example, "--skip-encrypt" Signed-off-by: Maxim Davydov <maxim.davydov(a)virtuozzo.com> --- v2v/v2v.ml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/v2v/v2v.ml b/v2v/v2v.ml index 68817da04..052c9f3ed 100644 --- a/v2v/v2v.ml +++ b/v2v/v2v.ml @@ -105,7 +105,7 @@ let rec main () = g#launch (); (* Decrypt the disks. *) - inspect_decrypt g cmdline.ks; + if not cmdline.in_place then inspect_decrypt g cmdline.ks; (* Inspection - this also mounts up the filesystems. *) (match conversion_mode with -- 2.25.1

2 2

libnbd | Failed pipeline for master | 41a9fea4
by GitLab 11 Dec '21

11 Dec '21

Pipeline #427476160 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 41a9fea4 ( https://gitlab.com/nbdkit/libnbd/-/commit/41a9fea46bbb1b66cee078d95dec5d458… ) Commit Message: ocaml/tests/test_505_aio_pread_structured_callb... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #427476160 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/427476160 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #1874368914 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1874368914/raw ) Stage: containers Name: x86_64-fedora-33-container -- You're receiving this email because of your account on gitlab.com.

1 0

[PATCH libnbd] golang: make-dist.sh: Use strict ISO 8601 format
by Nir Soffer 10 Dec '21

10 Dec '21

Go fail to parse the short date format (2021-11-30) from the @latest and .info file. Replace with %cI - strict ISO 8601 format. Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- golang/make-dist.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/golang/make-dist.sh b/golang/make-dist.sh index 86785e5d..c658f928 100755 --- a/golang/make-dist.sh +++ b/golang/make-dist.sh @@ -94,38 +94,40 @@ rm -rf libguestfs.org # └── v1.11.4.zip # # We create @latest and @v/*{.info,mod,zip} here. # # The "@v/list" file must be created on the web server after uploading # a new release: # # $ cd libguestfs.org/libnbd/@v # $ ls -1 v*.info | awk -F.info '{print $1}' > list # $ cat list # v1.11.3 # v1.11.4 # # See https://golang.org/ref/mod#serving-from-proxy module_dir=libguestfs.org/libnbd v_dir=$module_dir/@v mkdir -p $v_dir +# Go wants a string in RFC 3339, git strict ISO 8601 format is +# compatible. info="{ \"Version\": \"$version\", - \"Time\": \"$(git show -s --format=%cs)\" + \"Time\": \"$(git show -s --format=%cI)\" }" echo "$info" > $module_dir/@latest echo "$info" > $v_dir/$version.info cp go.mod $v_dir/$version.mod mv $version.zip $v_dir # Create tarball to upload and extract on the webserver. It should be # extracted in the directory pointed by the "go-import" meta tag. output=$PWD/libnbd-golang-$version.tar.gz tar czf $output libguestfs.org rm -rf libguestfs.org echo output written to $output -- 2.33.1

3 6

[virt-v2v PATCH v2 0/3] lib/create_ovf: populate "actual size" attributes again
by Laszlo Ersek 10 Dec '21

10 Dec '21

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2027598 v1: https://listman.redhat.com/archives/libguestfs/2021-December/msg00096.html In v2, move the new functions to the Utils module, from Nbdkit. Thanks, Laszlo Laszlo Ersek (3): lib/utils: add the "Utils.with_nbd_connect_unix" helper function lib/utils: add the "Utils.get_disk_allocated" helper function lib/create_ovf: populate "actual size" attributes again lib/Makefile.am | 2 +- lib/create_ovf.ml | 33 +++++++-------- lib/create_ovf.mli | 2 +- lib/utils.ml | 42 ++++++++++++++++++++ lib/utils.mli | 18 +++++++++ output/output_rhv.ml | 2 +- output/output_rhv_upload.ml | 2 +- output/output_vdsm.ml | 1 + tests/test-v2v-o-rhv.ovf.expected | 4 +- tests/test-v2v-o-vdsm-options.ovf.expected | 4 +- 10 files changed, 84 insertions(+), 26 deletions(-) base-commit: 9bb0e7f1d22936bf2c6c328c5ca48f32c7b9420a -- 2.19.1.3.g30247aa5d201

2 5

[PATCH] Add detection support for Rocky Linux (CentOS/RHEL-like)
by Neil Hanlon 09 Dec '21

09 Dec '21

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2001636 Thanks: label(a)rockylinux.org --- daemon/inspect_fs.ml | 2 ++ daemon/inspect_fs_unix.ml | 13 ++++++++++++- daemon/inspect_types.ml | 2 ++ generator/actions_inspection.ml | 4 ++++ lib/inspect-icon.c | 1 + lib/inspect-osinfo.c | 4 ++++ website/index.html.in | 2 +- 7 files changed, 26 insertions(+), 2 deletions(-) diff --git a/daemon/inspect_fs.ml b/daemon/inspect_fs.ml index 77f0f6aea..9c73d97ef 100644 --- a/daemon/inspect_fs.ml +++ b/daemon/inspect_fs.ml @@ -259,6 +259,7 @@ and check_package_format { distro } = | None -> None | Some DISTRO_ALTLINUX | Some DISTRO_CENTOS + | Some DISTRO_ROCKY | Some DISTRO_FEDORA | Some DISTRO_MAGEIA | Some DISTRO_MANDRIVA @@ -329,6 +330,7 @@ and check_package_management { distro; version } = Some PACKAGE_MANAGEMENT_DNF | Some DISTRO_CENTOS + | Some DISTRO_ROCKY | Some DISTRO_ORACLE_LINUX | Some DISTRO_REDHAT_BASED | Some DISTRO_RHEL diff --git a/daemon/inspect_fs_unix.ml b/daemon/inspect_fs_unix.ml index 7f6eb92e9..63cb279d0 100644 --- a/daemon/inspect_fs_unix.ml +++ b/daemon/inspect_fs_unix.ml @@ -32,6 +32,8 @@ let re_rhel_no_minor = PCRE.compile "Red Hat.*release (\\d+)" let re_centos_old = PCRE.compile "CentOS.*release (\\d+).*Update (\\d+)" let re_centos = PCRE.compile "CentOS.*release (\\d+)\\.(\\d+)" let re_centos_no_minor = PCRE.compile "CentOS.*release (\\d+)" +let re_rocky = PCRE.compile "Rocky Linux.*release (\\d+)\\.(\\d+)" +let re_rocky_no_minor = PCRE.compile "Rocky Linux.*release (\\d+)" let re_scientific_linux_old = PCRE.compile "Scientific Linux.*release (\\d+).*Update (\\d+)" let re_scientific_linux = @@ -106,7 +108,7 @@ let rec parse_os_release release_file data = * we detect that situation then bail out and use the release * files instead. *) - | { distro = Some (DISTRO_DEBIAN|DISTRO_CENTOS); + | { distro = Some (DISTRO_DEBIAN|DISTRO_CENTOS|DISTRO_ROCKY); version = Some (_, 0) } -> false @@ -155,6 +157,7 @@ and distro_of_os_release_id = function | "pardus" -> Some DISTRO_PARDUS | "pld" -> Some DISTRO_PLD_LINUX | "rhel" -> Some DISTRO_RHEL + | "rocky" -> Some DISTRO_ROCKY | "sles" | "sled" -> Some DISTRO_SLES | "ubuntu" -> Some DISTRO_UBUNTU | "void" -> Some DISTRO_VOID_LINUX @@ -405,6 +408,10 @@ let linux_root_tests : tests = [ DISTRO_CENTOS; "/etc/centos-release", parse_generic ~rex:re_centos_no_minor DISTRO_CENTOS; + "/etc/rocky-release", parse_generic ~rex:re_rocky + DISTRO_ROCKY; + "/etc/rocky-release", parse_generic ~rex:re_rocky_no_minor + DISTRO_ROCKY; "/etc/altlinux-release", parse_generic DISTRO_ALTLINUX; "/etc/redhat-release", parse_generic ~rex:re_fedora DISTRO_FEDORA; @@ -420,6 +427,10 @@ let linux_root_tests : tests = [ DISTRO_CENTOS; "/etc/redhat-release", parse_generic ~rex:re_centos_no_minor DISTRO_CENTOS; + "/etc/redhat-release", parse_generic ~rex:re_rocky + DISTRO_ROCKY; + "/etc/redhat-release", parse_generic ~rex:re_rocky_no_minor + DISTRO_ROCKY; "/etc/redhat-release", parse_generic ~rex:re_scientific_linux_old DISTRO_SCIENTIFIC_LINUX; "/etc/redhat-release", parse_generic ~rex:re_scientific_linux diff --git a/daemon/inspect_types.ml b/daemon/inspect_types.ml index e2bc7165c..9395c51f9 100644 --- a/daemon/inspect_types.ml +++ b/daemon/inspect_types.ml @@ -95,6 +95,7 @@ and distro = | DISTRO_PLD_LINUX | DISTRO_REDHAT_BASED | DISTRO_RHEL + | DISTRO_ROCKY | DISTRO_SCIENTIFIC_LINUX | DISTRO_SLACKWARE | DISTRO_SLES @@ -228,6 +229,7 @@ and string_of_distro = function | DISTRO_PLD_LINUX -> "pldlinux" | DISTRO_REDHAT_BASED -> "redhat-based" | DISTRO_RHEL -> "rhel" + | DISTRO_ROCKY -> "rocky" | DISTRO_SCIENTIFIC_LINUX -> "scientificlinux" | DISTRO_SLACKWARE -> "slackware" | DISTRO_SLES -> "sles" diff --git a/generator/actions_inspection.ml b/generator/actions_inspection.ml index 0c6d39b43..6ace7cbd4 100644 --- a/generator/actions_inspection.ml +++ b/generator/actions_inspection.ml @@ -278,6 +278,10 @@ Some Red Hat-derived distro. Red Hat Enterprise Linux. +=item \"rhel\" + +Rocky Linux. + =item \"scientificlinux\" Scientific Linux. diff --git a/lib/inspect-icon.c b/lib/inspect-icon.c index 725af574b..3bffa4f80 100644 --- a/lib/inspect-icon.c +++ b/lib/inspect-icon.c @@ -138,6 +138,7 @@ guestfs_impl_inspect_get_icon (guestfs_h *g, const char *root, size_t *size_r, else if (STREQ (distro, "rhel") || STREQ (distro, "redhat-based") || STREQ (distro, "centos") || + STREQ (distro, "rocky") || STREQ (distro, "scientificlinux") || STREQ (distro, "oraclelinux")) { r = icon_rhel (g, guestfs_inspect_get_major_version (g, root), &size); diff --git a/lib/inspect-osinfo.c b/lib/inspect-osinfo.c index db38d87f7..90e57e6df 100644 --- a/lib/inspect-osinfo.c +++ b/lib/inspect-osinfo.c @@ -47,6 +47,10 @@ guestfs_impl_inspect_get_osinfo (guestfs_h *g, const char *root) else if (major == 6) return safe_asprintf (g, "%s%d.%d", distro, major, minor); } + else if (STREQ (distro, "rocky")) { + if (major >= 8) + return safe_asprintf (g, "%s%d", distro, major); + } else if (STREQ (distro, "debian")) { if (major >= 4) return safe_asprintf (g, "%s%d", distro, major); diff --git a/website/index.html.in b/website/index.html.in index 7453129d6..08b7a6015 100644 --- a/website/index.html.in +++ b/website/index.html.in @@ -261,7 +261,7 @@ enterprise environments, and with many happy and successful users. <h3>Getting started</h3> <pre> -sudo yum install libguestfs-tools # Fedora/RHEL/CentOS +sudo yum install libguestfs-tools # Fedora/RHEL/CentOS/Rocky sudo apt-get install libguestfs-tools # Debian/Ubuntu guestfish --ro -i -a disk.img </pre> -- 2.33.1

1 0

[PATCH common v2] utils: Fix usage of strerror_r
by Richard W.M. Jones 09 Dec '21

09 Dec '21

Slightly modified version of the previous fix, but essentially the same thing. I have put the change into common/utils/utils.c so that I could use the same code to fix the Lua bindings which were also calling strerror_r incorrectly. Rich.

2 4