[PATCH] convert_linux: translate the first CD-ROM's references in boot conf files
If the only CD-ROM in "s_removables" is on an IDE controller, and the guest kernel represents it with a /dev/hdX device node, then convert references to this device node, in the boot config files, to /dev/cdrom. On the destination (after conversion), /dev/cdrom will point to whataver node we converted the CD-ROM to, masking a potential i440fx -> q35 (IDE -> SATA) board change. If the only CD-ROM is not on an IDE controller, or the guest is modern enough to represent the IDE CD-ROM as /dev/sr0, then perform no translation. Namely, /dev/sr0 survives a potential i440fx -> q35 (IDE -> SATA) board change intact. When multiple CD-ROMs exist, emit a warning, and attempt the conversion on the first CD-ROM, as a guess. This may be inexact, but we can't do better, because: - SATA, SCSI, and (on modern guests) IDE CD-ROMs are lumped together in the /dev/sr* namespace, on the source side, and "s_removable_slot" is useless for telling them apart, as we don't know the exact controller topology (and OS enumeration order); - after conversion: some OSes don't create /dev/cdrom* symlinks to all CD-ROMs, and even if multiple such symlinks are created, their order is basically random. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1637857 Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- convert/convert_linux.ml | 25 +++++++++++++++++++++++++ tests/test-v2v-cdrom.expected | 2 +- tests/test-v2v-cdrom.xml.in | 4 +++- tests/test-v2v-i-ova.xml | 2 +- 4 files changed, 30 insertions(+), 3 deletions(-) diff --git a/convert/convert_linux.ml b/convert/convert_linux.ml index 8dc648169dcb..9ea94b75bfc9 100644 --- a/convert/convert_linux.ml +++ b/convert/convert_linux.ml @@ -1020,6 +1020,31 @@ let convert (g : G.guestfs) source inspect keep_serial_console _ = "xvd" ^ drive_name i, block_prefix_after_conversion ^ drive_name i ) source.s_disks in + (* Check the first CD-ROM. If its controller is IDE, and the OS is RHEL<=5, + * then translate the CD-ROM from "/dev/hd[SLOT]" to "/dev/cdrom". See + * RHBZ#1637857 for details. + *) + let cdroms = List.filter + (fun removable -> removable.s_removable_type = CDROM) + source.s_removables in + + match cdroms with + | _ :: _ :: _ -> warning (f_"multiple CD-ROMs found; translation of \ + CD-ROM references may be inexact") + | _ -> (); + + let map = map @ + (match cdroms with + | cdrom :: _ -> + (match (cdrom.s_removable_controller, cdrom.s_removable_slot, + family, inspect.i_major_version) with + | Some Source_IDE, Some slot, `RHEL_family, v when v <= 5 -> + [("hd" ^ drive_name slot, "cdrom")] + | _ -> [] + ) + | [] -> [] + ) in + if verbose () then ( eprintf "block device map:\n"; List.iter ( diff --git a/tests/test-v2v-cdrom.expected b/tests/test-v2v-cdrom.expected index 34d2bf5961b0..17bd152d8e64 100644 --- a/tests/test-v2v-cdrom.expected +++ b/tests/test-v2v-cdrom.expected @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> diff --git a/tests/test-v2v-cdrom.xml.in b/tests/test-v2v-cdrom.xml.in index 6bad5eab1cd4..a6e1e3f514d5 100644 --- a/tests/test-v2v-cdrom.xml.in +++ b/tests/test-v2v-cdrom.xml.in @@ -35,7 +35,9 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='@abs_top_builddir@/test-data/phony-guests/blank-disk.img'/> - <!-- virt-v2v should preserve the device name and bus --> + <!-- virt-v2v should change the bus to sata, due to Windows 7 + triggering a machine type change from i440fx to q35. Beyond that, + virt-v2v should preserve the on-bus index. --> <target dev='hdc' bus='ide'/> </disk> </devices> diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml index d7383905fdc0..9f3c1974243f 100644 --- a/tests/test-v2v-i-ova.xml +++ b/tests/test-v2v-i-ova.xml @@ -28,7 +28,7 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hda' bus='ide'/> + <target dev='sda' bus='sata'/> </disk> <disk device='floppy' type='file'> <driver name='qemu' type='raw'/> -- 2.19.1.3.g30247aa5d201
On Tue, Dec 14, 2021 at 04:17:49PM +0100, Laszlo Ersek wrote:
+ match cdroms with + | _ :: _ :: _ -> warning (f_"multiple CD-ROMs found; translation of \ + CD-ROM references may be inexact") + | _ -> (); + + let map = map @
This part is either wrong or indented incorrectly. The "let map ..." part is part of the second branch of the match statement and doesn't run if the warning is printed. In any case it's probably better to replace the confusing first match with something simpler such as: if List.length cdroms > 2 then warning (f_"multiple CD-ROMs found; translation of CD-ROM references may be inexact"); The \ is also not necessary in the original since OCaml lets you have arbitrary length strings over multiple lines (a good thing), plus we automatically wrap warning and error strings on whitespace.
+ (match cdroms with + | cdrom :: _ -> + (match (cdrom.s_removable_controller, cdrom.s_removable_slot, + family, inspect.i_major_version) with + | Some Source_IDE, Some slot, `RHEL_family, v when v <= 5 ->
You can just match to arbitrary depth in single match statements, so the whole match would become something like: let map = map @ match cdroms with | { Some Source_IDE, Some slot, `RHEL_family v } :: _ when v <= 5 -> [("hd" ^ drive_name slot, "cdrom")] | _ -> [] in Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Tue, Dec 14, 2021 at 08:46:45PM +0000, Richard W.M. Jones wrote:
On Tue, Dec 14, 2021 at 04:17:49PM +0100, Laszlo Ersek wrote:
+ match cdroms with + | _ :: _ :: _ -> warning (f_"multiple CD-ROMs found; translation of \ + CD-ROM references may be inexact") + | _ -> (); + + let map = map @
This part is either wrong or indented incorrectly. The "let map ..." part is part of the second branch of the match statement and doesn't run if the warning is printed.
In any case it's probably better to replace the confusing first match with something simpler such as:
if List.length cdroms > 2 then
">=" ? I said it was confusing :-) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Tue, Dec 14, 2021 at 08:46:45PM +0000, Richard W.M. Jones wrote:
On Tue, Dec 14, 2021 at 04:17:49PM +0100, Laszlo Ersek wrote:
+ match cdroms with + | _ :: _ :: _ -> warning (f_"multiple CD-ROMs found; translation of \ + CD-ROM references may be inexact") + | _ -> (); + + let map = map @
This part is either wrong or indented incorrectly. The "let map ..." part is part of the second branch of the match statement and doesn't run if the warning is printed.
In any case it's probably better to replace the confusing first match with something simpler such as:
if List.length cdroms > 2 then warning (f_"multiple CD-ROMs found; translation of CD-ROM references may be inexact");
The \ is also not necessary in the original since OCaml lets you have arbitrary length strings over multiple lines (a good thing), plus we automatically wrap warning and error strings on whitespace.
+ (match cdroms with + | cdrom :: _ -> + (match (cdrom.s_removable_controller, cdrom.s_removable_slot, + family, inspect.i_major_version) with + | Some Source_IDE, Some slot, `RHEL_family, v when v <= 5 ->
You can just match to arbitrary depth in single match statements, so the whole match would become something like:
let map = map @ match cdroms with | { Some Source_IDE, Some slot, `RHEL_family v } :: _ when v <= 5 ->
This line should have been: | { s_removable_slot = Some Source_IDE; s_removable_slot = Some slot } :: _ when family = `RHEL_family && inspect.i_major_version <= 5 -> Rich.
[("hd" ^ drive_name slot, "cdrom")] | _ -> [] in
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On 12/14/21 21:51, Richard W.M. Jones wrote:
On Tue, Dec 14, 2021 at 08:46:45PM +0000, Richard W.M. Jones wrote:
On Tue, Dec 14, 2021 at 04:17:49PM +0100, Laszlo Ersek wrote:
+ match cdroms with + | _ :: _ :: _ -> warning (f_"multiple CD-ROMs found; translation of \ + CD-ROM references may be inexact") + | _ -> (); + + let map = map @
This part is either wrong or indented incorrectly. The "let map ..." part is part of the second branch of the match statement and doesn't run if the warning is printed.
In any case it's probably better to replace the confusing first match with something simpler such as:
if List.length cdroms > 2 then warning (f_"multiple CD-ROMs found; translation of CD-ROM references may be inexact");
The \ is also not necessary in the original since OCaml lets you have arbitrary length strings over multiple lines (a good thing), plus we automatically wrap warning and error strings on whitespace.
+ (match cdroms with + | cdrom :: _ -> + (match (cdrom.s_removable_controller, cdrom.s_removable_slot, + family, inspect.i_major_version) with + | Some Source_IDE, Some slot, `RHEL_family, v when v <= 5 ->
You can just match to arbitrary depth in single match statements, so the whole match would become something like:
let map = map @ match cdroms with | { Some Source_IDE, Some slot, `RHEL_family v } :: _ when v <= 5 ->
This line should have been:
| { s_removable_slot = Some Source_IDE; s_removable_slot = Some slot } :: _ when family = `RHEL_family && inspect.i_major_version <= 5 ->
Yes, in an earlier version, I used some structure field matching, and just found it impossible to remember the syntax for that. Had to look it up in the book... So, now I'm even less sure that this is more readable :) Please advise, I'll heed it. Thanks, Laszlo
Rich.
[("hd" ^ drive_name slot, "cdrom")] | _ -> [] in
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 12/14/21 21:46, Richard W.M. Jones wrote:
On Tue, Dec 14, 2021 at 04:17:49PM +0100, Laszlo Ersek wrote:
+ match cdroms with + | _ :: _ :: _ -> warning (f_"multiple CD-ROMs found; translation of \ + CD-ROM references may be inexact") + | _ -> (); + + let map = map @
This part is either wrong or indented incorrectly. The "let map ..." part is part of the second branch of the match statement and doesn't run if the warning is printed.
*groan* I admit the binding strengths of these operators keep confusing me. (Also, I meant to post this as an RFC in the first round...)
In any case it's probably better to replace the confusing first match with something simpler such as:
if List.length cdroms > 2 then warning (f_"multiple CD-ROMs found; translation of CD-ROM references may be inexact");
Yes, this was my original version, but I figured not counting the full length (until the end of the list) is more welcome. I can update it though.
The \ is also not necessary in the original since OCaml lets you have arbitrary length strings over multiple lines (a good thing),
I'm aware, however. :) I *really* dislike the one aspect of the current multi-line messages that they are nailed to column #0. It disrupts the code flow for me. For example, consider the following snippet from this very file [convert/convert_linux.ml]: g#cp grub_config uefi_grub_conf; let fix_script = sprintf "#!/bin/bash efibootmgr -c -L \"CentOS 6\" rm -rf %s" uefi_fallback_path in Firstboot.add_firstboot_script g inspect.i_root "fix uefi boot" fix_script) I find it less than ideal. I'd really like to indent such strings logically. And that's exactly what the backslash is for (I had checked the ocaml documentation): after an escaped newline, leading spaces and tabs are dropped. I think that's a fantastic feature of OCaml; I used it intentionally.
plus we automatically wrap warning and error strings on whitespace.
That sounds very user friendly (I didn't expect it!), but does not contradict my point about formatting the source code, AFAICT.
+ (match cdroms with + | cdrom :: _ -> + (match (cdrom.s_removable_controller, cdrom.s_removable_slot, + family, inspect.i_major_version) with + | Some Source_IDE, Some slot, `RHEL_family, v when v <= 5 ->
You can just match to arbitrary depth in single match statements, so the whole match would become something like:
let map = map @ match cdroms with | { Some Source_IDE, Some slot, `RHEL_family v } :: _ when v <= 5 -> [("hd" ^ drive_name slot, "cdrom")] | _ -> [] in
Does this make the code easier to read to you? (Honest question.) To me, this does not separate the steps "take first" and "investigate first", and so it's harder to read. But if that's only because I don't have enough OCaml experience yet, I'm happy to change it. (Because, in comparison, you might find the two "match" expressions to be the distraction.) Thanks! Laszlo
On Wed, Dec 15, 2021 at 11:17:51AM +0100, Laszlo Ersek wrote:
I *really* dislike the one aspect of the current multi-line messages that they are nailed to column #0. It disrupts the code flow for me. For example, consider the following snippet from this very file [convert/convert_linux.ml]:
g#cp grub_config uefi_grub_conf; let fix_script = sprintf "#!/bin/bash efibootmgr -c -L \"CentOS 6\" rm -rf %s" uefi_fallback_path in Firstboot.add_firstboot_script g inspect.i_root "fix uefi boot" fix_script)
I find it less than ideal.
I'd really like to indent such strings logically. And that's exactly what the backslash is for (I had checked the ocaml documentation): after an escaped newline, leading spaces and tabs are dropped. I think that's a fantastic feature of OCaml; I used it intentionally.
Right, this is indeed a reason to use the \, as it drops the following whitespace.
You can just match to arbitrary depth in single match statements, so the whole match would become something like:
let map = map @ match cdroms with | { Some Source_IDE, Some slot, `RHEL_family v } :: _ when v <= 5 -> [("hd" ^ drive_name slot, "cdrom")] | _ -> [] in
Does this make the code easier to read to you? (Honest question.) To me, this does not separate the steps "take first" and "investigate first", and so it's harder to read. But if that's only because I don't have enough OCaml experience yet, I'm happy to change it. (Because, in comparison, you might find the two "match" expressions to be the distraction.)
For me it's less code => better, within reason. The above (after fixing the bugs - see later emails) is more natural. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On 12/15/21 11:57, Richard W.M. Jones wrote:
On Wed, Dec 15, 2021 at 11:17:51AM +0100, Laszlo Ersek wrote:
I *really* dislike the one aspect of the current multi-line messages that they are nailed to column #0. It disrupts the code flow for me. For example, consider the following snippet from this very file [convert/convert_linux.ml]:
g#cp grub_config uefi_grub_conf; let fix_script = sprintf "#!/bin/bash efibootmgr -c -L \"CentOS 6\" rm -rf %s" uefi_fallback_path in Firstboot.add_firstboot_script g inspect.i_root "fix uefi boot" fix_script)
I find it less than ideal.
I'd really like to indent such strings logically. And that's exactly what the backslash is for (I had checked the ocaml documentation): after an escaped newline, leading spaces and tabs are dropped. I think that's a fantastic feature of OCaml; I used it intentionally.
Right, this is indeed a reason to use the \, as it drops the following whitespace.
You can just match to arbitrary depth in single match statements, so the whole match would become something like:
let map = map @ match cdroms with | { Some Source_IDE, Some slot, `RHEL_family v } :: _ when v <= 5 -> [("hd" ^ drive_name slot, "cdrom")] | _ -> [] in
Does this make the code easier to read to you? (Honest question.) To me, this does not separate the steps "take first" and "investigate first", and so it's harder to read. But if that's only because I don't have enough OCaml experience yet, I'm happy to change it. (Because, in comparison, you might find the two "match" expressions to be the distraction.)
For me it's less code => better, within reason. The above (after fixing the bugs - see later emails) is more natural.
Before posting v2, I intended to test it :) , so I installed a new RHEL5 guest, and tried to run virt-v2v (with the v2 patch) on it: virt-v2v -i libvirt -o local -os ~/output rhel5.11 Unfortunately, things break before my code gets a chance to run; here: (* Fail early if i_apps is empty. Certain steps such as kernel * detection won't work without this. If the list is empty it * likely indicates that libguestfs inspection is broken for * this guest. See for example RHBZ#1965147. *) if inspect.i_apps = [] then error (f_"inspection of the package database failed for ... This seems to come from an empty apps list, from "convert/inspect_source.ml", which in turn invokes "g#inspect_list_applications2". I turned to guestfish, and ran the following manually: inspect-os inspect-get-mountpoints /dev/VolGroup00/LogVol00 inspect-list-applications2 /dev/VolGroup00/LogVol00 Indeed the last command does not return anything. I turned on "verbose" and "trace" then, and then I see a failure in the daemon: libguestfs: trace: inspect_list_applications2 "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type = "linux" libguestfs: trace: inspect_get_package_format "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_package_format = "rpm" libguestfs: trace: internal_list_rpm_applications \x1b[31;1merror: \x1b[0m\x1b[1mUnable to open /usr/lib/rpm/rpmrc for reading: No such file or directory. \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mno dbpath has been set \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mcannot open Packages database in \x1b[0mlibrpm returned 0 installed packages libguestfs: trace: internal_list_rpm_applications = <struct guestfs_application2_list(0)> libguestfs: trace: inspect_list_applications2 = <struct guestfs_application2_list(0)> The file "/usr/lib/rpm/rpmrc" absolutely exists in the guest (I booted it up separately, and checked it). It seems that "internal_list_rpm_applications" [daemon/rpm.ml] calls chroot, and uses a child project -- can that be related to the issue? Anyway, I think I'll post v2, just saying that I could not actually test it. Thanks, Laszlo
On 12/15/21 16:14, Laszlo Ersek wrote:
Before posting v2, I intended to test it :) , so I installed a new RHEL5 guest, and tried to run virt-v2v (with the v2 patch) on it:
virt-v2v -i libvirt -o local -os ~/output rhel5.11
Unfortunately, things break before my code gets a chance to run; here:
(* Fail early if i_apps is empty. Certain steps such as kernel * detection won't work without this. If the list is empty it * likely indicates that libguestfs inspection is broken for * this guest. See for example RHBZ#1965147. *) if inspect.i_apps = [] then error (f_"inspection of the package database failed for ...
This seems to come from an empty apps list, from "convert/inspect_source.ml", which in turn invokes "g#inspect_list_applications2".
I turned to guestfish, and ran the following manually:
inspect-os inspect-get-mountpoints /dev/VolGroup00/LogVol00 inspect-list-applications2 /dev/VolGroup00/LogVol00
Indeed the last command does not return anything. I turned on "verbose" and "trace" then, and then I see a failure in the daemon:
libguestfs: trace: inspect_list_applications2 "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type = "linux" libguestfs: trace: inspect_get_package_format "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_package_format = "rpm" libguestfs: trace: internal_list_rpm_applications \x1b[31;1merror: \x1b[0m\x1b[1mUnable to open /usr/lib/rpm/rpmrc for reading: No such file or directory. \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mno dbpath has been set \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mcannot open Packages database in \x1b[0mlibrpm returned 0 installed packages libguestfs: trace: internal_list_rpm_applications = <struct guestfs_application2_list(0)> libguestfs: trace: inspect_list_applications2 = <struct guestfs_application2_list(0)>
The file "/usr/lib/rpm/rpmrc" absolutely exists in the guest (I booted it up separately, and checked it).
It seems that "internal_list_rpm_applications" [daemon/rpm.ml] calls chroot, and uses a child project -- can that be related to the issue?
Anyway, I think I'll post v2, just saying that I could not actually test it.
BTW my libguestfs has been last built at commit d829f9ff9ae0 ("daemon/listfs: don't call "sgdisk -i" on bogus MBR partition table entry", 2021-11-26). Thanks Laszlo
On Wed, Dec 15, 2021 at 04:14:42PM +0100, Laszlo Ersek wrote:
Before posting v2, I intended to test it :) , so I installed a new RHEL5 guest, and tried to run virt-v2v (with the v2 patch) on it:
virt-v2v -i libvirt -o local -os ~/output rhel5.11
As a general comment, you might want to check virt-v2v 1.44 branch to see if these problems are regressions in virt-v2v or something else. In this case ...
Unfortunately, things break before my code gets a chance to run; here:
(* Fail early if i_apps is empty. Certain steps such as kernel * detection won't work without this. If the list is empty it * likely indicates that libguestfs inspection is broken for * this guest. See for example RHBZ#1965147. *) if inspect.i_apps = [] then error (f_"inspection of the package database failed for ...
This seems to come from an empty apps list, from "convert/inspect_source.ml", which in turn invokes "g#inspect_list_applications2".
I turned to guestfish, and ran the following manually:
inspect-os inspect-get-mountpoints /dev/VolGroup00/LogVol00 inspect-list-applications2 /dev/VolGroup00/LogVol00
Indeed the last command does not return anything. I turned on "verbose" and "trace" then, and then I see a failure in the daemon:
libguestfs: trace: inspect_list_applications2 "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type = "linux" libguestfs: trace: inspect_get_package_format "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_package_format = "rpm" libguestfs: trace: internal_list_rpm_applications \x1b[31;1merror: \x1b[0m\x1b[1mUnable to open /usr/lib/rpm/rpmrc for reading: No such file or directory. \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mno dbpath has been set \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mcannot open Packages database in \x1b[0mlibrpm returned 0 installed packages libguestfs: trace: internal_list_rpm_applications = <struct guestfs_application2_list(0)> libguestfs: trace: inspect_list_applications2 = <struct guestfs_application2_list(0)>
... I think this is potentially an instance of: [sorry, BZ is down - I will follow up with the bug number] The new thing here (not really related to virt-v2v) is that RPM itself changed from Berkley DB to sqlite. However old libguestfs actually accessed the BDB directly and could not read the RPM databases of new guests. So we switched to using librpm (instead of direct access), but this meant for a time we couldn't access the RPM DBs of *old* guests. However that was eventually fixed in RPM. https://github.com/libguestfs/libguestfs/commit/c9ee831affed55abe0f928134cbb... Rich.
The file "/usr/lib/rpm/rpmrc" absolutely exists in the guest (I booted it up separately, and checked it).
It seems that "internal_list_rpm_applications" [daemon/rpm.ml] calls chroot, and uses a child project -- can that be related to the issue?
Anyway, I think I'll post v2, just saying that I could not actually test it.
Thanks, Laszlo
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Wed, Dec 15, 2021 at 03:33:00PM +0000, Richard W.M. Jones wrote:
On Wed, Dec 15, 2021 at 04:14:42PM +0100, Laszlo Ersek wrote:
Before posting v2, I intended to test it :) , so I installed a new RHEL5 guest, and tried to run virt-v2v (with the v2 patch) on it:
virt-v2v -i libvirt -o local -os ~/output rhel5.11
As a general comment, you might want to check virt-v2v 1.44 branch to see if these problems are regressions in virt-v2v or something else.
In this case ...
Unfortunately, things break before my code gets a chance to run; here:
(* Fail early if i_apps is empty. Certain steps such as kernel * detection won't work without this. If the list is empty it * likely indicates that libguestfs inspection is broken for * this guest. See for example RHBZ#1965147. *) if inspect.i_apps = [] then error (f_"inspection of the package database failed for ...
This seems to come from an empty apps list, from "convert/inspect_source.ml", which in turn invokes "g#inspect_list_applications2".
I turned to guestfish, and ran the following manually:
inspect-os inspect-get-mountpoints /dev/VolGroup00/LogVol00 inspect-list-applications2 /dev/VolGroup00/LogVol00
Indeed the last command does not return anything. I turned on "verbose" and "trace" then, and then I see a failure in the daemon:
libguestfs: trace: inspect_list_applications2 "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type = "linux" libguestfs: trace: inspect_get_package_format "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_package_format = "rpm" libguestfs: trace: internal_list_rpm_applications \x1b[31;1merror: \x1b[0m\x1b[1mUnable to open /usr/lib/rpm/rpmrc for reading: No such file or directory. \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mno dbpath has been set \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mcannot open Packages database in \x1b[0mlibrpm returned 0 installed packages libguestfs: trace: internal_list_rpm_applications = <struct guestfs_application2_list(0)> libguestfs: trace: inspect_list_applications2 = <struct guestfs_application2_list(0)>
... I think this is potentially an instance of:
[sorry, BZ is down - I will follow up with the bug number]
https://bugzilla.redhat.com/show_bug.cgi?id=1965147
The new thing here (not really related to virt-v2v) is that RPM itself changed from Berkley DB to sqlite. However old libguestfs actually accessed the BDB directly and could not read the RPM databases of new guests. So we switched to using librpm (instead of direct access), but this meant for a time we couldn't access the RPM DBs of *old* guests. However that was eventually fixed in RPM.
https://github.com/libguestfs/libguestfs/commit/c9ee831affed55abe0f928134cbb...
Rich.
The file "/usr/lib/rpm/rpmrc" absolutely exists in the guest (I booted it up separately, and checked it).
It seems that "internal_list_rpm_applications" [daemon/rpm.ml] calls chroot, and uses a child project -- can that be related to the issue?
Anyway, I think I'll post v2, just saying that I could not actually test it.
Thanks, Laszlo
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On 12/15/21 16:34, Richard W.M. Jones wrote:
On Wed, Dec 15, 2021 at 03:33:00PM +0000, Richard W.M. Jones wrote:
On Wed, Dec 15, 2021 at 04:14:42PM +0100, Laszlo Ersek wrote:
Before posting v2, I intended to test it :) , so I installed a new RHEL5 guest, and tried to run virt-v2v (with the v2 patch) on it:
virt-v2v -i libvirt -o local -os ~/output rhel5.11
As a general comment, you might want to check virt-v2v 1.44 branch to see if these problems are regressions in virt-v2v or something else.
In this case ...
Unfortunately, things break before my code gets a chance to run; here:
(* Fail early if i_apps is empty. Certain steps such as kernel * detection won't work without this. If the list is empty it * likely indicates that libguestfs inspection is broken for * this guest. See for example RHBZ#1965147. *) if inspect.i_apps = [] then error (f_"inspection of the package database failed for ...
This seems to come from an empty apps list, from "convert/inspect_source.ml", which in turn invokes "g#inspect_list_applications2".
I turned to guestfish, and ran the following manually:
inspect-os inspect-get-mountpoints /dev/VolGroup00/LogVol00 inspect-list-applications2 /dev/VolGroup00/LogVol00
Indeed the last command does not return anything. I turned on "verbose" and "trace" then, and then I see a failure in the daemon:
libguestfs: trace: inspect_list_applications2 "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type = "linux" libguestfs: trace: inspect_get_package_format "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_package_format = "rpm" libguestfs: trace: internal_list_rpm_applications \x1b[31;1merror: \x1b[0m\x1b[1mUnable to open /usr/lib/rpm/rpmrc for reading: No such file or directory. \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mno dbpath has been set \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mcannot open Packages database in \x1b[0mlibrpm returned 0 installed packages libguestfs: trace: internal_list_rpm_applications = <struct guestfs_application2_list(0)> libguestfs: trace: inspect_list_applications2 = <struct guestfs_application2_list(0)>
... I think this is potentially an instance of:
[sorry, BZ is down - I will follow up with the bug number]
Yep, that's exactly the BZ number that's highlighted in the code comment above. I had read the BZ quickly before posting my findings. However, the RPM error messages in the BZ are different. (I can't open the BZ ticket again because bugzilla is down (yay!), but I seem to remember that the librpm complaints were not about failures to open existent files. I could be wrong though, from memory.) Anyway, I guess this indicates that my Fedora 34 host does not have a recent enough librpm (and so the appliance too has an out-of-date librpm). Thanks, Laszlo
The new thing here (not really related to virt-v2v) is that RPM itself changed from Berkley DB to sqlite. However old libguestfs actually accessed the BDB directly and could not read the RPM databases of new guests. So we switched to using librpm (instead of direct access), but this meant for a time we couldn't access the RPM DBs of *old* guests. However that was eventually fixed in RPM.
https://github.com/libguestfs/libguestfs/commit/c9ee831affed55abe0f928134cbb...
Rich.
The file "/usr/lib/rpm/rpmrc" absolutely exists in the guest (I booted it up separately, and checked it).
It seems that "internal_list_rpm_applications" [daemon/rpm.ml] calls chroot, and uses a child project -- can that be related to the issue?
Anyway, I think I'll post v2, just saying that I could not actually test it.
Thanks, Laszlo
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Wed, Dec 15, 2021 at 04:40:22PM +0100, Laszlo Ersek wrote:
On 12/15/21 16:34, Richard W.M. Jones wrote:
On Wed, Dec 15, 2021 at 03:33:00PM +0000, Richard W.M. Jones wrote:
On Wed, Dec 15, 2021 at 04:14:42PM +0100, Laszlo Ersek wrote:
Before posting v2, I intended to test it :) , so I installed a new RHEL5 guest, and tried to run virt-v2v (with the v2 patch) on it:
virt-v2v -i libvirt -o local -os ~/output rhel5.11
As a general comment, you might want to check virt-v2v 1.44 branch to see if these problems are regressions in virt-v2v or something else.
In this case ...
Unfortunately, things break before my code gets a chance to run; here:
(* Fail early if i_apps is empty. Certain steps such as kernel * detection won't work without this. If the list is empty it * likely indicates that libguestfs inspection is broken for * this guest. See for example RHBZ#1965147. *) if inspect.i_apps = [] then error (f_"inspection of the package database failed for ...
This seems to come from an empty apps list, from "convert/inspect_source.ml", which in turn invokes "g#inspect_list_applications2".
I turned to guestfish, and ran the following manually:
inspect-os inspect-get-mountpoints /dev/VolGroup00/LogVol00 inspect-list-applications2 /dev/VolGroup00/LogVol00
Indeed the last command does not return anything. I turned on "verbose" and "trace" then, and then I see a failure in the daemon:
libguestfs: trace: inspect_list_applications2 "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_type = "linux" libguestfs: trace: inspect_get_package_format "/dev/VolGroup00/LogVol00" libguestfs: trace: inspect_get_package_format = "rpm" libguestfs: trace: internal_list_rpm_applications \x1b[31;1merror: \x1b[0m\x1b[1mUnable to open /usr/lib/rpm/rpmrc for reading: No such file or directory. \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mno dbpath has been set \x1b[0m\x1b[31;1merror: \x1b[0m\x1b[1mcannot open Packages database in \x1b[0mlibrpm returned 0 installed packages libguestfs: trace: internal_list_rpm_applications = <struct guestfs_application2_list(0)> libguestfs: trace: inspect_list_applications2 = <struct guestfs_application2_list(0)>
... I think this is potentially an instance of:
[sorry, BZ is down - I will follow up with the bug number]
Yep, that's exactly the BZ number that's highlighted in the code comment above. I had read the BZ quickly before posting my findings. However, the RPM error messages in the BZ are different. (I can't open the BZ ticket again because bugzilla is down (yay!), but I seem to remember that the librpm complaints were not about failures to open existent files. I could be wrong though, from memory.)
Anyway, I guess this indicates that my Fedora 34 host does not have a recent enough librpm (and so the appliance too has an out-of-date librpm).
The bottom line is still that it's librpm's problem now to handle any guest rpmdb, no matter how ancient (well, at least RHEL >= 5). If you have the latest rpm installed and it still doesn't work then it could be a bug in librpm, although I'm pretty sure I tested this case. Easy test is something like: $ virt-builder rhel-5.10 $ virt-inspector -a rhel-5.10.img (I'll send you privately some links to make those commands work) Rich.
Thanks, Laszlo
The new thing here (not really related to virt-v2v) is that RPM itself changed from Berkley DB to sqlite. However old libguestfs actually accessed the BDB directly and could not read the RPM databases of new guests. So we switched to using librpm (instead of direct access), but this meant for a time we couldn't access the RPM DBs of *old* guests. However that was eventually fixed in RPM.
https://github.com/libguestfs/libguestfs/commit/c9ee831affed55abe0f928134cbb...
Rich.
The file "/usr/lib/rpm/rpmrc" absolutely exists in the guest (I booted it up separately, and checked it).
It seems that "internal_list_rpm_applications" [daemon/rpm.ml] calls chroot, and uses a child project -- can that be related to the issue?
Anyway, I think I'll post v2, just saying that I could not actually test it.
Thanks, Laszlo
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 12/15/21 16:51, Richard W.M. Jones wrote:
The bottom line is still that it's librpm's problem now to handle any guest rpmdb, no matter how ancient (well, at least RHEL >= 5). If you have the latest rpm installed and it still doesn't work then it could be a bug in librpm, although I'm pretty sure I tested this case.
The latest "rpm" package in Fedora 34, rpm-4.16.1.3-1.fc34: https://koji.fedoraproject.org/koji/buildinfo?buildID=1726304 built on 22 Mar 2021, does not (cannot) have the fix. The upstream rpm fix is: https://github.com/rpm-software-management/rpm/commit/ea2754b4f14e0a195f4633... which is between tags "rpm-4.17.0-alpha" and "rpm-4.17.0-beta1", and has commit date 2021-06-11. The fix has been backported to RHEL-9 (1965147): https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1824746 but not to Fedora 34. Fedora 34 is supposed to be maintained until 2022-05-17, according to Wikipedia, so this is definitely a Fedora bug. I guess I should update my workstation to Fedora 35 regardless; that should be faster. (Fedora 35 has rpm-4.17.0-1.fc35, so it should have the fix.) Thanks Laszlo
On Thu, Dec 16, 2021 at 09:35:31AM +0100, Laszlo Ersek wrote:
On 12/15/21 16:51, Richard W.M. Jones wrote:
The bottom line is still that it's librpm's problem now to handle any guest rpmdb, no matter how ancient (well, at least RHEL >= 5). If you have the latest rpm installed and it still doesn't work then it could be a bug in librpm, although I'm pretty sure I tested this case.
The latest "rpm" package in Fedora 34, rpm-4.16.1.3-1.fc34:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1726304
built on 22 Mar 2021, does not (cannot) have the fix. The upstream rpm fix is:
https://github.com/rpm-software-management/rpm/commit/ea2754b4f14e0a195f4633...
which is between tags "rpm-4.17.0-alpha" and "rpm-4.17.0-beta1", and has commit date 2021-06-11. The fix has been backported to RHEL-9 (1965147):
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1824746
but not to Fedora 34.
Fedora 34 is supposed to be maintained until 2022-05-17, according to Wikipedia, so this is definitely a Fedora bug. I guess I should update my workstation to Fedora 35 regardless; that should be faster.
(Fedora 35 has rpm-4.17.0-1.fc35, so it should have the fix.)
You might find you can just install the Fedora 35 package on F34, eg: dnf update rpm --releasever=35 Small possibility it could do Bad Stuff to your system -- since it is a rather fundamental package ... Rich.
Thanks Laszlo
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
participants (2)
-
Laszlo Ersek -
Richard W.M. Jones