Re: Can't boot UEFI VM after migration: "error: can't find command `linux16`"
by Srihari Parimi
Hi Richard,
This is a more comprehensive description of the problem, where I respond to
the specific issue and questions that you brought up during the discussion
last week. I also cover the proposed solution towards the end. Kindly bear
with the somewhat longish one
- *[Comment 1] - Specific behaviors of the GRUB bootloader when
encountering "Linux 16" versus "Linux EFI" directives.*
- linux16 - Directive in GRUB boot loader to load the Linux Kernel
image. It is documented in the GNU GRUB documentation - linux16
<https://doc.guix.gnu.org/grub/latest/en/grub.html#linux16>
- The linux16 command forces GRUB load Linux Kernel image by using
legacy *16-bit real-mode boot protocol*.
- GRUB hands control to the kernel *while the CPU is still in
16-bit Real Mode*.
- This allows the kernel's real-mode setup code to perform early
hardware initialization using BIOS services (e.g., querying
video modes,
memory map)"
- The Linux Kernel switches itself into 32-bit protected mode.
- initrd16 loads the initramfs and is paired with linux16
- linuxefi - downstream GRUB2 bootloader directives specific to RHEL
7 for UEFI systems. Documented in RHEL 7 System Administrator Guide
<https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/...>
- Directive specific to RHEL 7. These directives are deprecated in
RHEL 8 upwards which use standard GRUB directives linux, initrd
instead and these work uniformly across BIOS or UEFI systems.
- linuxefi and initrdefi directives use UEFI boot services to load
the Linux Kernel image and initramfs respectively.
- Based on the boot mode of the system
- These GRUB configuration files are created using scripts in
/etc/grub.d/ - primary script which generates the kernel image,
initramfs boot loader commands is /etc/grub.d/10_linux.
- RHEL 7.9 VM in BIOS Mode - GRUB reads the file
/boot/grub2/grub.cfg which has the directives {linux16,
initrd16}
[root@localhost ~]# cat /boot/grub2/grub.cfg | egrep
"linux16|initrd16"
*linux16* /vmlinuz-3.10.0-1160.el7.x86_64
root=UUID=b635e330-0173-4866-9e5e-693086760507
ro crashkernel=auto rhgb quiet LANG=en_US.UTF-8
*initrd16* /initramfs-3.10.0-1160.el7.x86_64.img
*linux16* /vmlinuz-0-rescue-bb246b817b814a44b5e960404fdc7ae3
root=UUID=b635e330-0173-4866
-9e5e-693086760507 ro crashkernel=auto rhgb quiet
*initrd16*
/initramfs-0-rescue-bb246b817b814a44b5e960404fdc7ae3.img
- RHEL 7.9 VM in UEFI Mode - GRUB reads the file
/boot/efi/EFI/redhat/grub.cfg which has the directives {linux16,
initrd16}
[root@localhost ~]# cat /boot/efi/EFI/redhat/grub.cfg | egrep
"linuxefi|initrdefi"
*linuxefi* /vmlinuz-3.10.0-1160.el7.x86_64
root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root
rd.lvm.lv=rhel/swap rhgb quiet
*initrdefi* /initramfs-3.10.0-1160.el7.x86_64.img
*linuxefi* /vmlinuz-0-rescue-6e4e30972e924630b169d78059b932d2
root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root
rd.lvm.lv=rhel/swap rhgb quiet
*initrdefi* /initramfs-0-rescue-6e4e30972e924630b169d78059b932d2.img
- Code snippet from /etc/grub.d/10_linux for RHEL9 and RHEL7. The root
device UUID passed to the kernel (e.g., root=UUID=...) is dynamically
queried and formatted by helper routines in
/usr/share/grub/grub-mkconfig_lib using grub-probe .
- Code snippet from */etc/grub.d/10_linux in RHEL9*
if [ -d /sys/firmware/efi ]; then
bootefi_device="`${grub_probe} --target=device /boot/efi/`"
prepare_grub_to_access_device_with_variable boot ${bootefi_device}
else
boot_device="`${grub_probe} --target=device /boot/`"
prepare_grub_to_access_device_with_variable boot ${boot_device}
fi
- Code snippet from */etc/grub.d/10_linux in RHEL7*
if [ -d /sys/firmware/efi ]; then
sed "s/^/$submenu_indentation/" << EOF
${linuxefi} ${rel_dirname}/${basename}
root=${linux_root_device_thisversion} ro ${args}
EOF
else
sed "s/^/$submenu_indentation/" << EOF
linux${sixteenbit} ${rel_dirname}/${basename}
root=${linux_root_device_thisversion} ro ${args}
EOF
fi
- *[Comment 2] Confirming that the virtual machine Linux file
format is fully compatible with the new directive*
- Based on the analysis done for the previous comment, we can
safely conclude that this problem is very specific to RHEL 7 only.
Further, the problem occurs when following conditions are met
- Source VM must be in UEFI boot mode
- In the /etc/fstab, the /boot and /boot/efi partitions must
be identified by the corresponding /dev/sd* files instead of the UUIDs
(default) generated during FS creation.
- The second condition forces virt-v2v conversion to configure GRUB.
- As the grub configuration executes in context of
libguestfs appliance (BIOS mode), it is generated with linux16,
initrd16 whereas the VM is configured to boot in UEFI.
- The VM powers up with UEFI firmware, but the linux16
directive switches the CPU into 16-bit real mode and look for BIOS
services (/boot/grub2/i386-pc) which are absent.
- *[Comment 3] Regarding the potential modification of file system
table entries and disk device naming conventions, Richard Jones
insisted that any statements regarding potential disk access problems
must be accompanied by a detailed explanation of the "why" and "what"
behind those risks*
- The kernel device discovery (names /dev/sd*) is
non-deterministic as it is based on order of driver initialization at
the boot time. The order variation is due to asynchronous driver
probe, kernel update or VM configuration changes.
- By "hard-coding" the device names inside /etc/fstab, there
could be a possibility of an unintended device being mounted onto an
incorrect mount point, thus causing the system to fail or corrupt.
- UUIDs (written into FS) are more reliable - where order of
initialization would not matter making it more deterministic.
- *The proposed solution*
- Convert the references to /dev/sd* files to corresponding FS
UUIDs *[Implemented and tested]*
- Following change applicable only if OS is RHEL 7 and Boot mode is UEFI
- Replace {linux16, initrd16} with {linuxefi, initrdefi}
*[Work in progress to change from using sed to using APIs]*
Thanks
Srihari
6 days, 20 hours
- 3
- 5
- ← Newer
- 1
- Older →