[Libguestfs] Re: Can't boot UEFI VM after migration: "error: can't find command `linux16`"

Hi Richard, This is a more comprehensive description of the problem, where I respond to the specific issue and questions that you brought up during the discussion last week. I also cover the proposed solution towards the end. Kindly bear with the somewhat longish one • [Comment 1]  - Specific behaviors of the GRUB bootloader when encountering "Linux 16" versus "Linux EFI" directives. □ linux16 - Directive in GRUB boot loader to load the Linux Kernel image. It is documented in the GNU GRUB documentation - linux16 ☆ The linux16 command forces GRUB load Linux Kernel image by using legacy 16-bit real-mode boot protocol. ☆ GRUB hands control to the kernel while the CPU is still in 16-bit Real Mode. ☆ This allows the kernel's real-mode setup code to perform early hardware initialization using BIOS services (e.g., querying video modes, memory map)" ☆ The Linux Kernel switches itself into 32-bit protected mode. ☆ initrd16 loads the initramfs and is paired with linux16 □ linuxefi - downstream GRUB2 bootloader directives specific to RHEL 7 for UEFI systems. Documented in RHEL 7 System Administrator Guide ☆ Directive specific to RHEL 7. These directives are deprecated in RHEL 8 upwards which use standard GRUB directives linux, initrd instead and these work uniformly across BIOS or UEFI systems. ☆ linuxefi and initrdefi directives use UEFI boot services to load the Linux Kernel image and initramfs respectively.

□ Based on the boot mode of the system ☆ These GRUB configuration files are created using scripts in /etc/ grub.d/ - primary script which generates the kernel image, initramfs boot loader commands is /etc/grub.d/10_linux. ○ RHEL 7.9 VM in BIOS Mode - GRUB reads the file /boot/grub2/ grub.cfg which has the directives {linux16, initrd16} [root@localhost ~]# cat /boot/grub2/grub.cfg | egrep "linux16|initrd16" linux16 /vmlinuz-3.10.0-1160.el7.x86_64 root=UUID=b635e330-0173-4866-9e5e-693086760507 ro crashkernel=auto rhgb quiet LANG=en_US.UTF-8 initrd16 /initramfs-3.10.0-1160.el7.x86_64.img linux16 /vmlinuz-0-rescue-bb246b817b814a44b5e960404fdc7ae3 root=UUID=b635e330-0173-4866 -9e5e-693086760507 ro crashkernel=auto rhgb quiet initrd16 /initramfs-0-rescue-bb246b817b814a44b5e960404fdc7ae3.img • RHEL 7.9 VM in UEFI Mode - GRUB reads the file /boot/efi/EFI/redhat/ grub.cfg which has the directives {linux16, initrd16}

[root@localhost ~]# cat /boot/efi/EFI/redhat/grub.cfg | egrep "linuxefi|initrdefi" linuxefi /vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet initrdefi /initramfs-3.10.0-1160.el7.x86_64.img linuxefi /vmlinuz-0-rescue-6e4e30972e924630b169d78059b932d2 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet initrdefi /initramfs-0-rescue-6e4e30972e924630b169d78059b932d2.img • Code snippet from /etc/grub.d/10_linux for RHEL9 and RHEL7. The root device UUID passed to the kernel (e.g., root=UUID=...) is dynamically queried and formatted by helper routines in /usr/share/grub/grub-mkconfig_lib using grub-probe . □ Code snippet from /etc/grub.d/10_linux in RHEL9 if [ -d /sys/firmware/efi ]; then bootefi_device="`${grub_probe} --target=device /boot/efi/`" prepare_grub_to_access_device_with_variable boot ${bootefi_device} else boot_device="`${grub_probe} --target=device /boot/`" prepare_grub_to_access_device_with_variable boot ${boot_device} fi • Code snippet from /etc/grub.d/10_linux in RHEL7 if [ -d /sys/firmware/efi ]; then sed "s/^/$submenu_indentation/" << EOF ${linuxefi} ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args} EOF else sed "s/^/$submenu_indentation/" << EOF linux${sixteenbit} ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args} EOF fi

• [Comment 2] Confirming that the virtual machine Linux file format is fully compatible with the new directive

□ Based on the analysis done for the previous comment, we can safely conclude that this problem is very specific to RHEL 7 only. Further, the problem occurs when following conditions are met ☆ Source VM must be in UEFI boot mode ☆ In the /etc/fstab, the /boot and /boot/efi partitions must be identified by the corresponding /dev/sd* files instead of the UUIDs (default) generated during FS creation. □ The second condition forces virt-v2v conversion to configure GRUB. ☆ As the grub configuration executes in context of libguestfs appliance (BIOS mode), it is generated with linux16, initrd16 whereas the VM is configured to boot in UEFI. ☆ The VM powers up with UEFI firmware, but the linux16 directive switches the CPU into 16-bit real mode and look for BIOS services (/boot/grub2/i386-pc) which are absent.