Re: Can't boot UEFI VM after migration: "error: can't find command `linux16`"
by Srihari Parimi
Hi Richard,
This is a more comprehensive description of the problem, where I respond to
the specific issue and questions that you brought up during the discussion
last week. I also cover the proposed solution towards the end. Kindly bear
with the somewhat longish one
- *[Comment 1] - Specific behaviors of the GRUB bootloader when
encountering "Linux 16" versus "Linux EFI" directives.*
- linux16 - Directive in GRUB boot loader to load the Linux Kernel
image. It is documented in the GNU GRUB documentation - linux16
<https://doc.guix.gnu.org/grub/latest/en/grub.html#linux16>
- The linux16 command forces GRUB load Linux Kernel image by using
legacy *16-bit real-mode boot protocol*.
- GRUB hands control to the kernel *while the CPU is still in
16-bit Real Mode*.
- This allows the kernel's real-mode setup code to perform early
hardware initialization using BIOS services (e.g., querying
video modes,
memory map)"
- The Linux Kernel switches itself into 32-bit protected mode.
- initrd16 loads the initramfs and is paired with linux16
- linuxefi - downstream GRUB2 bootloader directives specific to RHEL
7 for UEFI systems. Documented in RHEL 7 System Administrator Guide
<https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/...>
- Directive specific to RHEL 7. These directives are deprecated in
RHEL 8 upwards which use standard GRUB directives linux, initrd
instead and these work uniformly across BIOS or UEFI systems.
- linuxefi and initrdefi directives use UEFI boot services to load
the Linux Kernel image and initramfs respectively.
- Based on the boot mode of the system
- These GRUB configuration files are created using scripts in
/etc/grub.d/ - primary script which generates the kernel image,
initramfs boot loader commands is /etc/grub.d/10_linux.
- RHEL 7.9 VM in BIOS Mode - GRUB reads the file
/boot/grub2/grub.cfg which has the directives {linux16,
initrd16}
[root@localhost ~]# cat /boot/grub2/grub.cfg | egrep
"linux16|initrd16"
*linux16* /vmlinuz-3.10.0-1160.el7.x86_64
root=UUID=b635e330-0173-4866-9e5e-693086760507
ro crashkernel=auto rhgb quiet LANG=en_US.UTF-8
*initrd16* /initramfs-3.10.0-1160.el7.x86_64.img
*linux16* /vmlinuz-0-rescue-bb246b817b814a44b5e960404fdc7ae3
root=UUID=b635e330-0173-4866
-9e5e-693086760507 ro crashkernel=auto rhgb quiet
*initrd16*
/initramfs-0-rescue-bb246b817b814a44b5e960404fdc7ae3.img
- RHEL 7.9 VM in UEFI Mode - GRUB reads the file
/boot/efi/EFI/redhat/grub.cfg which has the directives {linux16,
initrd16}
[root@localhost ~]# cat /boot/efi/EFI/redhat/grub.cfg | egrep
"linuxefi|initrdefi"
*linuxefi* /vmlinuz-3.10.0-1160.el7.x86_64
root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root
rd.lvm.lv=rhel/swap rhgb quiet
*initrdefi* /initramfs-3.10.0-1160.el7.x86_64.img
*linuxefi* /vmlinuz-0-rescue-6e4e30972e924630b169d78059b932d2
root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root
rd.lvm.lv=rhel/swap rhgb quiet
*initrdefi* /initramfs-0-rescue-6e4e30972e924630b169d78059b932d2.img
- Code snippet from /etc/grub.d/10_linux for RHEL9 and RHEL7. The root
device UUID passed to the kernel (e.g., root=UUID=...) is dynamically
queried and formatted by helper routines in
/usr/share/grub/grub-mkconfig_lib using grub-probe .
- Code snippet from */etc/grub.d/10_linux in RHEL9*
if [ -d /sys/firmware/efi ]; then
bootefi_device="`${grub_probe} --target=device /boot/efi/`"
prepare_grub_to_access_device_with_variable boot ${bootefi_device}
else
boot_device="`${grub_probe} --target=device /boot/`"
prepare_grub_to_access_device_with_variable boot ${boot_device}
fi
- Code snippet from */etc/grub.d/10_linux in RHEL7*
if [ -d /sys/firmware/efi ]; then
sed "s/^/$submenu_indentation/" << EOF
${linuxefi} ${rel_dirname}/${basename}
root=${linux_root_device_thisversion} ro ${args}
EOF
else
sed "s/^/$submenu_indentation/" << EOF
linux${sixteenbit} ${rel_dirname}/${basename}
root=${linux_root_device_thisversion} ro ${args}
EOF
fi
- *[Comment 2] Confirming that the virtual machine Linux file
format is fully compatible with the new directive*
- Based on the analysis done for the previous comment, we can
safely conclude that this problem is very specific to RHEL 7 only.
Further, the problem occurs when following conditions are met
- Source VM must be in UEFI boot mode
- In the /etc/fstab, the /boot and /boot/efi partitions must
be identified by the corresponding /dev/sd* files instead of the UUIDs
(default) generated during FS creation.
- The second condition forces virt-v2v conversion to configure GRUB.
- As the grub configuration executes in context of
libguestfs appliance (BIOS mode), it is generated with linux16,
initrd16 whereas the VM is configured to boot in UEFI.
- The VM powers up with UEFI firmware, but the linux16
directive switches the CPU into 16-bit real mode and look for BIOS
services (/boot/grub2/i386-pc) which are absent.
- *[Comment 3] Regarding the potential modification of file system
table entries and disk device naming conventions, Richard Jones
insisted that any statements regarding potential disk access problems
must be accompanied by a detailed explanation of the "why" and "what"
behind those risks*
- The kernel device discovery (names /dev/sd*) is
non-deterministic as it is based on order of driver initialization at
the boot time. The order variation is due to asynchronous driver
probe, kernel update or VM configuration changes.
- By "hard-coding" the device names inside /etc/fstab, there
could be a possibility of an unintended device being mounted onto an
incorrect mount point, thus causing the system to fail or corrupt.
- UUIDs (written into FS) are more reliable - where order of
initialization would not matter making it more deterministic.
- *The proposed solution*
- Convert the references to /dev/sd* files to corresponding FS
UUIDs *[Implemented and tested]*
- Following change applicable only if OS is RHEL 7 and Boot mode is UEFI
- Replace {linux16, initrd16} with {linuxefi, initrdefi}
*[Work in progress to change from using sed to using APIs]*
Thanks
Srihari
2 weeks, 3 days
Fwd: Security Vulnerability in nbdkit QCOW2 ZSTD Decompression
by Richard W.M. Jones
----- Forwarded message from Tristan <TristanInSec(a)gmail.com> -----
Date: Mon, 24 Aug 2026 10:33:21 +0200
From: Tristan <TristanInSec(a)gmail.com>
To: rjones(a)redhat.com
Subject: Security Vulnerability in nbdkit QCOW2 ZSTD Decompression
Hello,
I am writing to report a security vulnerability in nbdkit's qcow2dec
filter, confirmed at HEAD (commit 053f5a3).
Summary of findings:
ZSTD decompression heap over-read via incorrect buffer size (CVSS 6.5)
The zstd_compressed_cluster() function sets the ZSTD input buffer
size to cluster_size (uncompressed size) instead of compressed_size
(actual allocation size). The equivalent zlib path correctly uses
compressed_size, confirming this is a copy-paste bug. In worst case
(2MB clusters, 512-byte compressed), this causes a 2MB over-read
past a 512-byte allocation.
The finding detail is attached.
I would appreciate an acknowledgment of receipt.
Thank you and please let me know if you need anything else.
Regards,
Tristan
=================================================================
Heap Over-Read in ZSTD Decompression of QCOW2 Compressed Clusters
=================================================================
CVSS 3.1: 6.5 (AV:L/AC:L/PR:N/UI:R/S:U/C:L/I:N/A:H)
CWE: CWE-125 (Out-of-bounds Read)
Auth: None (malicious QCOW2 file)
Version: nbdkit 1.42.3, commit 053f5a3 (HEAD, 2026-07-21)
File: filters/qcow2dec/qcow2dec.c, zstd_compressed_cluster(), line 712
Root Cause:
The ZSTD input buffer size is set to cluster_size (the uncompressed
cluster size) instead of compressed_size (the actual allocated buffer
size). The equivalent zlib/deflate path correctly uses compressed_size.
ZSTD path (INCORRECT) -- line 710-712:
ZSTD_outBuffer out = { .dst = buf, .size = cluster_size, .pos = 0 };
ZSTD_inBuffer in =
{ .src = compressed_cluster, .size = cluster_size, .pos = 0 };
// ^^^^^^^^^^^^
// BUG: should be compressed_size
zlib/deflate path (CORRECT) -- line 664-665:
strm.next_in = (void *) compressed_cluster;
strm.avail_in = compressed_size;
// ^^^^^^^^^^^^^^^ correct: uses actual buffer size
Buffer allocation -- line 840:
compressed_cluster = malloc(compressed_size);
Data Flow:
1. qcow2dec filter parses QCOW2 image with ZSTD-compressed clusters
2. L2 entry encodes nr_sectors (line 789), minimum 1 sector (512 bytes)
3. compressed_size = nr_sectors * 512 (line 806)
4. malloc(compressed_size) allocates buffer (line 840)
5. ZSTD input buffer declares .size = cluster_size (line 712)
6. ZSTD_decompressStream() may read up to cluster_size bytes
7. Over-read: cluster_size - compressed_size bytes past allocation
Worst Case:
cluster_bits=21 (2 MB clusters), single-sector compressed data:
- compressed_size = 512 bytes
- cluster_size = 2,097,152 bytes
- Over-read: up to 2,096,640 bytes past the 512-byte allocation
Impact:
Crash (DoS) when serving a crafted QCOW2 image via nbdkit. In cloud
and virtualization environments where untrusted disk images are served,
an attacker can trigger this by providing a crafted QCOW2 image with
ZSTD-compressed clusters. Potential heap data disclosure if the
process survives the over-read.
Fix:
Change line 712 from:
{ .src = compressed_cluster, .size = cluster_size, .pos = 0 };
to:
{ .src = compressed_cluster, .size = compressed_size, .pos = 0 };
----- End forwarded message -----
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
1 month
Re: [RFC PATCH guestfs-tools] builder: templates: Add Ubuntu 24.04 cloud image support
by Aravind Sivamani (asivaman)
Hi Richard, hi all, I am following up on this RFC
The implementation is now ready for review in guestfs-tools PR #51:
https://github.com/libguestfs/guestfs-tools/pull/51
It is rebased onto the current master. End-to-end Noble template generation, focused tests, image checks, and native-size BIOS/UEFI boots passed locally. The exact virt-builder ubuntu-24.04 --size 6G flow also completes and boots under UEFI; resized legacy-BIOS boot remains an open item documented in the PR.
The Fedora workflow is waiting for first-time-contributor approval, so CI has not run yet. Richard, when you have a chance, could you please review the pinned, checksum-verified Canonical image approach and approve the workflow so CI can begin?
Thanks,
Aravind
[Cisco Midnight Blue v1]<https://www.cisco.com/>
Aravind Sivamani
Software Engineer
asivaman(a)cisco.com<mailto:asivaman@cisco.com>
Phone: +1 984 999 1776
[cid:a530ddc6-4317-4b32-9b0f-18c68e49e229]<https://www.linkedin.com/in/aravind-sivamani>
Cisco Systems
NSO Packages and Tooling Services
Sveavagen 25
Stockholm
Sweden
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Update Profile<https://www.cisco.com/c/en/us/about/help/login-account-help.html#~profile> - Privacy<https://www.cisco.com/c/en/us/about/legal/privacy-full.html> - Company Registration<http://www.cisco.com/c/en/us/about/legal/terms-sale-software-license-agre...>
________________________________
From: Aravind Sivamani (asivaman)
Sent: Tuesday, July 21, 2026 1:51 AM
To: guestfs(a)lists.libguestfs.org <guestfs(a)lists.libguestfs.org>
Cc: rjones(a)redhat.com <rjones(a)redhat.com>
Subject: [RFC PATCH guestfs-tools] builder: templates: Add Ubuntu 24.04 cloud image support
Hi all,
I’m working on adding Ubuntu 24.04 (Noble) support to the virt-builder
template generator, and I’d appreciate feedback on the proposed approach.
Ubuntu 24.04 no longer publishes the Debian Installer tree used by the
existing virt-install/preseed path. This RFC imports a dated Canonical Noble
cloud image, verifies its pinned SHA-256, converts it to raw, and then reuses
the existing inspection, sysprep, compression, and index-generation stages.
Existing installer-based templates remain unchanged.
I tested this end to end on Ubuntu 24.04. The focused catalog/import tests,
shell syntax, OCaml parsing, XZ integrity, and generated-index checks passed.
The generated image had an empty machine ID, no SSH host keys or stale
cloud-init state, and a locked root account.
I have not run the full repository make check because this checkout lacked
the generated Autotools bootstrap files.
I’d especially appreciate your thoughts on:
- Using a pinned dated image versus SimpleStreams
- Preserving Noble’s native virtual size and cloud-init behavior
- The expected BIOS/UEFI boot validation before publication
Public branch:
https://github.com/aravindsivamani/guestfs-tools/tree/ubuntu-24.04-cloud-...
The patch is attached. Thanks for taking a look!
Best,
Aravind
[Cisco Midnight Blue v1]<https://www.cisco.com/>
Aravind Sivamani
Software Engineer
asivaman(a)cisco.com<mailto:asivaman@cisco.com>
Phone: +1 984 999 1776
[cid:0f519272-8687-4198-bbaf-78420df04f06]<https://www.linkedin.com/in/aravind-sivamani>
Cisco Systems
NSO Packages and Tooling Services
Sveavagen 25
Stockholm
Sweden
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Update Profile<https://www.cisco.com/c/en/us/about/help/login-account-help.html#~profile> - Privacy<https://www.cisco.com/c/en/us/about/legal/privacy-full.html> - Company Registration<http://www.cisco.com/c/en/us/about/legal/terms-sale-software-license-agre...>
1 month, 1 week
