virt-builder fails in fedora 34 container:
I'm running this inside a fedora 34 container in ovirt-imageio CI: 'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', '--root-password', 'password:', '-o', '/var/tmp/imageio-images/cirros-0.3.5' The command fails in: supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory") libguestfs: trace: launch = -1 (error) virt-builder: error: libguestfs error: /usr/bin/supermin exited with error status 1, see debug messages above For full error log see: https://travis-ci.com/github/nirs/ovirt-imageio/jobs/513305268 The container file is: https://github.com/oVirt/ovirt-imageio/blob/master/docker/Dockerfile.fedora-... The same code works with centos stream 8 container and fedora 32, 33 containers. Based on this error: supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory") I think I'm missing some package removed from the fedora 34 container, but I don't have any idea which one. I hope someone has more insight on this. Nir
On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote:
I'm running this inside a fedora 34 container in ovirt-imageio CI:
'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', '--root-password', 'password:', '-o', '/var/tmp/imageio-images/cirros-0.3.5'
The command fails in:
supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory")
We use this file in order to construct the minimal initrd with some virtio modules, which is sufficient to bring up the supermin root filesystem. The file itself is used to calculate the correct set of module dependencies (so that, for example, if a virtio kmod needs another kmod then the other kmod will be pulled in). See: https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13... I believe in Fedora the file is created via a %post script in kernel-core which runs /bin/kernel-install (from systemd-udev). This in turn is a shell script which runs snippets from /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied from all kinds of other packages. On my machine one of these is /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev) which eventually runs depmod -a which creates this file. So those are some of the packages which might be missing. Does modules.dep get created anywhere inside the container? It's possible it moved. It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1 which should show (in the dnf install output I guess?) if/when the depmod command runs.
I think I'm missing some package removed from the fedora 34 container, but I don't have any idea which one.
I hope someone has more insight on this.
It's curious that things have changed since F33. Could you build containers for both and diff the list of package names? Maybe a dependency is missing or has become a soft dep. rpm -qa --qf '%{name}\n' | sort Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Sun, Jun 13, 2021 at 1:38 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote:
I'm running this inside a fedora 34 container in ovirt-imageio CI:
'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', '--root-password', 'password:', '-o', '/var/tmp/imageio-images/cirros-0.3.5'
The command fails in:
supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory")
We use this file in order to construct the minimal initrd with some virtio modules, which is sufficient to bring up the supermin root filesystem. The file itself is used to calculate the correct set of module dependencies (so that, for example, if a virtio kmod needs another kmod then the other kmod will be pulled in). See:
https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13...
I believe in Fedora the file is created via a %post script in kernel-core which runs /bin/kernel-install (from systemd-udev). This in turn is a shell script which runs snippets from /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied from all kinds of other packages. On my machine one of these is /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev) which eventually runs depmod -a which creates this file.
So those are some of the packages which might be missing. Does modules.dep get created anywhere inside the container? It's possible it moved.
It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1 which should show (in the dnf install output I guess?) if/when the depmod command runs.
I think I'm missing some package removed from the fedora 34 container, but I don't have any idea which one.
I hope someone has more insight on this.
It's curious that things have changed since F33. Could you build containers for both and diff the list of package names? Maybe a dependency is missing or has become a soft dep.
rpm -qa --qf '%{name}\n' | sort
I ran this: $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 rpm -qa --qf '%{name}\n' | sort > fedora-34.out $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 rpm -qa --qf '%{name}\n' | sort > fedora-33.out $ diff -u fedora-33.out fedora-34.out > fedora.diff $ egrep -- ^- fedora.diff > removed-in-fedora-34 $ egrep -- '^[+]' fedora.diff > added-in-fedora-34 $ cat removed-in-fedora-34 --- fedora-33.out 2021-06-13 22:51:13.156150245 +0300 -autogen-libopts -genisoimage -iptables -isl -kbd-legacy -libcbor -libdb-utils -libfido2 -libguestfs-tools-c -libmodman -libusal -nmap-ncat -pciutils -pciutils-libs -python3-idna -rdma-core -selinux-policy-minimum $ cat added-in-fedora-34 +++ fedora-34.out 2021-06-13 22:50:59.650016271 +0300 +guestfs-tools +inih +iptables-legacy +iptables-legacy-libs +langpacks-core-en_GB +langpacks-en_GB +libbsd +libcanberra +libcanberra-gtk3 +libcloudproviders +libpmemobj +libretls +libtdb +netcat +pcsc-lite-libs +selinux-policy-targeted +sound-theme-freedesktop I don't see anything that can explain the difference. I also compared contents of the /lib/modules/xxx/ directories: $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 ls -1 /lib/modules/5.12.9-200.fc33.x86_64 > fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 ls -1 /lib/modules/5.12.9-300.fc34.x86_64 > fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out $ diff -u fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out --- fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out 2021-06-13 23:12:04.177539132 +0300 +++ fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out 2021-06-13 23:12:18.072675988 +0300 @@ -3,23 +3,14 @@ build config kernel -modules.alias -modules.alias.bin modules.block modules.builtin modules.builtin.alias.bin -modules.builtin.bin modules.builtin.modinfo -modules.dep -modules.dep.bin -modules.devname modules.drm modules.modesetting modules.networking modules.order -modules.softdep -modules.symbols -modules.symbols.bin source symvers.gz updates Looks like some post install step is missing on the fedora 34 container.
On Sun, Jun 13, 2021 at 11:15:31PM +0300, Nir Soffer wrote:
On Sun, Jun 13, 2021 at 1:38 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote:
I'm running this inside a fedora 34 container in ovirt-imageio CI:
'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', '--root-password', 'password:', '-o', '/var/tmp/imageio-images/cirros-0.3.5'
The command fails in:
supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory")
We use this file in order to construct the minimal initrd with some virtio modules, which is sufficient to bring up the supermin root filesystem. The file itself is used to calculate the correct set of module dependencies (so that, for example, if a virtio kmod needs another kmod then the other kmod will be pulled in). See:
https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13...
I believe in Fedora the file is created via a %post script in kernel-core which runs /bin/kernel-install (from systemd-udev). This in turn is a shell script which runs snippets from /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied from all kinds of other packages. On my machine one of these is /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev) which eventually runs depmod -a which creates this file.
So those are some of the packages which might be missing. Does modules.dep get created anywhere inside the container? It's possible it moved.
It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1 which should show (in the dnf install output I guess?) if/when the depmod command runs.
I think I'm missing some package removed from the fedora 34 container, but I don't have any idea which one.
I hope someone has more insight on this.
It's curious that things have changed since F33. Could you build containers for both and diff the list of package names? Maybe a dependency is missing or has become a soft dep.
rpm -qa --qf '%{name}\n' | sort
I ran this:
$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 rpm -qa --qf '%{name}\n' | sort > fedora-34.out $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 rpm -qa --qf '%{name}\n' | sort > fedora-33.out $ diff -u fedora-33.out fedora-34.out > fedora.diff $ egrep -- ^- fedora.diff > removed-in-fedora-34 $ egrep -- '^[+]' fedora.diff > added-in-fedora-34
$ cat removed-in-fedora-34 --- fedora-33.out 2021-06-13 22:51:13.156150245 +0300 -autogen-libopts -genisoimage -iptables -isl -kbd-legacy -libcbor -libdb-utils -libfido2 -libguestfs-tools-c -libmodman -libusal -nmap-ncat -pciutils -pciutils-libs -python3-idna -rdma-core -selinux-policy-minimum
$ cat added-in-fedora-34 +++ fedora-34.out 2021-06-13 22:50:59.650016271 +0300 +guestfs-tools +inih +iptables-legacy +iptables-legacy-libs +langpacks-core-en_GB +langpacks-en_GB +libbsd +libcanberra +libcanberra-gtk3 +libcloudproviders +libpmemobj +libretls +libtdb +netcat +pcsc-lite-libs +selinux-policy-targeted +sound-theme-freedesktop
I don't see anything that can explain the difference.
I also compared contents of the /lib/modules/xxx/ directories:
$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 ls -1 /lib/modules/5.12.9-200.fc33.x86_64 > fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 ls -1 /lib/modules/5.12.9-300.fc34.x86_64 > fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out $ diff -u fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out --- fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out 2021-06-13 23:12:04.177539132 +0300 +++ fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out 2021-06-13 23:12:18.072675988 +0300 @@ -3,23 +3,14 @@ build config kernel -modules.alias -modules.alias.bin modules.block modules.builtin modules.builtin.alias.bin -modules.builtin.bin modules.builtin.modinfo -modules.dep -modules.dep.bin -modules.devname modules.drm modules.modesetting modules.networking modules.order -modules.softdep -modules.symbols -modules.symbols.bin source symvers.gz updates
Looks like some post install step is missing on the fedora 34 container.
Just to be clear, that is on the ovirt/imageio container, not any usual container out there. No upstream fedora container has anything in /lib/modules because it usually is pointless to distribute something that is not going to be needed as the container is not coupled with any kernel because it runs on the host one. I tested upstream fedora:33 and fedora:34 and fedora-minimal:34 (with microdnf) containers and all got modules.dep file from the kernel-(debug-)core package, although it is not owned by the package, so it is clearly installed in a postscript or similar. So it has to be something like not installing the package correctly in the imageio container image. I've got a tiny hunch, so I must ask. Is someone, by any stroke of misfortune, using bazel for building that imageio container image? Have a nice day. Martin
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
On Mon, Jun 14, 2021 at 11:57 AM Martin Kletzander <mkletzan@redhat.com> wrote:
On Sun, Jun 13, 2021 at 11:15:31PM +0300, Nir Soffer wrote:
On Sun, Jun 13, 2021 at 1:38 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote:
I'm running this inside a fedora 34 container in ovirt-imageio CI:
'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', '--root-password', 'password:', '-o', '/var/tmp/imageio-images/cirros-0.3.5'
The command fails in:
supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory")
We use this file in order to construct the minimal initrd with some virtio modules, which is sufficient to bring up the supermin root filesystem. The file itself is used to calculate the correct set of module dependencies (so that, for example, if a virtio kmod needs another kmod then the other kmod will be pulled in). See:
https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13...
I believe in Fedora the file is created via a %post script in kernel-core which runs /bin/kernel-install (from systemd-udev). This in turn is a shell script which runs snippets from /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied from all kinds of other packages. On my machine one of these is /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev) which eventually runs depmod -a which creates this file.
So those are some of the packages which might be missing. Does modules.dep get created anywhere inside the container? It's possible it moved.
It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1 which should show (in the dnf install output I guess?) if/when the depmod command runs.
I think I'm missing some package removed from the fedora 34 container, but I don't have any idea which one.
I hope someone has more insight on this.
It's curious that things have changed since F33. Could you build containers for both and diff the list of package names? Maybe a dependency is missing or has become a soft dep.
rpm -qa --qf '%{name}\n' | sort
I ran this:
$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 rpm -qa --qf '%{name}\n' | sort > fedora-34.out $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 rpm -qa --qf '%{name}\n' | sort > fedora-33.out $ diff -u fedora-33.out fedora-34.out > fedora.diff $ egrep -- ^- fedora.diff > removed-in-fedora-34 $ egrep -- '^[+]' fedora.diff > added-in-fedora-34
$ cat removed-in-fedora-34 --- fedora-33.out 2021-06-13 22:51:13.156150245 +0300 -autogen-libopts -genisoimage -iptables -isl -kbd-legacy -libcbor -libdb-utils -libfido2 -libguestfs-tools-c -libmodman -libusal -nmap-ncat -pciutils -pciutils-libs -python3-idna -rdma-core -selinux-policy-minimum
$ cat added-in-fedora-34 +++ fedora-34.out 2021-06-13 22:50:59.650016271 +0300 +guestfs-tools +inih +iptables-legacy +iptables-legacy-libs +langpacks-core-en_GB +langpacks-en_GB +libbsd +libcanberra +libcanberra-gtk3 +libcloudproviders +libpmemobj +libretls +libtdb +netcat +pcsc-lite-libs +selinux-policy-targeted +sound-theme-freedesktop
I don't see anything that can explain the difference.
I also compared contents of the /lib/modules/xxx/ directories:
$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 ls -1 /lib/modules/5.12.9-200.fc33.x86_64 > fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 ls -1 /lib/modules/5.12.9-300.fc34.x86_64 > fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out $ diff -u fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out --- fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out 2021-06-13 23:12:04.177539132 +0300 +++ fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out 2021-06-13 23:12:18.072675988 +0300 @@ -3,23 +3,14 @@ build config kernel -modules.alias -modules.alias.bin modules.block modules.builtin modules.builtin.alias.bin -modules.builtin.bin modules.builtin.modinfo -modules.dep -modules.dep.bin -modules.devname modules.drm modules.modesetting modules.networking modules.order -modules.softdep -modules.symbols -modules.symbols.bin source symvers.gz updates
Looks like some post install step is missing on the fedora 34 container.
Just to be clear, that is on the ovirt/imageio container, not any usual container out there. No upstream fedora container has anything in /lib/modules because it usually is pointless to distribute something that is not going to be needed as the container is not coupled with any kernel because it runs on the host one.
I think this is up to the package installing virt-builder. We cannot know about virt-builder dependencies and make sure they are met.
I tested upstream fedora:33 and fedora:34 and fedora-minimal:34 (with microdnf) containers and all got modules.dep file from the kernel-(debug-)core package, although it is not owned by the package, so it is clearly installed in a postscript or similar. So it has to be something like not installing the package correctly in the imageio container image.
Thanks, I'll test adding this package. But shouldn't it be a dependency of the package installing virt-builder?
I've got a tiny hunch, so I must ask. Is someone, by any stroke of misfortune, using bazel for building that imageio container image?
We build the images in quay.io, I don't know how they build them :-) Nir
On Mon, Jun 14, 2021 at 02:20:40PM +0300, Nir Soffer wrote:
On Mon, Jun 14, 2021 at 11:57 AM Martin Kletzander <mkletzan@redhat.com> wrote:
I've got a tiny hunch, so I must ask. Is someone, by any stroke of misfortune, using bazel for building that imageio container image?
We build the images in quay.io, I don't know how they build them :-)
The issue that Martin is hinting at is that bazel does not run %post scripts (or create an RPM database). IOW the tools is broken. If you have an RPM database in the final container then it's probably _not_ bazel. If anyone tells you to use bazel, run away! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Mon, Jun 14, 2021 at 01:01:37PM +0100, Richard W.M. Jones wrote:
On Mon, Jun 14, 2021 at 02:20:40PM +0300, Nir Soffer wrote:
On Mon, Jun 14, 2021 at 11:57 AM Martin Kletzander <mkletzan@redhat.com> wrote:
I've got a tiny hunch, so I must ask. Is someone, by any stroke of misfortune, using bazel for building that imageio container image?
We build the images in quay.io, I don't know how they build them :-)
The issue that Martin is hinting at is that bazel does not run %post scripts (or create an RPM database). IOW the tools is broken.
If you have an RPM database in the final container then it's probably _not_ bazel. If anyone tells you to use bazel, run away!
+1M Anyway, I can't stop thinking about what would be the reasonable way of fixing the issue at hand properly. It seems like a huge deal to depend on kernel-core, but if you need a kernel and modules during runtime, then I guess it makes sense.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Mon, Jun 14, 2021 at 7:47 PM Martin Kletzander <mkletzan@redhat.com> wrote:
On Mon, Jun 14, 2021 at 01:01:37PM +0100, Richard W.M. Jones wrote:
On Mon, Jun 14, 2021 at 02:20:40PM +0300, Nir Soffer wrote:
On Mon, Jun 14, 2021 at 11:57 AM Martin Kletzander <mkletzan@redhat.com> wrote:
I've got a tiny hunch, so I must ask. Is someone, by any stroke of misfortune, using bazel for building that imageio container image?
We build the images in quay.io, I don't know how they build them :-)
The issue that Martin is hinting at is that bazel does not run %post scripts (or create an RPM database). IOW the tools is broken.
If you have an RPM database in the final container then it's probably _not_ bazel. If anyone tells you to use bazel, run away!
+1M
Anyway, I can't stop thinking about what would be the reasonable way of fixing the issue at hand properly. It seems like a huge deal to depend on kernel-core, but if you need a kernel and modules during runtime, then I guess it makes sense.
I tried installing kernel-core package: $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 # dnf install kernel-core ... Package kernel-core-5.12.9-300.fc34.x86_64 is already installed. So I reinstalled it: # dnf reinstall kernel-core ... And after that modules.dep was created: # ls /lib/modules/5.12.9-300.fc34.x86_64/ System.map modules.alias modules.builtin.bin modules.drm modules.symbols vdso bls.conf modules.alias.bin modules.builtin.modinfo modules.modesetting modules.symbols.bin vmlinuz build modules.block modules.dep modules.networking source weak-updates config modules.builtin modules.dep.bin modules.order symvers.gz kernel modules.builtin.alias.bin modules.devname modules.softdep updates And virt-builder is working: # LIBGUESTFS_BACKEND=direct virt-builder cirros-0.3.5 -o test.img [ 2.2] Downloading: http://builder.libguestfs.org/cirros-0.3.5.xz [ 2.2] Planning how to build this image [ 2.2] Uncompressing [ 2.6] Opening the new disk [ 26.6] Setting a random seed [ 26.9] Setting passwords virt-builder: warning: password: using insecure md5 password encryption for guest of type cirros version 0.3. If this is incorrect, use --password-crypto option and file a bug. virt-builder: Setting random password of root to mbeNkcd6DVbSWmNn [ 31.9] Finishing off Output file: test.img Output size: 39.2M Output format: raw Total usable space: 22.2M Free space: 5.3M (23%) So kernel-core was already installed, but for some reason its scriptlet did not create modules.dep when the container was built. I looked at the quay.io logs creating this container: https://quay.io/repository/ovirt/imageio-test-fedora-34/build/08d6965f-9d8f-... And we do see the scriptlet running: 6/8/2021, 1:30:44 PMInstalling : kernel-core-5.12.9-300.fc34.x86_64 254/373 6/8/2021, 1:30:44 PMRunning scriptlet: kernel-core-5.12.9-300.fc34.x86_64 254/373 ... 6/8/2021, 1:31:05 PMRunning scriptlet: kernel-core-5.12.9-300.fc34.x86_64 373/373 So maybe this is an issue with the install scriptlet when building in quay.io When I build the container locally on Fedora 32, modules.dep is created and virt-builder works. But I want to build in quay.io, it is much more convenient. Nir
On Mon, Jun 14, 2021 at 11:50 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Jun 14, 2021 at 7:47 PM Martin Kletzander <mkletzan@redhat.com> wrote:
On Mon, Jun 14, 2021 at 01:01:37PM +0100, Richard W.M. Jones wrote:
On Mon, Jun 14, 2021 at 02:20:40PM +0300, Nir Soffer wrote:
On Mon, Jun 14, 2021 at 11:57 AM Martin Kletzander <mkletzan@redhat.com> wrote:
I've got a tiny hunch, so I must ask. Is someone, by any stroke of misfortune, using bazel for building that imageio container image?
We build the images in quay.io, I don't know how they build them :-)
The issue that Martin is hinting at is that bazel does not run %post scripts (or create an RPM database). IOW the tools is broken.
If you have an RPM database in the final container then it's probably _not_ bazel. If anyone tells you to use bazel, run away!
+1M
Anyway, I can't stop thinking about what would be the reasonable way of fixing the issue at hand properly. It seems like a huge deal to depend on kernel-core, but if you need a kernel and modules during runtime, then I guess it makes sense.
I tried installing kernel-core package:
$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 # dnf install kernel-core ... Package kernel-core-5.12.9-300.fc34.x86_64 is already installed.
So I reinstalled it: # dnf reinstall kernel-core ...
And after that modules.dep was created:
# ls /lib/modules/5.12.9-300.fc34.x86_64/ System.map modules.alias modules.builtin.bin modules.drm modules.symbols vdso bls.conf modules.alias.bin modules.builtin.modinfo modules.modesetting modules.symbols.bin vmlinuz build modules.block modules.dep modules.networking source weak-updates config modules.builtin modules.dep.bin modules.order symvers.gz kernel modules.builtin.alias.bin modules.devname modules.softdep updates
And virt-builder is working:
# LIBGUESTFS_BACKEND=direct virt-builder cirros-0.3.5 -o test.img [ 2.2] Downloading: http://builder.libguestfs.org/cirros-0.3.5.xz [ 2.2] Planning how to build this image [ 2.2] Uncompressing [ 2.6] Opening the new disk [ 26.6] Setting a random seed [ 26.9] Setting passwords virt-builder: warning: password: using insecure md5 password encryption for guest of type cirros version 0.3. If this is incorrect, use --password-crypto option and file a bug. virt-builder: Setting random password of root to mbeNkcd6DVbSWmNn [ 31.9] Finishing off Output file: test.img Output size: 39.2M Output format: raw Total usable space: 22.2M Free space: 5.3M (23%)
So kernel-core was already installed, but for some reason its scriptlet did not create modules.dep when the container was built.
I looked at the quay.io logs creating this container: https://quay.io/repository/ovirt/imageio-test-fedora-34/build/08d6965f-9d8f-...
And we do see the scriptlet running:
6/8/2021, 1:30:44 PMInstalling : kernel-core-5.12.9-300.fc34.x86_64 254/373 6/8/2021, 1:30:44 PMRunning scriptlet: kernel-core-5.12.9-300.fc34.x86_64 254/373 ... 6/8/2021, 1:31:05 PMRunning scriptlet: kernel-core-5.12.9-300.fc34.x86_64 373/373
So maybe this is an issue with the install scriptlet when building in quay.io
When I build the container locally on Fedora 32, modules.dep is created and virt-builder works. But I want to build in quay.io, it is much more convenient.
Fixed also by running depmod before running virt-builder: # depmod -a 5.12.9-300.fc34.x86_64
On Tue, Jun 15, 2021 at 12:15:35AM +0300, Nir Soffer wrote:
On Mon, Jun 14, 2021 at 11:50 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Jun 14, 2021 at 7:47 PM Martin Kletzander <mkletzan@redhat.com> wrote:
On Mon, Jun 14, 2021 at 01:01:37PM +0100, Richard W.M. Jones wrote:
On Mon, Jun 14, 2021 at 02:20:40PM +0300, Nir Soffer wrote:
On Mon, Jun 14, 2021 at 11:57 AM Martin Kletzander <mkletzan@redhat.com> wrote:
I've got a tiny hunch, so I must ask. Is someone, by any stroke of misfortune, using bazel for building that imageio container image?
We build the images in quay.io, I don't know how they build them :-)
The issue that Martin is hinting at is that bazel does not run %post scripts (or create an RPM database). IOW the tools is broken.
If you have an RPM database in the final container then it's probably _not_ bazel. If anyone tells you to use bazel, run away!
+1M
Anyway, I can't stop thinking about what would be the reasonable way of fixing the issue at hand properly. It seems like a huge deal to depend on kernel-core, but if you need a kernel and modules during runtime, then I guess it makes sense.
I tried installing kernel-core package:
$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 # dnf install kernel-core ... Package kernel-core-5.12.9-300.fc34.x86_64 is already installed.
So I reinstalled it: # dnf reinstall kernel-core ...
And after that modules.dep was created:
# ls /lib/modules/5.12.9-300.fc34.x86_64/ System.map modules.alias modules.builtin.bin modules.drm modules.symbols vdso bls.conf modules.alias.bin modules.builtin.modinfo modules.modesetting modules.symbols.bin vmlinuz build modules.block modules.dep modules.networking source weak-updates config modules.builtin modules.dep.bin modules.order symvers.gz kernel modules.builtin.alias.bin modules.devname modules.softdep updates
And virt-builder is working:
# LIBGUESTFS_BACKEND=direct virt-builder cirros-0.3.5 -o test.img [ 2.2] Downloading: http://builder.libguestfs.org/cirros-0.3.5.xz [ 2.2] Planning how to build this image [ 2.2] Uncompressing [ 2.6] Opening the new disk [ 26.6] Setting a random seed [ 26.9] Setting passwords virt-builder: warning: password: using insecure md5 password encryption for guest of type cirros version 0.3. If this is incorrect, use --password-crypto option and file a bug. virt-builder: Setting random password of root to mbeNkcd6DVbSWmNn [ 31.9] Finishing off Output file: test.img Output size: 39.2M Output format: raw Total usable space: 22.2M Free space: 5.3M (23%)
So kernel-core was already installed, but for some reason its scriptlet did not create modules.dep when the container was built.
I looked at the quay.io logs creating this container: https://quay.io/repository/ovirt/imageio-test-fedora-34/build/08d6965f-9d8f-...
And we do see the scriptlet running:
6/8/2021, 1:30:44 PMInstalling : kernel-core-5.12.9-300.fc34.x86_64 254/373 6/8/2021, 1:30:44 PMRunning scriptlet: kernel-core-5.12.9-300.fc34.x86_64 254/373 ... 6/8/2021, 1:31:05 PMRunning scriptlet: kernel-core-5.12.9-300.fc34.x86_64 373/373
So maybe this is an issue with the install scriptlet when building in quay.io
When I build the container locally on Fedora 32, modules.dep is created and virt-builder works. But I want to build in quay.io, it is much more convenient.
Fixed also by running depmod before running virt-builder:
# depmod -a 5.12.9-300.fc34.x86_64
Maybe they are overusing multi-stage builds or something. It is difficult to figure out as I do not have access to those logs (even though it lets me "cancel the build" - yes, the old and finished one) and I do not know from which Containerfile it was built. I looked at GitHub and saw this: https://github.com/oVirt/ovirt-imageio/tree/d163d80f04e5ae64f8e0138463aae5e8... if that is where the images are built from, then it clearly is an issue in quay because a) there is nothing that suggests a missing file and b) I tried it locally and it works.
On Tue, Jun 15, 2021 at 1:26 AM Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, Jun 15, 2021 at 12:15:35AM +0300, Nir Soffer wrote:
On Mon, Jun 14, 2021 at 11:50 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Jun 14, 2021 at 7:47 PM Martin Kletzander <mkletzan@redhat.com> wrote:
On Mon, Jun 14, 2021 at 01:01:37PM +0100, Richard W.M. Jones wrote:
On Mon, Jun 14, 2021 at 02:20:40PM +0300, Nir Soffer wrote:
On Mon, Jun 14, 2021 at 11:57 AM Martin Kletzander <mkletzan@redhat.com> wrote:
> I've got a tiny hunch, so I must ask. Is someone, by any stroke > of misfortune, using bazel for building that imageio container > image?
We build the images in quay.io, I don't know how they build them :-)
The issue that Martin is hinting at is that bazel does not run %post scripts (or create an RPM database). IOW the tools is broken.
If you have an RPM database in the final container then it's probably _not_ bazel. If anyone tells you to use bazel, run away!
+1M
Anyway, I can't stop thinking about what would be the reasonable way of fixing the issue at hand properly. It seems like a huge deal to depend on kernel-core, but if you need a kernel and modules during runtime, then I guess it makes sense.
I tried installing kernel-core package:
$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 # dnf install kernel-core ... Package kernel-core-5.12.9-300.fc34.x86_64 is already installed.
So I reinstalled it: # dnf reinstall kernel-core ...
And after that modules.dep was created:
# ls /lib/modules/5.12.9-300.fc34.x86_64/ System.map modules.alias modules.builtin.bin modules.drm modules.symbols vdso bls.conf modules.alias.bin modules.builtin.modinfo modules.modesetting modules.symbols.bin vmlinuz build modules.block modules.dep modules.networking source weak-updates config modules.builtin modules.dep.bin modules.order symvers.gz kernel modules.builtin.alias.bin modules.devname modules.softdep updates
And virt-builder is working:
# LIBGUESTFS_BACKEND=direct virt-builder cirros-0.3.5 -o test.img [ 2.2] Downloading: http://builder.libguestfs.org/cirros-0.3.5.xz [ 2.2] Planning how to build this image [ 2.2] Uncompressing [ 2.6] Opening the new disk [ 26.6] Setting a random seed [ 26.9] Setting passwords virt-builder: warning: password: using insecure md5 password encryption for guest of type cirros version 0.3. If this is incorrect, use --password-crypto option and file a bug. virt-builder: Setting random password of root to mbeNkcd6DVbSWmNn [ 31.9] Finishing off Output file: test.img Output size: 39.2M Output format: raw Total usable space: 22.2M Free space: 5.3M (23%)
So kernel-core was already installed, but for some reason its scriptlet did not create modules.dep when the container was built.
I looked at the quay.io logs creating this container: https://quay.io/repository/ovirt/imageio-test-fedora-34/build/08d6965f-9d8f-...
And we do see the scriptlet running:
6/8/2021, 1:30:44 PMInstalling : kernel-core-5.12.9-300.fc34.x86_64 254/373 6/8/2021, 1:30:44 PMRunning scriptlet: kernel-core-5.12.9-300.fc34.x86_64 254/373 ... 6/8/2021, 1:31:05 PMRunning scriptlet: kernel-core-5.12.9-300.fc34.x86_64 373/373
So maybe this is an issue with the install scriptlet when building in quay.io
When I build the container locally on Fedora 32, modules.dep is created and virt-builder works. But I want to build in quay.io, it is much more convenient.
Fixed also by running depmod before running virt-builder:
# depmod -a 5.12.9-300.fc34.x86_64
Maybe they are overusing multi-stage builds or something. It is difficult to figure out as I do not have access to those logs (even though it lets me "cancel the build" - yes, the old and finished one) and I do not know from which Containerfile it was built.
I looked at GitHub and saw this:
https://github.com/oVirt/ovirt-imageio/tree/d163d80f04e5ae64f8e0138463aae5e8...
if that is where the images are built from,
Correct
then it clearly is an issue in quay because a) there is nothing that suggests a missing file and b) I tried it locally and it works.
Thanks for looking. Attached the secret build logs if you want to look at them.
On Tue, Jun 15, 2021 at 02:12:00AM +0300, Nir Soffer wrote:
On Tue, Jun 15, 2021 at 1:26 AM Martin Kletzander <mkletzan@redhat.com> wrote:
[...]
I looked at GitHub and saw this:
https://github.com/oVirt/ovirt-imageio/tree/d163d80f04e5ae64f8e0138463aae5e8...
if that is where the images are built from,
Correct
then it clearly is an issue in quay because a) there is nothing that suggests a missing file and b) I tried it locally and it works.
Thanks for looking.
Attached the secret build logs if you want to look at them.
Wow, there really is nothing that would suggest anything non-standard in that log. Maybe this is a one time thing and some older releases will not do it? Maybe next time it is built it's going to be fine? If not, then I must say I am baffled about how to figure this one out. Contacting someone related to quay maybe... Have a nice day, Martin
On Tue, Jun 15, 2021 at 7:35 AM Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, Jun 15, 2021 at 02:12:00AM +0300, Nir Soffer wrote:
On Tue, Jun 15, 2021 at 1:26 AM Martin Kletzander <mkletzan@redhat.com> wrote:
[...]
I looked at GitHub and saw this:
https://github.com/oVirt/ovirt-imageio/tree/d163d80f04e5ae64f8e0138463aae5e8...
if that is where the images are built from,
Correct
then it clearly is an issue in quay because a) there is nothing that suggests a missing file and b) I tried it locally and it works.
Thanks for looking.
Attached the secret build logs if you want to look at them.
Wow, there really is nothing that would suggest anything non-standard in that log. Maybe this is a one time thing and some older releases will not do it? Maybe next time it is built it's going to be fine? If not, then I must say I am baffled about how to figure this one out. Contacting someone related to quay maybe...
It is reproducible in every build. Here is a minimal reproducer: https://github.com/nirs/quay-io-bug/blob/master/Dockerfile https://quay.io/repository/nirsof/quay-bug Works locally, broken when pulling the image from quay.io.
On Tue, Jun 15, 2021 at 1:28 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Jun 15, 2021 at 7:35 AM Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, Jun 15, 2021 at 02:12:00AM +0300, Nir Soffer wrote:
On Tue, Jun 15, 2021 at 1:26 AM Martin Kletzander <mkletzan@redhat.com> wrote:
[...]
I looked at GitHub and saw this:
https://github.com/oVirt/ovirt-imageio/tree/d163d80f04e5ae64f8e0138463aae5e8...
if that is where the images are built from,
Correct
then it clearly is an issue in quay because a) there is nothing that suggests a missing file and b) I tried it locally and it works.
Thanks for looking.
Attached the secret build logs if you want to look at them.
Wow, there really is nothing that would suggest anything non-standard in that log. Maybe this is a one time thing and some older releases will not do it? Maybe next time it is built it's going to be fine? If not, then I must say I am baffled about how to figure this one out. Contacting someone related to quay maybe...
It is reproducible in every build. Here is a minimal reproducer:
https://github.com/nirs/quay-io-bug/blob/master/Dockerfile https://quay.io/repository/nirsof/quay-bug
Works locally, broken when pulling the image from quay.io.
I filed quay bug: https://issues.redhat.com/browse/PROJQUAY-2130
On Sun, Jun 13, 2021 at 1:38 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote:
I'm running this inside a fedora 34 container in ovirt-imageio CI:
'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', '--root-password', 'password:', '-o', '/var/tmp/imageio-images/cirros-0.3.5'
The command fails in:
supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory")
We use this file in order to construct the minimal initrd with some virtio modules, which is sufficient to bring up the supermin root filesystem. The file itself is used to calculate the correct set of module dependencies (so that, for example, if a virtio kmod needs another kmod then the other kmod will be pulled in). See:
Is it correct to use /lib/modules/kernel-X/modules.dep when the file was created on the system building the container while running the container on kernel-Y? For example, we build the container in quay.io (no idea which kernel) and run the container in travis (Ubuntu xenial, kernel 5.4), or locally (Fedora 32, kernel 5.11). Should virt builder/supermin use "demod --show" to generate a temporary modules.dep, instead of depending on system modules.dep when running in a container?
https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13...
I believe in Fedora the file is created via a %post script in kernel-core which runs /bin/kernel-install (from systemd-udev). This in turn is a shell script which runs snippets from /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied from all kinds of other packages. On my machine one of these is /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev) which eventually runs depmod -a which creates this file.
So those are some of the packages which might be missing. Does modules.dep get created anywhere inside the container? It's possible it moved.
It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1 which should show (in the dnf install output I guess?) if/when the depmod command runs.
I think I'm missing some package removed from the fedora 34 container, but I don't have any idea which one.
I hope someone has more insight on this.
It's curious that things have changed since F33. Could you build containers for both and diff the list of package names? Maybe a dependency is missing or has become a soft dep.
rpm -qa --qf '%{name}\n' | sort
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Tue, Jun 15, 2021 at 03:22:23PM +0300, Nir Soffer wrote:
On Sun, Jun 13, 2021 at 1:38 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote:
I'm running this inside a fedora 34 container in ovirt-imageio CI:
'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', '--root-password', 'password:', '-o', '/var/tmp/imageio-images/cirros-0.3.5'
The command fails in:
supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' supermin: exception: Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such file or directory")
We use this file in order to construct the minimal initrd with some virtio modules, which is sufficient to bring up the supermin root filesystem. The file itself is used to calculate the correct set of module dependencies (so that, for example, if a virtio kmod needs another kmod then the other kmod will be pulled in). See:
Is it correct to use /lib/modules/kernel-X/modules.dep when the file was created on the system building the container while running the container on kernel-Y?
Yes, because libguestfs runs the kernel from /boot + /lib/modules on top of qemu. It doesn't use the host kernel.
For example, we build the container in quay.io (no idea which kernel) and run the container in travis (Ubuntu xenial, kernel 5.4), or locally (Fedora 32, kernel 5.11).
Should virt builder/supermin use "demod --show" to generate a temporary modules.dep, instead of depending on system modules.dep when running in a container?
It's not clear to me where depmod --show gets its information from. The host kernel? Anyway, possibly ... but correctly built guests have modules.dep so it's not been a problem until now. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
participants (3)
-
Martin Kletzander -
Nir Soffer -
Richard W.M. Jones