7:59 a.m.
Hi,
I am getting a weird libguestfs failure from time to time:
2020-08-27 08:48:40 T libguestfs - 0 - appliance - SeaBIOS (version 1.12.0-1)
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Booting from ROM...
2020-08-27 08:48:40 T libguestfs - 0 - library - child_cleanup:
0xd5a840: child process died
Let me elaborate more about my environment
I am running libguestfs 1.42.0 on GCP - that's nested virtualization.
Using python bindings.
Inside a docker.
I wrote an output hook in python to emit logs more nicely.
The supermin image is debian 10.
The failure seems to happen from time to time just as I launch 16
"libguestfs"s on my 16 core machine.
(This is a testing framework, so we are using libguestfs here to
inspect and change different workloads, very similar to virt-v2v)
Here is the full log of the sporadic failure:
2020-08-27 08:48:38 T libguestfs - 0 - enter - set_memsize
2020-08-27 08:48:38 T libguestfs - 0 - enter - add_drive
2020-08-27 08:48:38 T libguestfs - 0 - enter - launch
2020-08-27 08:48:38 T libguestfs - 0 - enter - max_disks
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_tmpdir
2020-08-27 08:48:38 T libguestfs - 0 - enter - version
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_backend
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: program=my-v2v
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: version=1.42.0
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend
registered: unix
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend registered: uml
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend
registered: libvirt
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend
registered: direct
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend=direct
2020-08-27 08:48:38 T libguestfs - 0 - library - launch:
tmpdir=/tmp/libguestfszNoLuV
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: umask=0022
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: euid=0
2020-08-27 08:48:38 T libguestfs - 0 - library - begin testing qemu features
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_cachedir
2020-08-27 08:48:38 T libguestfs - 0 - library - checking for
previously cached test results of /usr/bin/kvm, in /var/tmp/.guestfs-0
2020-08-27 08:48:38 T libguestfs - 0 - library - loading previously
cached test results
2020-08-27 08:48:38 T libguestfs - 0 - library - qemu version: 3.1
2020-08-27 08:48:38 T libguestfs - 0 - library - qemu mandatory locking: yes
2020-08-27 08:48:38 T libguestfs - 0 - library - qemu KVM: disabled
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_backend_setting
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_sockdir
2020-08-27 08:48:38 T libguestfs - 0 - library - finished testing qemu features
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_backend_setting
2020-08-27 08:48:38 T libguestfs - 0 - library - command: run: dmesg |
grep -Eoh 'lpj=[[:digit:]]+'
2020-08-27 08:48:38 T libguestfs - 0 - library - read_lpj_from_dmesg:
calculated lpj=8800000
2020-08-27 08:48:38 T libguestfs - 0 - library - command: run: file
2020-08-27 08:48:38 T libguestfs - 0 - library - command: run: \ --
/appliance/root
2020-08-27 08:48:38 T libguestfs - 0 - appliance - /usr/bin/kvm \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -global
virtio-blk-pci.scsi=off \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -no-user-config \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -enable-fips \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -nodefaults \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -display none \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -machine accel=kvm:tcg \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -m 2048 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -no-reboot \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -rtc driftfix=slew \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -no-hpet \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -global
kvm-pit.lost_tick_policy=discard \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -kernel
/appliance/kernel \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -initrd
/appliance/initrd \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -object
rng-random,filename=/dev/urandom,id=rng0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
virtio-rng-pci,rng=rng0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
virtio-scsi-pci,id=scsi \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -drive
file=/tmp/xww-a1dtzg8i.qcow2,cache=writeback,format=qcow2,id=hd0,if=none
\
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
scsi-hd,drive=hd0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -drive
file=/appliance/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw
\
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
scsi-hd,drive=appliance \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
virtio-serial-pci \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -serial stdio \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device sga \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -chardev
socket,path=/tmp/libguestfsISN6Xx/guestfsd.sock,id=channel0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -append
"panic=1 console=ttyS0 edd=off udevtimeout=6000
udev.event-timeout=6000 no_timer_check lpj=8800000 printk.time=1
cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable
8250.nr_uarts=1 root=UUID=dd6872eb-d621-499d-be42-64d78754c255
selinux=0 guestfs_verbose=1 TERM=linux vsyscall=emulate"
2020-08-27 08:48:40 T libguestfs - 0 - library - responding to serial
console Device Status Report
2020-08-27 08:48:40 T libguestfs - 0 - appliance - ^[[1;256r^[[256;256H^[[6n
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Google, Inc.
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Serial Graphics
Adapter 03/05/17
2020-08-27 08:48:40 T libguestfs - 0 - appliance - SGABIOS $Id:
sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (generic@generic) Sun, 05
Mar 2017 16:09:17 +0100
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Term: 80x24
2020-08-27 08:48:40 T libguestfs - 0 - appliance - 4 0
2020-08-27 08:48:40 T libguestfs - 0 - appliance - ^[[2J
2020-08-27 08:48:40 T libguestfs - 0 - appliance - SeaBIOS (version 1.12.0-1)
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Booting from ROM...
2020-08-27 08:48:40 T libguestfs - 0 - library - child_cleanup:
0xd5a840: child process died
2020-08-27 08:48:40 T libguestfs - 0 - library - sending SIGTERM to process 459
2020-08-27 08:48:40 T libguestfs - 0 - library - qemu maxrss 60136K
2020-08-27 08:48:40 T libguestfs - 0 - subprocess_quit
2020-08-27 08:48:40 T libguestfs - 0 - library - closing guestfs
handle 0xd5a840 (state 0)
2020-08-27 08:48:40 T libguestfs - 0 - close
2020-08-27 08:48:40 T libguestfs - 0 - library - command: run: rm
2020-08-27 08:48:40 T libguestfs - 0 - library - command: run: \ -rf
/tmp/libguestfszNoLuV
2020-08-27 08:48:40 T libguestfs - 0 - library - command: run: rm
2020-08-27 08:48:40 T libguestfs - 0 - library - command: run: \ -rf
/tmp/libguestfsISN6Xx
Usually everything is working and the output is:
2020-08-27 07:41:10 T libguestfs - 0 - appliance - SeaBIOS (version 1.12.0-1)
2020-08-27 07:41:11 T libguestfs - 0 - appliance - Booting from ROM...
2020-08-27 07:41:12 T libguestfs - 0 - appliance - ^[[2J[ 0.000000]
Linux version 4.19.0-10-amd64 (debian-kernel(a)lists.debian.org) (gcc
version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.132-1 (2020-07-24)
2020-08-27 07:41:12 T libguestfs - 0 - appliance - [ 0.000000]
Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000
udev.event-timeout=6000 no_timer_check lpj=8800000 printk.time=1
cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable
8250.nr_uarts=1 root=UUID=dd6872eb-d621-499d-be42-64d78754c255
selinux=0 guestfs_verbose=1 TERM=linux vsyscall=emulate
It seems that the qemu process has died somewhere between the ROM boot
and the kernel loading.
Any ideas how to debug this?
Thanks!
Sam