10:25 a.m.
Hello,
I am using https://github.com/clalancette/oz/tree/v0.16.0 to customise
qcow2 images. In CentOS 7.5.1804 (yum updated to latest) the image
customisation fails during guestfs launch. Following is the output
of libguestfs-test-tool run. Please help in debugging the problem -
************************************************************
* IMPORTANT NOTICE
*
* When reporting bugs, include the COMPLETE, UNEDITED
* output below in your bug report.
*
************************************************************
LIBGUESTFS_BACKEND=direct
PATH=/home/ciuser/gradle-2.4/bin:/home/ciuser/JAVA/jdk1.8.0_77/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/etc/oz:/home/ciuser/.local/bin:/home/ciuser/bin
XDG_RUNTIME_DIR=/run/user/1000
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: direct
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 500
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /run/user/1000
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.36.10rhel=7,release=6.el7_5.2,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfsIHeKLD
libguestfs: launch: umask=0002
libguestfs: launch: euid=1000
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin5
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-1000/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-1000/appliance.d
supermin: version: 5.1.19
supermin: rpm: detected RPM version 4.11
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-1000/lock
supermin: if-newer: output does not need rebuilding
libguestfs: finished building supermin appliance
libguestfs: begin testing qemu features
libguestfs: checking for previously cached test results of
/usr/libexec/qemu-kvm, in /var/tmp/.guestfs-1000
libguestfs: loading previously cached test results
libguestfs: QMP parse error: parse error: premature EOF\n
\n (right here) ------^\n (ignored)
libguestfs: qemu version: 1.5
libguestfs: qemu mandatory locking: no
libguestfs: finished testing qemu features
[00034ms] /usr/libexec/qemu-kvm \
-global virtio-blk-pci.scsi=off \
-nodefconfig \
-enable-fips \
-nodefaults \
-display none \
-machine accel=kvm:tcg \
-cpu host \
-m 500 \
-no-reboot \
-rtc driftfix=slew \
-no-hpet \
-global kvm-pit.lost_tick_policy=discard \
-kernel /var/tmp/.guestfs-1000/appliance.d/kernel \
-initrd /var/tmp/.guestfs-1000/appliance.d/initrd \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0 \
-device virtio-scsi-pci,id=scsi \
-drive
file=/tmp/libguestfsIHeKLD/scratch1.img,cache=unsafe,format=raw,id=hd0,if=none
\
-device scsi-hd,drive=hd0 \
-drive
file=/var/tmp/.guestfs-1000/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw
\
-device scsi-hd,drive=appliance \
-device virtio-serial-pci \
-serial stdio \
-device sga \
-chardev
socket,path=/run/user/1000/libguestfsoBqGwV/guestfsd.sock,id=channel0 \
-device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
-append 'panic=1 console=ttyS0 edd=off udevtimeout=6000
udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory
usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb
selinux=0 guestfs_verbose=1 TERM=xterm'
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 06/09/14
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@) Mon
Jun 9 21:33:48 UTC 2014
Term: 80x24
4 0
SeaBIOS (version 1.11.0-2.el7)
Booting from ROM...
\x1b[2J[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.10.0-862.14.4.el7.x86_64 (
mockbuild(a)kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat
4.8.5-28) (GCC) ) #1 SMP Wed Sep 26 15:12:11 UTC 2018
[ 0.000000] Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000
udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory
usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb
selinux=0 guestfs_verbose=1 TERM=xterm
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001f3fbfff] usable
[ 0.000000] BIOS-e820: [mem 0x000000001f3fc000-0x000000001f3fffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff]
reserved
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.4 present.
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] e820: last_pfn = 0x1f3fc max_arch_pfn = 0x400000000
[ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC
[ 0.000000] found SMP MP-table at [mem 0x000f63b0-0x000f63bf] mapped at
[ffffffffff2003b0]
[ 0.000000] RAMDISK: [mem 0x1f0c0000-0x1f3effff]
[ 0.000000] Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 00000000000f6210 00014 (v00 BOCHS )
[ 0.000000] ACPI: RSDT 000000001f3ffad7 00030 (v01 BOCHS BXPCRSDT
00000001 BXPC 00000001)
[ 0.000000] ACPI: FACP 000000001f3ff177 00074 (v01 BOCHS BXPCFACP
00000001 BXPC 00000001)
[ 0.000000] ACPI: DSDT 000000001f3fe040 01137 (v01 BOCHS BXPCDSDT
00000001 BXPC 00000001)
[ 0.000000] ACPI: FACS 000000001f3fe000 00040
[ 0.000000] ACPI: SSDT 000000001f3ff1eb 00874 (v01 BOCHS BXPCSSDT
00000001 BXPC 00000001)
[ 0.000000] ACPI: APIC 000000001f3ffa5f 00078 (v01 BOCHS BXPCAPIC
00000001 BXPC 00000001)
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000001f3fbfff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x1f099000-0x1f0bffff]
[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[ 0.000000] kvm-clock: cpu 0, msr 0:1f049001, primary cpu clock
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00001000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00001000-0x0009efff]
[ 0.000000] node 0: [mem 0x00100000-0x1f3fbfff]
[ 0.000000] Initmem setup node 0 [mem 0x00001000-0x1f3fbfff]
[ 0.000000] ACPI: PM-Timer IO Port: 0x608
[ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI
0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[ 0.000000] e820: [mem 0x1f400000-0xfeffbfff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on KVM
[ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1
nr_node_ids:1
[ 0.000000] PERCPU: Embedded 35 pages/cpu @ffff93c95ee00000 s104920
r8192 d30248 u2097152
[ 0.000000] KVM setup async PF for cpu 0
[ 0.000000] kvm-stealtime: cpu 0, msr 1ee13500
[ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on.
Total pages: 125877
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: panic=1 console=ttyS0 edd=off
udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1
cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable
8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm
[ 0.000000] Disabling memory control group subsystem
[ 0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
[ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using
standard form
[ 0.000000] Memory: 480520k/511984k available (7348k kernel code, 392k
absent, 31072k reserved, 6281k data, 1844k init)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] x86/pti: Unmapping kernel while in userspace
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] \tRCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.
[ 0.000000] NR_IRQS:327936 nr_irqs:256 0
[ 0.000000] Console: colour *CGA 80x25
[ 0.000000] console [ttyS0] enabled
[ 0.000000] tsc: Detected 2494.218 MHz processor
[ 0.098000] Calibrating delay loop (skipped) preset value.. 4988.43
BogoMIPS (lpj=2494218)
[ 0.104230] pid_max: default: 32768 minimum: 301
[ 0.110254] Security Framework initialized
[ 0.116237] SELinux: Disabled at boot.
[ 0.117234] Yama: becoming mindful.
[ 0.131312] Dentry cache hash table entries: 65536 (order: 7, 524288
bytes)
[ 0.141311] Inode-cache hash table entries: 32768 (order: 6, 262144
bytes)
[ 0.153253] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[ 0.157238] Mountpoint-cache hash table entries: 1024 (order: 1, 8192
bytes)
[ 0.162358] Initializing cgroup subsys memory
[ 0.168236] Initializing cgroup subsys devices
[ 0.192231] Initializing cgroup subsys freezer
[ 0.211226] Initializing cgroup subsys net_cls
[ 0.229237] Initializing cgroup subsys blkio
[ 0.240227] Initializing cgroup subsys perf_event
[ 0.242232] Initializing cgroup subsys hugetlb
[ 0.256002] random: fast init done
[ 0.281241] Initializing cgroup subsys pids
[ 0.297210] Initializing cgroup subsys net_prio
[ 0.857224] mce: CPU supports 10 MCE banks
[ 1.161221] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[ 1.175212] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0
[ 1.189209] tlb_flushall_shift: 6
[ 1.198225] Speculative Store Bypass: Vulnerable
[ 1.206223] FEATURE SPEC_CTRL Not Present
[ 1.218208] FEATURE IBPB_SUPPORT Not Present
[ 4.853233] Spectre V2 : Vulnerable: Retpoline without IBPB
[ 13.376012] random: crng init done
[ 101.375237] Freeing SMP alternatives: 24k freed
[ 110.720007] BUG: unable to handle kernel NULL pointer dereference at
0000000000000102
[ 110.721000] IP: [<ffffffffad8b3ef2>] __queue_work+0x32/0x3e0
[ 110.721000] PGD 0
[ 110.721000] Oops: 0000 [#1] SMP
[ 110.721000] Modules linked in:
[ 110.721000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted
3.10.0-862.14.4.el7.x86_64 #1
[ 110.721000] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 110.721000] task: ffffffffae416480 ti: ffffffffae400000 task.ti:
ffffffffae400000
[ 110.721000] RIP: 0010:[<ffffffffad8b3ef2>] [<ffffffffad8b3ef2>]
__queue_work+0x32/0x3e0
[ 110.721000] RSP: 0000:ffff93c95ee03e20 EFLAGS: 00010046
[ 110.721000] RAX: 0000000000000082 RBX: 0000000000000087 RCX:
0000000000000000
[ 110.721000] RDX: ffffffffae4e0340 RSI: 0000000000000000 RDI:
0000000000001400
[ 110.721000] RBP: ffff93c95ee03e58 R08: 0000000000000000 R09:
0000000000004000
[ 110.721000] R10: ffffffffaea17998 R11: 0000000000007ffe R12:
ffffffffae4e0340
[ 110.721000] R13: 0000000000001400 R14: 0000000000000000 R15:
ffffffffae2b3a62
[ 110.721000] FS: 0000000000000000(0000) GS:ffff93c95ee00000(0000)
knlGS:0000000000000000
[ 110.721000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 110.721000] CR2: 0000000000000102 CR3: 000000001200e000 CR4:
00000000001606f0
[ 110.721000] Call Trace:
[ 110.721000] <IRQ>
[ 110.721000] [<ffffffffad8b44f5>] queue_work_on+0x45/0x50
[ 110.721000] [<ffffffffadc564d6>] credit_entropy_bits+0x1c6/0x290
[ 110.721000] [<ffffffffadc571c4>] ? add_interrupt_randomness+0x1c4/0x230
[ 110.721000] [<ffffffffadc571c4>] add_interrupt_randomness+0x1c4/0x230
[ 110.721000] [<ffffffffad943f7f>] handle_irq_event_percpu+0x3f/0x80
[ 110.721000] [<ffffffffad943ffc>] handle_irq_event+0x3c/0x60
[ 110.721000] [<ffffffffad9470a3>] handle_level_irq+0x73/0xd0
[ 110.721000] [<ffffffffad82e504>] handle_irq+0xe4/0x1a0
[ 110.721000] [<ffffffffad89cf08>] ? __local_bh_enable+0x28/0x90
[ 110.721000] [<ffffffffadf29f9d>] do_IRQ+0x4d/0xf0
[ 110.721000] [<ffffffffadf28c83>] ? simd_coprocessor_error+0x3/0x30
[ 110.721000] [<ffffffffadf1c362>] common_interrupt+0x162/0x162
[ 110.721000] <EOI>
[ 110.721000] [<ffffffffad877b80>] ? __native_set_fixmap+0x40/0x40
[ 110.721000] [<ffffffffad86b14c>] ?
end_pv_cpu_ops_usergs_sysret32+0x3/0x3
[ 110.721000] [<ffffffffadf28c83>] ? simd_coprocessor_error+0x3/0x30
[ 110.721000] [<ffffffffad86a4b6>] ? native_restore_fl+0x6/0x10
[ 110.721000] [<ffffffffad833126>] text_poke_early+0x36/0x40
[ 110.721000] [<ffffffffad833383>] apply_paravirt+0xb3/0xe0
[ 110.721000] [<ffffffffad8973f4>] ? vprintk_emit+0x3c4/0x510
[ 110.721000] [<ffffffffad9a00e6>] ? free_hot_cold_page+0x106/0x160
[ 110.721000] [<ffffffffad99bc6d>] ? adjust_managed_page_count+0x5d/0x80
[ 110.721000] [<ffffffffad86fc6a>] ? free_init_pages+0x15a/0x1b0
[ 110.721000] [<ffffffffae579537>] alternative_instructions+0xc3/0xca
[ 110.721000] [<ffffffffae57b0cb>] check_bugs+0x25f/0x2b6
[ 110.721000] [<ffffffffae56c19d>] start_kernel+0x422/0x46c
[ 110.721000] [<ffffffffae56bb7b>] ? repair_env_string+0x5c/0x5c
[ 110.721000] [<ffffffffae56b120>] ? early_idt_handler_array+0x120/0x120
[ 110.721000] [<ffffffffae56b72f>] x86_64_start_reservations+0x24/0x26
[ 110.721000] [<ffffffffae56b885>] x86_64_start_kernel+0x154/0x177
[ 110.721000] [<ffffffffad8000d5>] start_cpu+0x5/0x14
[ 110.721000] Code: 89 e5 41 57 41 56 49 89 f6 41 55 41 89 fd 41 54 49 89
d4 53 48 83 ec 10 89 7d d4 ff 14 25 00 07 43 ae f6 c4 02 0f 85 de 02 00 00
<41> f6 86 02 01 00 00 01 0f 85 78 02 00 00 49 c7 c7 08 5a 01 00
[ 110.721000] RIP [<ffffffffad8b3ef2>] __queue_work+0x32/0x3e0
[ 110.721000] RSP <ffff93c95ee03e20>
[ 110.721000] CR2: 0000000000000102
[ 110.721000] ---[ end trace aaa07a7d70f6c07b ]---
[ 110.721000] Kernel panic - not syncing: Fatal exception in interrupt
[ 110.721000] Rebooting in 1 seconds..libguestfs: error: appliance closed
the connection unexpectedly, see earlier error messages
libguestfs: child_cleanup: 0x55690ed44c60: child process died
libguestfs: sending SIGTERM to process 31847
libguestfs: qemu maxrss 96256K
libguestfs: error: guestfs_launch failed, see earlier error messages
libguestfs: closing guestfs handle 0x55690ed44c60 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsIHeKLD
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /run/user/1000/libguestfsoBqGwV
This is running on a VM with nested virtualization enabled.
rpm -qa | grep kernel
kernel-tools-3.10.0-862.14.4.el7.x86_64
kernel-tools-libs-3.10.0-862.14.4.el7.x86_64
kernel-3.10.0-862.14.4.el7.x86_64
kernel-3.10.0-862.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
rpm -qa | grep supermin
supermin5-5.1.19-1.el7.x86_64
Possibly similar issue reported in CentOS
https://bugzilla.redhat.com/show_bug.cgi?id=1535973 ?
Thanks,
Sarbajit