8:35 a.m.
Hi everybody,
Wanted to hear your opinion and to receive a smart advice.
I'm trying to use virt-v2v in order to convert ova image (exported from
vcenter) to run on libvirt/kvm - all this inside a VM of fedora.
The converted image is also a fedora.
During the conversion process, in some point of libguestfs activity, I get
double fault panic from L2 (printed as part of libguest output) and the
conversion process fails - no errors appear neither in L0 not in L1 message
logs.
This is more detailed explanation:
Description of problem:
During v2v conversion for ova fedora image in a VM there is double
fault panic happens and L2 crashes during libguestfs conversion.
The crash happens in different stages, but usually upon memory pressure in L0.
There are no error logs in L1 and cannot find strong correlation to
patches that were added to L0-KVM to avoid L0 crash upon nested vm
with high memory pressure -
http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=9....
Command within L1 to perform the conversion (fedora.ova - vmdk image
of fedora - was placed in advance on the VM):
LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os
default -oc qemu:///system -of qcow2 -n default
/var/tmp/fedora-v2v.ova
Version-Release number of selected component (if applicable):
L0:
Kernel: 3.11.8-200.fc19 + nested crash patches
libvritd: 1.0.5.8
qemu: 1.6.1
libguestfs-test-tool 1.22.7fedora=19,release=1.fc19,libvirt
L1:
Kernel: 3.11.8-200..fc19.x86_64
libvirtd: 1.0.5.8
qemu: 1.6.1 + v2v patch (skip vmdk version verification)
libguestfs-test-tool 1.22.7fedora=19,release=2.fc19,libvirt
virt-v2v 0.9.0
L2:
Kernel: 3.11.10-301.fc20
How reproducible:
LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os
default -oc qemu:///system -of qcow2 -n default
/var/tmp/fedora-v2v.ova
Steps to Reproduce:
1. Upload ova image to VM
2. Run v2v to perform the conversion
3. Add some memory pressure on L0 (dd if=/dev/urandom of=/tmp/bigfile count=6M)
Actual results:
libguestfs: checksum of existing appliance:
63a0a4c8acb7f789c7a725bebbb16e64a3bc0999fdca61931babda50292d83b8
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: [00337ms] begin testing qemu features
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -help
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -version
libguestfs: qemu version 1.6
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: [00661ms] finished testing qemu features
[00663ms] /usr/bin/qemu-kvm \
-global virtio-blk-pci.scsi=off \
-nodefconfig \
-nodefaults \
-nographic \
-machine accel=kvm:tcg \
-m 500 \
-no-reboot \
-no-hpet \
-kernel /var/tmp/.guestfs-0/kernel.1416 \
-initrd /var/tmp/.guestfs-0/initrd.1416 \
-device virtio-scsi-pci,id=scsi \
-drive
file=/var/lib/libvirt/images/fedora-v2v-disk1.vmdk,cache=none,format=qcow2,id=hd0,if=ide
\
-drive file=/tmp/3pEC1jAXQs,snapshot=on,format=raw,id=hd1,if=ide \
-drive
file=/var/tmp/.guestfs-0/root.1416,snapshot=on,id=appliance,if=none,cache=unsafe
\
-device scsi-hd,drive=appliance \
-device virtio-serial \
-serial stdio \
-device sga \
-chardev socket,path=/tmp/libguestfsKr0x6j/guestfsd.sock,id=channel0 \
-device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
-netdev user,id=usernet,net=169.254.0.0/16 \
-device virtio-net-pci,netdev=usernet \
-append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdc selinux=0
guestfs_verbose=1 TERM=dumb'
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 02/18/13
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@)
Mon Feb 18 08:29:02 UTC 2013
Term: 80x24
4 0
SeaBIOS (version ?-20131118_112708-)
iPXE (http://ipxe.org) 00:04.0 C100 PCI2.10 PnP PMM+1F3C7C60+1F327C60 C100
Booting from ROM...
Probing EDD (edd=off to disable)... ok
\x1b[2J[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64
(mockbuild@ellaria) (gcc version 4.8.2 20131017 (Red Hat 4.8.2-1)
(GCC) ) #1 SMP Mon Nov 18 17:54:34 IST 2013
[ 0.000000] Command line: panic=1 console=ttyS0 udevtimeout=600
no_timer_check acpi=off printk.time=1 cgroup_disable=memory
root=/dev/sdc selinux=0 guestfs_verbose=1 TERM=dumb
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
............................................
.........More logs here............
.............................................
guestfsd: main_loop: new request, len 0x3c
mount -o ro /dev/sdb /sysroot/
[ 12.645305] PANIC: double fault, error_code: 0x0
[ 12.645305] CPU: 0 PID: 141 Comm: mount Not tainted
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
[ 12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 12.645305] task: ffff88001cc816e0 ti: ffff88001cde6000 task.ti:
ffff88001cde6000
[ 12.645305] RIP: 0033:[<00007fa602c5b99b>] [<00007fa602c5b99b>]
0x7fa602c5b99a
[ 12.645305] RSP: 002b:00007fff4f5884a0 EFLAGS: 00010216
[ 12.645305] RAX: 00007fa602008ff8 RBX: 00007fa601ff0000 RCX: 00007fa601ff0000
[ 12.645305] RDX: 00000000003b7068 RSI: 00007fff4f588560 RDI: 00007fa601ff3d18
[ 12.645305] RBP: 00007fff4f5885d0 R08: 00007fa60200f310 R09: 0000000000000000
[ 12.645305] R10: 0000000000000022 R11: 00007fa60200f310 R12: 00007fa60200e9b0
[ 12.645305] R13: 0000000000000000 R14: 0000000000000000 R15: 00007fa602e6e990
[ 12.645305] FS: 00007fa602e69880(0000) GS:ffff88001f000000(0000)
knlGS:0000000000000000
[ 12.645305] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 12.645305] CR2: 0000000000000000 CR3: 000000001d7fb000 CR4: 00000000000006f0
[ 12.645305]
[ 12.645305] Kernel panic - not syncing: Machine halted.
[ 12.645305] CPU: 0 PID: 141 Comm: mount Not tainted
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
[ 12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 12.645305] ffff88001f005f58 ffff88001f005e90 ffffffff8164024b
ffffffff819e89dc
[ 12.645305] ffff88001f005f08 ffffffff8163c272 0000000000000008
ffff88001f005f18
[ 12.645305] ffff88001f005eb8 ffffffff8163c8e5 0000000000000046
00000000000000b1
[ 12.645305] Call Trace:
[ 12.645305] <#DF> [<ffffffff8164024b>] dump_stack+0x45/0x56
[ 12.645305] [<ffffffff8163c272>] panic+0xc8/0x1d7
[ 12.645305] [<ffffffff8163c8e5>] ? printk+0x67/0x69
[ 12.645305] [<ffffffff81048ae1>] df_debug+0x31/0x40
[ 12.645305] [<ffffffff810132ed>] do_double_fault+0x5d/0x80
[ 12.645305] [<ffffffff81650b88>] double_fault+0x28/0x30
[ 12.645305] <<EOE>>
[ 12.645305] Rebooting in 1 seconds..libguestfs: child_cleanup:
0x3a05f50: child process died
libguestfs: sending SIGTERM to process 1526
libguestfs: trace: mount_ro = -1 (error)
libguestfs: trace: vfs_type "/dev/sda1"
libguestfs: trace: vfs_type = NULL (error)
libguestfs: check_for_filesystem_on: /dev/sda1 (failed to get vfs type)
libguestfs: trace: internal_parse_mountable "/dev/sda1"
libguestfs: trace: internal_parse_mountable = NULL (error)
libguestfs: trace: inspect_os = NULL (error)
libguestfs: trace: close
Additional info:
The same crash also happens when L0 is 3.11.9 (with kvm patch to avoid
L0 crash in nested environment -
http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=9...)
but more rarely.
Thanks a lot,
Rom