[PATCH] Enable coredumps to be captured from the appliance (RHBZ#619334).
https://bugzilla.redhat.com/show_bug.cgi?id=619334 This is a slightly unsatisfactory patch which allows coredumps to be captured when they occur inside the appliance. You can capture coredumps by doing: export LIBGUESTFS_COREDUMP=/sysroot/core.%t.%p.%e or equivalently: g.set_coredump ("/sysroot/core.%t.%p.%e") or variations thereof, see the manual page. When a coredump occurs, a core file will be dropped into the root directory of the currently mounted guest disk. If it is guestfsd which crashes, then the appliance will exit, but I have checked that the coredump is written and synched first. You can then copy this out of the guest disk using guestfish and debug using gdb: $ guestfish --ro -a guest_disk -m /dev/vg/lv_root
<fs> ll / <fs> download /core.xxx.yyy.zzz core
$ gdb daemon/guestfsd core http://fedoraproject.org/wiki/StackTraces#Obtaining_a_stack_trace_from_a_cor... Clearly you need to have a writable guest disk mounted for this to work, and you have to not care too much about having large core files written to it. Ideally we would have liked to use a special capture disk, but it turns out that the kernel contains code which disallows using a /dev device (or any non-regular file) for coredumps: http://lxr.linux.no/linux+v2.6.34.1/fs/exec.c#L1930 We could do it using a coredump-to-pipe, but I think it would be better to just remove this limitation in the kernel. Another shortcoming is that a coredump isn't a stack trace, and to get a stack trace you really need a copy of the daemon around with debugging enabled. It is possible we could change the RPM packaging to ship this. Luckily the suspected bug in Augeas which we are chasing (RHBZ#613967) happens in aug_init or aug_save when the conditions above are satisfied and we should be able to get a useful coredump. However since I am unable to reproduce this bug at all, I cannot help with this, so it's over to Marek and Matt. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
On Sat, Jul 31, 2010 at 05:20:19PM +0100, Richard W.M. Jones wrote:
This is a slightly unsatisfactory patch which allows coredumps to be captured when they occur inside the appliance. You can capture coredumps by doing:
export LIBGUESTFS_COREDUMP=/sysroot/core.%t.%p.%e
or equivalently:
g.set_coredump ("/sysroot/core.%t.%p.%e")
or variations thereof, see the manual page.
Actually it strikes me that this API is wrong. It should be something like: set-coredump-file eg. set-coredump-file /core.%t.%p.%e - Coredump to a mounted guest filesystem. This would add the /sysroot/ prefix automatically. The %-patterns would be some limited subset of what the kernel supports. set-coredump-device eg. set-coredump-device /dev/sdb - Coredump to a capture device. This would be implemented at first using a coredump-to-pipe, but we could later change the implemention to coredump directly to a device if we got a patch into the kernel to support it. clear-coredump - Clear the coredump file/device. Equivalent to setting it to NULL right now. This interface is more extensible and future-proof. We can extend it by adding more set-coredump-* methods. It's future-proof because it doesn't depend so closely on the current kernel implementation. The environment variable would have to be changed as well. - - - - The ideal future enhancement would be to get a stack trace without needing the coredump + gdb step. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
On Sat, Jul 31, 2010 at 06:26:17PM +0100, Richard W.M. Jones wrote:
The ideal future enhancement would be to get a stack trace without needing the coredump + gdb step.
Notes to self: gstack(1) backtrace(3) http://samba.org/ftp/unpacked/junkcode/segv_handler/ Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
The last version omitted to implement LIBGUESTFS_COREDUMP. This fixes that ... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
participants (1)
-
Richard W.M. Jones