simplify debugging of guestfsd
Sometimes guestfsd fails to gather info, and the virt-<tool> -v output is usually not useful to figure out whats going on within the temporary guest. I see the /init script has support to run guestfsd with a debug tool, which is currently valgrind. Granted, valgrind support is a compile time thing. What would be a good way to optionally run guestfsd with strace or gdb? Should each tool get a new option, something like 'virt-<tool> --debug=strace', or should it be like 'XYZ_STRACE=1 virt-<tool>'? Olaf
On Fri, Sep 21, 2012 at 02:44:52PM +0200, Olaf Hering wrote:
Sometimes guestfsd fails to gather info, and the virt-<tool> -v output is usually not useful to figure out whats going on within the temporary guest. I see the /init script has support to run guestfsd with a debug tool, which is currently valgrind.
Granted, valgrind support is a compile time thing. What would be a good way to optionally run guestfsd with strace or gdb?
Stepping back a bit, we have a long-open bug about collecting core dumps from the appliance: https://bugzilla.redhat.com/show_bug.cgi?id=619334 The problem with collecting core dumps is how you get them out of the appliance, given that it is short-lived and disappears on exit. We looked at various methods including writing them to a block device or squirting it down a virtio-serial connection, but none of them are very satisfactory. strace output can do through the usual console logging mechanism. gdb seems intractable to me if you want to use gdb interactively. Just printing stack traces can go via the console logging mechanism. Note there's very limited development tools in the appliance (no debuginfo), and I'd be cautious about how much stuff gdb will pull in via dependencies. That's why we looked at core dumps.
Should each tool get a new option, something like 'virt-<tool> --debug=strace', or should it be like 'XYZ_STRACE=1 virt-<tool>'?
Adding options to every tool seems to be a lot of work. The guestfs handle lets you set several useful properties on it using explicit calls (eg. guestfs_set_attach_method). However we don't expose all of these properties in every tool. Instead we allow the properties to take default values when the handle is created from environment variables (eg. $LIBGUESTFS_ATTACH_METHOD), and this means that every tool doesn't need to support every property. The '-v' and '-x' options on most tools are convenient, but are arguably superfluous since people could set LIBGUESTFS_DEBUG=1 and LIBGUESTFS_TRACE=1 and get almost precisely the same effect. So environment variables are the way to go. Another complication is that guestfs_create doesn't have a rich way to return errors, and thus if the environment variables cannot be parsed things start to go wrong. Currently it spits error messages out to stderr in this case, which is very bad behaviour. With the benefit of hindsight, splitting handle creation and environment parsing into two calls would have been better: g = guestfs_create (); if (!g) error; if (guestfs_set_environment (g) == -1) { /* with the handle created, richer error methods are now available */ } But we cannot change that now without breaking clients. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On Fri, Sep 21, Richard W.M. Jones wrote:
The problem with collecting core dumps is how you get them out of the appliance, given that it is short-lived and disappears on exit. We looked at various methods including writing them to a block device or squirting it down a virtio-serial connection, but none of them are very satisfactory.
Yes, getting a file out of the guest is tricky.
strace output can do through the usual console logging mechanism.
Yes, my take would be 'var="strace -f -s 123 -tt" ; $var guestfsd'.
gdb seems intractable to me if you want to use gdb interactively. Just printing stack traces can go via the console logging mechanism. Note there's very limited development tools in the appliance (no debuginfo), and I'd be cautious about how much stuff gdb will pull in via dependencies. That's why we looked at core dumps.
gdb is certainly more advanced, something like 'var="gdb --readnow -ex r -ex bt -ex quit' ; $var guestfsd' would be good enough, or gdb --command=FILE. As you say, without adding debuginfo into guest the backtrace would be not that useful. But its better than nothing. Yesterday I tried to run guestfsd via gdb, and it required a few python files. But it turned out that the actual segfault happend in the host, not the guest. So I did not look further if gdb did really work. Olaf
On Fri, Sep 21, 2012 at 03:40:35PM +0200, Olaf Hering wrote:
gdb is certainly more advanced, something like 'var="gdb --readnow -ex r -ex bt -ex quit' ; $var guestfsd' would be good enough, or gdb --command=FILE. As you say, without adding debuginfo into guest the backtrace would be not that useful. But its better than nothing.
glibc -- on Fedora -- prints a stack trace on exit, so we've not had a need to get one out using gdb. A core dump would be the really useful thing for us. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On Fri, Sep 21, Richard W.M. Jones wrote:
On Fri, Sep 21, 2012 at 03:40:35PM +0200, Olaf Hering wrote:
gdb is certainly more advanced, something like 'var="gdb --readnow -ex r -ex bt -ex quit' ; $var guestfsd' would be good enough, or gdb --command=FILE. As you say, without adding debuginfo into guest the backtrace would be not that useful. But its better than nothing.
glibc -- on Fedora -- prints a stack trace on exit, so we've not had a need to get one out using gdb. A core dump would be the really useful thing for us.
How does it do that, via a segfault handler? I mean, is that a private patch or something in upstream glibc? Olaf
On Fri, Sep 21, 2012 at 03:51:22PM +0200, Olaf Hering wrote:
On Fri, Sep 21, Richard W.M. Jones wrote:
On Fri, Sep 21, 2012 at 03:40:35PM +0200, Olaf Hering wrote:
gdb is certainly more advanced, something like 'var="gdb --readnow -ex r -ex bt -ex quit' ; $var guestfsd' would be good enough, or gdb --command=FILE. As you say, without adding debuginfo into guest the backtrace would be not that useful. But its better than nothing.
glibc -- on Fedora -- prints a stack trace on exit, so we've not had a need to get one out using gdb. A core dump would be the really useful thing for us.
How does it do that, via a segfault handler? I mean, is that a private patch or something in upstream glibc?
What I said there wasn't quite correct. Every C binary in Fedora is compiled with -D_FORTIFY_SOURCE=2 which provides a stack trace when string functions fail. I was probably confused by seeing those. But segfaults are not included in this, and they in fact don't give a stack trace. Doing LD_PRELOAD=/lib64/libSegFault.so does cause all segfaulting programs to produce a stack trace. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On Fri, Sep 21, 2012 at 02:24:13PM +0100, Richard W.M. Jones wrote:
So environment variables are the way to go.
Another complication is that guestfs_create doesn't have a rich way to return errors, and thus if the environment variables cannot be parsed things start to go wrong. Currently it spits error messages out to stderr in this case, which is very bad behaviour.
With the benefit of hindsight, splitting handle creation and environment parsing into two calls would have been better:
g = guestfs_create (); if (!g) error; if (guestfs_set_environment (g) == -1) { /* with the handle created, richer error methods are now available */ }
But we cannot change that now without breaking clients.
I thought of a way to do this which is backwards compatible. Introduce a new guestfs_create_flags () call which takes a flag variable. By default, passing flag == 0 is entirely the same as guestfs_create. g = guestfs_create_flags (0); if (!g) error; However you could also use it like this: g = guestfs_create_flags (GUESTFS_CREATE_NO_ENVIRONMENT); if (!g) error; where no environment variable processing is done (hence no errors are possible except ENOMEM). Callers can do environment variable processing later (see guestfs_set_environment above). Or maybe not if they don't want to be affected by the environment. Errors can be handled properly in those later calls. In the language backends, we'd have to map this to more flexible functions, eg. in Perl: my $g = Sys::Guestfs->new (environment => 0); (which is of course backwards compatible with existing code). I'll post a patch for this tomorrow if I have the time. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Sun, Oct 14, 2012 at 07:02:29PM +0100, Richard W.M. Jones wrote:
Introduce a new guestfs_create_flags () call which takes a flag variable. By default, passing flag == 0 is entirely the same as guestfs_create.
g = guestfs_create_flags (0); if (!g) error;
However you could also use it like this:
g = guestfs_create_flags (GUESTFS_CREATE_NO_ENVIRONMENT); if (!g) error;
where no environment variable processing is done (hence no errors are possible except ENOMEM). Callers can do environment variable processing later (see guestfs_set_environment above). Or maybe not if they don't want to be affected by the environment. Errors can be handled properly in those later calls.
In the language backends, we'd have to map this to more flexible functions, eg. in Perl:
my $g = Sys::Guestfs->new (environment => 0);
(which is of course backwards compatible with existing code).
That took much longer than I thought it would ... https://github.com/libguestfs/libguestfs/commit/9466060201600db47016133d80af... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
participants (2)
-
Olaf Hering -
Richard W.M. Jones