7:44 a.m.
Sometimes guestfsd fails to gather info, and the virt-<tool> -v output
is usually not useful to figure out whats going on within the temporary
guest. I see the /init script has support to run guestfsd with a debug
tool, which is currently valgrind.
Granted, valgrind support is a compile time thing. What would be a good
way to optionally run guestfsd with strace or gdb?
Should each tool get a new option, something like 'virt-<tool>
--debug=strace', or should it be like 'XYZ_STRACE=1 virt-<tool>'?
Olaf
8:24 a.m.
On Fri, Sep 21, 2012 at 02:44:52PM +0200, Olaf Hering wrote:
Sometimes guestfsd fails to gather info, and the virt-<tool> -v output
is usually not useful to figure out whats going on within the temporary
guest. I see the /init script has support to run guestfsd with a debug
tool, which is currently valgrind.
Granted, valgrind support is a compile time thing. What would be a good
way to optionally run guestfsd with strace or gdb?
Stepping back a bit, we have a long-open bug about collecting core
dumps from the appliance:
https://bugzilla.redhat.com/show_bug.cgi?id=619334
The problem with collecting core dumps is how you get them out of the
appliance, given that it is short-lived and disappears on exit. We
looked at various methods including writing them to a block device or
squirting it down a virtio-serial connection, but none of them are
very satisfactory.
strace output can do through the usual console logging mechanism.
gdb seems intractable to me if you want to use gdb interactively.
Just printing stack traces can go via the console logging mechanism.
Note there's very limited development tools in the appliance (no
debuginfo), and I'd be cautious about how much stuff gdb will pull in
via dependencies. That's why we looked at core dumps.
Should each tool get a new option, something like
'virt-<tool>
--debug=strace', or should it be like 'XYZ_STRACE=1 virt-<tool>'?
Adding options to every tool seems to be a lot of work.
The guestfs handle lets you set several useful properties on it using
explicit calls (eg. guestfs_set_attach_method). However we don't
expose all of these properties in every tool. Instead we allow the
properties to take default values when the handle is created from
environment variables (eg. $LIBGUESTFS_ATTACH_METHOD), and this means
that every tool doesn't need to support every property.
The '-v' and '-x' options on most tools are convenient, but are
arguably superfluous since people could set LIBGUESTFS_DEBUG=1 and
LIBGUESTFS_TRACE=1 and get almost precisely the same effect.
So environment variables are the way to go.
Another complication is that guestfs_create doesn't have a rich way to
return errors, and thus if the environment variables cannot be parsed
things start to go wrong. Currently it spits error messages out to
stderr in this case, which is very bad behaviour.
With the benefit of hindsight, splitting handle creation and
environment parsing into two calls would have been better:
g = guestfs_create ();
if (!g) error;
if (guestfs_set_environment (g) == -1) {
/* with the handle created, richer error methods are now available */
}
But we cannot change that now without breaking clients.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
8:40 a.m.
On Fri, Sep 21, Richard W.M. Jones wrote:
The problem with collecting core dumps is how you get them out of
the
appliance, given that it is short-lived and disappears on exit. We
looked at various methods including writing them to a block device or
squirting it down a virtio-serial connection, but none of them are
very satisfactory.
Yes, getting a file out of the guest is tricky.
strace output can do through the usual console logging mechanism.
Yes, my take would be 'var="strace -f -s 123 -tt" ; $var guestfsd'.
gdb seems intractable to me if you want to use gdb interactively.
Just printing stack traces can go via the console logging mechanism.
Note there's very limited development tools in the appliance (no
debuginfo), and I'd be cautious about how much stuff gdb will pull in
via dependencies. That's why we looked at core dumps.
gdb is certainly more advanced, something like 'var="gdb --readnow -ex r
-ex bt -ex quit' ; $var guestfsd' would be good enough, or gdb
--command=FILE.
As you say, without adding debuginfo into guest the backtrace would be
not that useful. But its better than nothing.
Yesterday I tried to run guestfsd via gdb, and it required a few python
files. But it turned out that the actual segfault happend in the host,
not the guest. So I did not look further if gdb did really work.
Olaf
8:49 a.m.
On Fri, Sep 21, 2012 at 03:40:35PM +0200, Olaf Hering wrote:
gdb is certainly more advanced, something like 'var="gdb
--readnow -ex r
-ex bt -ex quit' ; $var guestfsd' would be good enough, or gdb
--command=FILE.
As you say, without adding debuginfo into guest the backtrace would be
not that useful. But its better than nothing.
glibc -- on Fedora -- prints a stack trace on exit, so we've not had a
need to get one out using gdb. A core dump would be the really useful
thing for us.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
8:51 a.m.
On Fri, Sep 21, Richard W.M. Jones wrote:
On Fri, Sep 21, 2012 at 03:40:35PM +0200, Olaf Hering wrote:
> gdb is certainly more advanced, something like 'var="gdb --readnow -ex r
> -ex bt -ex quit' ; $var guestfsd' would be good enough, or gdb
> --command=FILE.
> As you say, without adding debuginfo into guest the backtrace would be
> not that useful. But its better than nothing.
glibc -- on Fedora -- prints a stack trace on exit, so we've not had a
need to get one out using gdb. A core dump would be the really useful
thing for us.
How does it do that, via a segfault handler? I mean, is that a private
patch or something in upstream glibc?
Olaf
9:15 a.m.
On Fri, Sep 21, 2012 at 03:51:22PM +0200, Olaf Hering wrote:
On Fri, Sep 21, Richard W.M. Jones wrote:
> On Fri, Sep 21, 2012 at 03:40:35PM +0200, Olaf Hering wrote:
> > gdb is certainly more advanced, something like 'var="gdb --readnow -ex
r
> > -ex bt -ex quit' ; $var guestfsd' would be good enough, or gdb
> > --command=FILE.
> > As you say, without adding debuginfo into guest the backtrace would be
> > not that useful. But its better than nothing.
>
> glibc -- on Fedora -- prints a stack trace on exit, so we've not had a
> need to get one out using gdb. A core dump would be the really useful
> thing for us.
How does it do that, via a segfault handler? I mean, is that a private
patch or something in upstream glibc?
What I said there wasn't quite correct.
Every C binary in Fedora is compiled with -D_FORTIFY_SOURCE=2 which
provides a stack trace when string functions fail. I was probably
confused by seeing those.
But segfaults are not included in this, and they in fact don't
give a stack trace.
Doing
LD_PRELOAD=/lib64/libSegFault.so
does cause all segfaulting programs to produce a stack trace.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
1:02 p.m.
On Fri, Sep 21, 2012 at 02:24:13PM +0100, Richard W.M. Jones wrote:
So environment variables are the way to go.
Another complication is that guestfs_create doesn't have a rich way to
return errors, and thus if the environment variables cannot be parsed
things start to go wrong. Currently it spits error messages out to
stderr in this case, which is very bad behaviour.
With the benefit of hindsight, splitting handle creation and
environment parsing into two calls would have been better:
g = guestfs_create ();
if (!g) error;
if (guestfs_set_environment (g) == -1) {
/* with the handle created, richer error methods are now available */
}
But we cannot change that now without breaking clients.
I thought of a way to do this which is backwards compatible.
Introduce a new guestfs_create_flags () call which takes a flag
variable. By default, passing flag == 0 is entirely the same as
guestfs_create.
g = guestfs_create_flags (0);
if (!g) error;
However you could also use it like this:
g = guestfs_create_flags (GUESTFS_CREATE_NO_ENVIRONMENT);
if (!g) error;
where no environment variable processing is done (hence no errors are
possible except ENOMEM). Callers can do environment variable
processing later (see guestfs_set_environment above). Or maybe not if
they don't want to be affected by the environment. Errors can be
handled properly in those later calls.
In the language backends, we'd have to map this to more flexible
functions, eg. in Perl:
my $g = Sys::Guestfs->new (environment => 0);
(which is of course backwards compatible with existing code).
I'll post a patch for this tomorrow if I have the time.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
9:54 a.m.
On Sun, Oct 14, 2012 at 07:02:29PM +0100, Richard W.M. Jones wrote:
Introduce a new guestfs_create_flags () call which takes a flag
variable. By default, passing flag == 0 is entirely the same as
guestfs_create.
g = guestfs_create_flags (0);
if (!g) error;
However you could also use it like this:
g = guestfs_create_flags (GUESTFS_CREATE_NO_ENVIRONMENT);
if (!g) error;
where no environment variable processing is done (hence no errors are
possible except ENOMEM). Callers can do environment variable
processing later (see guestfs_set_environment above). Or maybe not if
they don't want to be affected by the environment. Errors can be
handled properly in those later calls.
In the language backends, we'd have to map this to more flexible
functions, eg. in Perl:
my $g = Sys::Guestfs->new (environment => 0);
(which is of course backwards compatible with existing code).
That took much longer than I thought it would ...
https://github.com/libguestfs/libguestfs/commit/9466060201600db47016133d8...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
4235
days inactive
4259
days old
7 comments
2 participants
participants (2)
-
Olaf Hering -
Richard W.M. Jones