8:41 p.m.
Hey Rich,
In the past I know you have mentioned that using
guestfs.kill_subprocess() isn't strictly necessary after you are finished
using libguestfs. I believe you said that in the python bindings, that
step would happen automatically when the object goes out of scope and the
__del__ method is called. However, since it never seemed to harm anything
and I wanted to be extra safe, I always called guestfs.kill_subprocess() in
Oz, without apparent harm.
However, Richard Su has been investigating an issue where
guestfs.kill_subprocess() actually does cause harm. I believe the
situation is that running Oz (and hence libguestfs) simultaneously from two
threads in the same process causes the entire process to hang. If we
remove the guestfs.kill_subprocess(), the problem goes away. rwsu, can you
please fill in a bit more detail here, as I'm not intimately familiar with
the issue? Things like host OS, libguestfs version, etc etc would be
helpful. Also confirmation that the problem is indeed in the
multi-threaded environment of the image factory like I think it is.
rjones, do you have any ideas here? I'm going to merge rwsu's patch
to remove the call to kill_subprocess(), but I thought it would be worth a
little more digging into why it was happening.
Thanks,
Chris Lalancette
3:29 a.m.
On Thu, Nov 03, 2011 at 09:41:40PM -0400, Chris Lalancette wrote:
Hey Rich,
In the past I know you have mentioned that using
guestfs.kill_subprocess() isn't strictly necessary after you are finished
using libguestfs. I believe you said that in the python bindings, that
step would happen automatically when the object goes out of scope and the
__del__ method is called. However, since it never seemed to harm anything
and I wanted to be extra safe, I always called guestfs.kill_subprocess() in
Oz, without apparent harm.
Python bindings now have an explicit close call:
8c5bdc3e12947580e91c018b71adf9ad3128bb75
http://bugzilla.redhat.com/717786
However, Richard Su has been investigating an issue where
guestfs.kill_subprocess() actually does cause harm. I believe the
situation is that running Oz (and hence libguestfs) simultaneously from two
threads in the same process causes the entire process to hang. If we
remove the guestfs.kill_subprocess(), the problem goes away. rwsu, can you
please fill in a bit more detail here, as I'm not intimately familiar with
the issue? Things like host OS, libguestfs version, etc etc would be
helpful. Also confirmation that the problem is indeed in the
multi-threaded environment of the image factory like I think it is.
rjones, do you have any ideas here? I'm going to merge rwsu's patch
to remove the call to kill_subprocess(), but I thought it would be worth a
little more digging into why it was happening.
You could be hitting this bug:
http://bugzilla.redhat.com/684980
To debug this I need to see:
- version of libguestfs
- platform that you're running it on
- full output with LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 set
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
11:47 a.m.
On 11/04/2011 01:29 AM, Richard W.M. Jones wrote:
On Thu, Nov 03, 2011 at 09:41:40PM -0400, Chris Lalancette wrote:
> Hey Rich,
> In the past I know you have mentioned that using
> guestfs.kill_subprocess() isn't strictly necessary after you are finished
> using libguestfs. I believe you said that in the python bindings, that
> step would happen automatically when the object goes out of scope and the
> __del__ method is called. However, since it never seemed to harm anything
> and I wanted to be extra safe, I always called guestfs.kill_subprocess() in
> Oz, without apparent harm.
Python bindings now have an explicit close call:
8c5bdc3e12947580e91c018b71adf9ad3128bb75
http://bugzilla.redhat.com/717786
> However, Richard Su has been investigating an issue where
> guestfs.kill_subprocess() actually does cause harm. I believe the
> situation is that running Oz (and hence libguestfs) simultaneously from two
> threads in the same process causes the entire process to hang. If we
> remove the guestfs.kill_subprocess(), the problem goes away. rwsu, can you
> please fill in a bit more detail here, as I'm not intimately familiar with
> the issue? Things like host OS, libguestfs version, etc etc would be
> helpful. Also confirmation that the problem is indeed in the
> multi-threaded environment of the image factory like I think it is.
> rjones, do you have any ideas here? I'm going to merge rwsu's patch
> to remove the call to kill_subprocess(), but I thought it would be worth a
> little more digging into why it was happening.
You could be hitting this bug:
http://bugzilla.redhat.com/684980
To debug this I need to see:
- version of libguestfs
- platform that you're running it on
- full output with LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 set
Rich.
The problem I'm running into is when I start two image builds, and thus
two guest installs, they run but eventually hang when killing guestfs
subprocess is called. Here's the last lines of the imagefactory logs.
2011-11-04 12:44:51,299 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 3: Open up the firewall
2011-11-04 12:44:51,299 INFO oz.Guest.FedoraGuest pid(23842) Message:
Collection Setup
2011-11-04 12:44:51,300 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 4: Guest announcement
2011-11-04 12:44:51,320 INFO oz.Guest.FedoraGuest pid(23842) Message:
Setting up guestfs handle for tmpl1-7dd71f1d-9bc5-43d1-a685-0d8397eb562c
2011-11-04 12:44:51,337 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 5: Set SELinux to permissive mode
2011-11-04 12:44:51,337 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Adding disk image
/var/lib/imagefactory/images/rhevm-image-7dd71f1d-9bc5-43d1-a685-0d8397eb562c.dsk
2011-11-04 12:44:51,356 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Launching guestfs
2011-11-04 12:44:53,974 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Inspecting guest OS
2011-11-04 12:44:55,049 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Getting mountpoints
2011-11-04 12:44:55,049 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Root device: /dev/VolGroup00/LogVol00
2011-11-04 12:44:55,152 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 1: Uploading ssh keys
2011-11-04 12:44:55,156 INFO oz.Guest.FedoraGuest pid(23842) Message:
Cleaning up guestfs handle for tmpl1-207802cf-4f02-4faa-8b28-1d896fd44ca9
2011-11-04 12:44:55,168 INFO oz.Guest.FedoraGuest pid(23842) Message:
Generating new openssh key
2011-11-04 12:44:55,169 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Syncing
2011-11-04 12:44:55,193 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Unmounting all
2011-11-04 12:44:55,195 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 2: setup sshd
2011-11-04 12:44:55,204 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Killing guestfs subprocess
One of the qemu process dies but the other one remains up but doesn't move.
I'm running this on RHEL 6.1 with libguestfs-1.7.17-17.el6.x86_64.
I'm having trouble producing trace output. I tried setting
LIBGUESTFS_TRACE and LIBGUESTFS_DEBUG and since this is through python,
I also tried calling set_trace and set_verbose in this way in
/usr/lib/python2.6/site-packages/oz/Guest.py:
g = guestfs.GuestFS()
g.set_trace(1)
g.set_verbose(1)
Any suggestions here?
12:06 p.m.
On Fri, Nov 04, 2011 at 09:47:43AM -0700, Richard Su wrote:
One of the qemu process dies but the other one remains up but
doesn't move.
I'm running this on RHEL 6.1 with libguestfs-1.7.17-17.el6.x86_64.
You should certainly try to update to the version that
fixes this bug:
>http://bugzilla.redhat.com/684980
http://people.redhat.com/~rjones/libguestfs-RHEL-6.2-preview/
or from the RHEL 6.2 beta,
or Brew.
I'm having trouble producing trace output. I tried setting
LIBGUESTFS_TRACE and LIBGUESTFS_DEBUG and since this is through
python, I also tried calling set_trace and set_verbose in this way
in /usr/lib/python2.6/site-packages/oz/Guest.py:
g = guestfs.GuestFS()
g.set_trace(1)
g.set_verbose(1)
Any suggestions here?
That should be fine. In that old version of libguestfs, the trace
output goes unconditionally to stderr, so you just need to arrange to
capture stderr.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
4739
days inactive
4739
days old
3 comments
3 participants
participants (3)
-
Chris Lalancette -
Richard Su -
Richard W.M. Jones