11:47 a.m.
On 11/04/2011 01:29 AM, Richard W.M. Jones wrote:
On Thu, Nov 03, 2011 at 09:41:40PM -0400, Chris Lalancette wrote:
> Hey Rich,
> In the past I know you have mentioned that using
> guestfs.kill_subprocess() isn't strictly necessary after you are finished
> using libguestfs. I believe you said that in the python bindings, that
> step would happen automatically when the object goes out of scope and the
> __del__ method is called. However, since it never seemed to harm anything
> and I wanted to be extra safe, I always called guestfs.kill_subprocess() in
> Oz, without apparent harm.
Python bindings now have an explicit close call:
8c5bdc3e12947580e91c018b71adf9ad3128bb75
http://bugzilla.redhat.com/717786
> However, Richard Su has been investigating an issue where
> guestfs.kill_subprocess() actually does cause harm. I believe the
> situation is that running Oz (and hence libguestfs) simultaneously from two
> threads in the same process causes the entire process to hang. If we
> remove the guestfs.kill_subprocess(), the problem goes away. rwsu, can you
> please fill in a bit more detail here, as I'm not intimately familiar with
> the issue? Things like host OS, libguestfs version, etc etc would be
> helpful. Also confirmation that the problem is indeed in the
> multi-threaded environment of the image factory like I think it is.
> rjones, do you have any ideas here? I'm going to merge rwsu's patch
> to remove the call to kill_subprocess(), but I thought it would be worth a
> little more digging into why it was happening.
You could be hitting this bug:
http://bugzilla.redhat.com/684980
To debug this I need to see:
- version of libguestfs
- platform that you're running it on
- full output with LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 set
Rich.
The problem I'm running into is when I start two image builds, and thus
two guest installs, they run but eventually hang when killing guestfs
subprocess is called. Here's the last lines of the imagefactory logs.
2011-11-04 12:44:51,299 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 3: Open up the firewall
2011-11-04 12:44:51,299 INFO oz.Guest.FedoraGuest pid(23842) Message:
Collection Setup
2011-11-04 12:44:51,300 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 4: Guest announcement
2011-11-04 12:44:51,320 INFO oz.Guest.FedoraGuest pid(23842) Message:
Setting up guestfs handle for tmpl1-7dd71f1d-9bc5-43d1-a685-0d8397eb562c
2011-11-04 12:44:51,337 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 5: Set SELinux to permissive mode
2011-11-04 12:44:51,337 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Adding disk image
/var/lib/imagefactory/images/rhevm-image-7dd71f1d-9bc5-43d1-a685-0d8397eb562c.dsk
2011-11-04 12:44:51,356 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Launching guestfs
2011-11-04 12:44:53,974 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Inspecting guest OS
2011-11-04 12:44:55,049 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Getting mountpoints
2011-11-04 12:44:55,049 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Root device: /dev/VolGroup00/LogVol00
2011-11-04 12:44:55,152 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 1: Uploading ssh keys
2011-11-04 12:44:55,156 INFO oz.Guest.FedoraGuest pid(23842) Message:
Cleaning up guestfs handle for tmpl1-207802cf-4f02-4faa-8b28-1d896fd44ca9
2011-11-04 12:44:55,168 INFO oz.Guest.FedoraGuest pid(23842) Message:
Generating new openssh key
2011-11-04 12:44:55,169 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Syncing
2011-11-04 12:44:55,193 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Unmounting all
2011-11-04 12:44:55,195 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Step 2: setup sshd
2011-11-04 12:44:55,204 DEBUG oz.Guest.FedoraGuest pid(23842) Message:
Killing guestfs subprocess
One of the qemu process dies but the other one remains up but doesn't move.
I'm running this on RHEL 6.1 with libguestfs-1.7.17-17.el6.x86_64.
I'm having trouble producing trace output. I tried setting
LIBGUESTFS_TRACE and LIBGUESTFS_DEBUG and since this is through python,
I also tried calling set_trace and set_verbose in this way in
/usr/lib/python2.6/site-packages/oz/Guest.py:
g = guestfs.GuestFS()
g.set_trace(1)
g.set_verbose(1)
Any suggestions here?