Using guestfs.kill_subprocess()
Hey Rich, In the past I know you have mentioned that using guestfs.kill_subprocess() isn't strictly necessary after you are finished using libguestfs. I believe you said that in the python bindings, that step would happen automatically when the object goes out of scope and the __del__ method is called. However, since it never seemed to harm anything and I wanted to be extra safe, I always called guestfs.kill_subprocess() in Oz, without apparent harm. However, Richard Su has been investigating an issue where guestfs.kill_subprocess() actually does cause harm. I believe the situation is that running Oz (and hence libguestfs) simultaneously from two threads in the same process causes the entire process to hang. If we remove the guestfs.kill_subprocess(), the problem goes away. rwsu, can you please fill in a bit more detail here, as I'm not intimately familiar with the issue? Things like host OS, libguestfs version, etc etc would be helpful. Also confirmation that the problem is indeed in the multi-threaded environment of the image factory like I think it is. rjones, do you have any ideas here? I'm going to merge rwsu's patch to remove the call to kill_subprocess(), but I thought it would be worth a little more digging into why it was happening. Thanks, Chris Lalancette
On Thu, Nov 03, 2011 at 09:41:40PM -0400, Chris Lalancette wrote:
Hey Rich, In the past I know you have mentioned that using guestfs.kill_subprocess() isn't strictly necessary after you are finished using libguestfs. I believe you said that in the python bindings, that step would happen automatically when the object goes out of scope and the __del__ method is called. However, since it never seemed to harm anything and I wanted to be extra safe, I always called guestfs.kill_subprocess() in Oz, without apparent harm.
Python bindings now have an explicit close call: 8c5bdc3e12947580e91c018b71adf9ad3128bb75 http://bugzilla.redhat.com/717786
However, Richard Su has been investigating an issue where guestfs.kill_subprocess() actually does cause harm. I believe the situation is that running Oz (and hence libguestfs) simultaneously from two threads in the same process causes the entire process to hang. If we remove the guestfs.kill_subprocess(), the problem goes away. rwsu, can you please fill in a bit more detail here, as I'm not intimately familiar with the issue? Things like host OS, libguestfs version, etc etc would be helpful. Also confirmation that the problem is indeed in the multi-threaded environment of the image factory like I think it is. rjones, do you have any ideas here? I'm going to merge rwsu's patch to remove the call to kill_subprocess(), but I thought it would be worth a little more digging into why it was happening.
You could be hitting this bug: http://bugzilla.redhat.com/684980 To debug this I need to see: - version of libguestfs - platform that you're running it on - full output with LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 set Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
On 11/04/2011 01:29 AM, Richard W.M. Jones wrote:
On Thu, Nov 03, 2011 at 09:41:40PM -0400, Chris Lalancette wrote:
Hey Rich, In the past I know you have mentioned that using guestfs.kill_subprocess() isn't strictly necessary after you are finished using libguestfs. I believe you said that in the python bindings, that step would happen automatically when the object goes out of scope and the __del__ method is called. However, since it never seemed to harm anything and I wanted to be extra safe, I always called guestfs.kill_subprocess() in Oz, without apparent harm. Python bindings now have an explicit close call:
8c5bdc3e12947580e91c018b71adf9ad3128bb75 http://bugzilla.redhat.com/717786
However, Richard Su has been investigating an issue where guestfs.kill_subprocess() actually does cause harm. I believe the situation is that running Oz (and hence libguestfs) simultaneously from two threads in the same process causes the entire process to hang. If we remove the guestfs.kill_subprocess(), the problem goes away. rwsu, can you please fill in a bit more detail here, as I'm not intimately familiar with the issue? Things like host OS, libguestfs version, etc etc would be helpful. Also confirmation that the problem is indeed in the multi-threaded environment of the image factory like I think it is. rjones, do you have any ideas here? I'm going to merge rwsu's patch to remove the call to kill_subprocess(), but I thought it would be worth a little more digging into why it was happening.
You could be hitting this bug: http://bugzilla.redhat.com/684980
To debug this I need to see:
- version of libguestfs
- platform that you're running it on
- full output with LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 set
Rich.
The problem I'm running into is when I start two image builds, and thus two guest installs, they run but eventually hang when killing guestfs subprocess is called. Here's the last lines of the imagefactory logs. 2011-11-04 12:44:51,299 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Step 3: Open up the firewall 2011-11-04 12:44:51,299 INFO oz.Guest.FedoraGuest pid(23842) Message: Collection Setup 2011-11-04 12:44:51,300 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Step 4: Guest announcement 2011-11-04 12:44:51,320 INFO oz.Guest.FedoraGuest pid(23842) Message: Setting up guestfs handle for tmpl1-7dd71f1d-9bc5-43d1-a685-0d8397eb562c 2011-11-04 12:44:51,337 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Step 5: Set SELinux to permissive mode 2011-11-04 12:44:51,337 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Adding disk image /var/lib/imagefactory/images/rhevm-image-7dd71f1d-9bc5-43d1-a685-0d8397eb562c.dsk 2011-11-04 12:44:51,356 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Launching guestfs 2011-11-04 12:44:53,974 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Inspecting guest OS 2011-11-04 12:44:55,049 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Getting mountpoints 2011-11-04 12:44:55,049 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Root device: /dev/VolGroup00/LogVol00 2011-11-04 12:44:55,152 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Step 1: Uploading ssh keys 2011-11-04 12:44:55,156 INFO oz.Guest.FedoraGuest pid(23842) Message: Cleaning up guestfs handle for tmpl1-207802cf-4f02-4faa-8b28-1d896fd44ca9 2011-11-04 12:44:55,168 INFO oz.Guest.FedoraGuest pid(23842) Message: Generating new openssh key 2011-11-04 12:44:55,169 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Syncing 2011-11-04 12:44:55,193 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Unmounting all 2011-11-04 12:44:55,195 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Step 2: setup sshd 2011-11-04 12:44:55,204 DEBUG oz.Guest.FedoraGuest pid(23842) Message: Killing guestfs subprocess One of the qemu process dies but the other one remains up but doesn't move. I'm running this on RHEL 6.1 with libguestfs-1.7.17-17.el6.x86_64. I'm having trouble producing trace output. I tried setting LIBGUESTFS_TRACE and LIBGUESTFS_DEBUG and since this is through python, I also tried calling set_trace and set_verbose in this way in /usr/lib/python2.6/site-packages/oz/Guest.py: g = guestfs.GuestFS() g.set_trace(1) g.set_verbose(1) Any suggestions here?
On Fri, Nov 04, 2011 at 09:47:43AM -0700, Richard Su wrote:
One of the qemu process dies but the other one remains up but doesn't move.
I'm running this on RHEL 6.1 with libguestfs-1.7.17-17.el6.x86_64.
You should certainly try to update to the version that fixes this bug:
http://people.redhat.com/~rjones/libguestfs-RHEL-6.2-preview/ or from the RHEL 6.2 beta, or Brew.
I'm having trouble producing trace output. I tried setting LIBGUESTFS_TRACE and LIBGUESTFS_DEBUG and since this is through python, I also tried calling set_trace and set_verbose in this way in /usr/lib/python2.6/site-packages/oz/Guest.py:
g = guestfs.GuestFS() g.set_trace(1) g.set_verbose(1)
Any suggestions here?
That should be fine. In that old version of libguestfs, the trace output goes unconditionally to stderr, so you just need to arrange to capture stderr. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
participants (3)
-
Chris Lalancette -
Richard Su -
Richard W.M. Jones