Re: [Libguestfs] [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Wed, Jan 13, 2016 at 10:18:42AM +0000, Richard W.M. Jones wrote:
As people may know, we frequently encounter errors caused by libvirt
when running the libguestfs appliance.
I wanted to find out exactly how frequently these happen and classify
the errors, so I ran the 'virt-df' tool overnight 1700 times. This
tool runs several parallel qemu:///session libvirt connections both
creating a short-lived appliance guest.
Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183
"XML-RPC error : Cannot write data: Transport endpoint is not connected"
Results:
The test failed 538 times (32% of the time), which is pretty dismal.
To be fair, virt-df is aggressive about how it launches parallel
libvirt connections. Most other virt-* tools use only a single
libvirt connection and are consequently more reliable.
Of the failures, 518 (96%) were of the form:
process exited while connecting to monitor: qemu: could not load kernel
'/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission
denied
which is https://bugzilla.redhat.com/921135 or maybe
https://bugzilla.redhat.com/1269975. It's not clear to me if these
bugs have different causes, but if they do then potentially we're
seeing a mix of both since my test has no way to distinguish them.
It looks to me as the same problem. And as the same problem we were
talking about bunch of time and, apparently, didn't get to a conclusion.
For each of the kernels, libvirt labels them (with both DAC and selinux
labels), then proceeds to launching qemu. If this is done parallel, the
race is pretty obvious. Could you remind me why you couldn't use
<seclabel model='none'/> or <seclabel relabel='no'/> or
something that
would mitigate this? If we cannot use this, then we need to implement
the <seclabel/> element for kernel and initrd.
19 of the failures (4%) were of the form:
process exited while connecting to monitor: fread() failed
which I believe is a previously unknown bug. I have filed it as
https://bugzilla.redhat.com/1298122
I think even this one might be the case, maybe selinux stops qemu from
reading the kernel/initrd.
Finally there was 1 failure:
Unable to read from monitor: Connection reset by peer
which I believe is also a new bug. I have filed it as
https://bugzilla.redhat.com/1298124
This, I believe, means QEMU exited (as in the previous one), just at
different point in time.
I would be good if libvirt could routinely test the case of multiple
parallel launches of qemu:///session, since it still contains bugs
even after Cole's fixes.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Attachments:
- signature.asc (application/pgp-signature — 819 bytes)