4:18 a.m.
As people may know, we frequently encounter errors caused by libvirt
when running the libguestfs appliance.
I wanted to find out exactly how frequently these happen and classify
the errors, so I ran the 'virt-df' tool overnight 1700 times. This
tool runs several parallel qemu:///session libvirt connections both
creating a short-lived appliance guest.
Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183
"XML-RPC error : Cannot write data: Transport endpoint is not connected"
Results:
The test failed 538 times (32% of the time), which is pretty dismal.
To be fair, virt-df is aggressive about how it launches parallel
libvirt connections. Most other virt-* tools use only a single
libvirt connection and are consequently more reliable.
Of the failures, 518 (96%) were of the form:
process exited while connecting to monitor: qemu: could not load kernel
'/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission
denied
which is https://bugzilla.redhat.com/921135 or maybe
https://bugzilla.redhat.com/1269975. It's not clear to me if these
bugs have different causes, but if they do then potentially we're
seeing a mix of both since my test has no way to distinguish them.
19 of the failures (4%) were of the form:
process exited while connecting to monitor: fread() failed
which I believe is a previously unknown bug. I have filed it as
https://bugzilla.redhat.com/1298122
Finally there was 1 failure:
Unable to read from monitor: Connection reset by peer
which I believe is also a new bug. I have filed it as
https://bugzilla.redhat.com/1298124
I would be good if libvirt could routinely test the case of multiple
parallel launches of qemu:///session, since it still contains bugs
even after Cole's fixes.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
6:28 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On 2016-01-13 10:18, Richard W.M. Jones wrote:
<snip>
I would be good if libvirt could routinely test the case of multiple
parallel launches of qemu:///session, since it still contains bugs
even after Cole's fixes.
Sounds like this testing script would be useful as a (weekly?) cronjob
or similar. :)
+ Justin
2:07 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Wed, Jan 13, 2016 at 2:28 PM, Justin Clift <justin(a)postgresql.org> wrote:
On 2016-01-13 10:18, Richard W.M. Jones wrote:
<snip>
> I would be good if libvirt could routinely test the case of multiple
> parallel launches of qemu:///session, since it still contains bugs
> even after Cole's fixes.
>
Sounds like this testing script would be useful as a (weekly?) cronjob or
similar. :)
I've actually seen such issues while running Lago[1].
It launches multiple (5 or so) virt-* tools.
[1] https://github.com/oVirt/lago
Y.
+ Justin
9:25 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Wed, Jan 13, 2016 at 10:18:42AM +0000, Richard W.M. Jones wrote:
As people may know, we frequently encounter errors caused by libvirt
when running the libguestfs appliance.
I wanted to find out exactly how frequently these happen and classify
the errors, so I ran the 'virt-df' tool overnight 1700 times. This
tool runs several parallel qemu:///session libvirt connections both
creating a short-lived appliance guest.
Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183
"XML-RPC error : Cannot write data: Transport endpoint is not connected"
Results:
The test failed 538 times (32% of the time), which is pretty dismal.
To be fair, virt-df is aggressive about how it launches parallel
libvirt connections. Most other virt-* tools use only a single
libvirt connection and are consequently more reliable.
Of the failures, 518 (96%) were of the form:
process exited while connecting to monitor: qemu: could not load kernel
'/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission
denied
which is https://bugzilla.redhat.com/921135 or maybe
https://bugzilla.redhat.com/1269975. It's not clear to me if these
bugs have different causes, but if they do then potentially we're
seeing a mix of both since my test has no way to distinguish them.
It looks to me as the same problem. And as the same problem we were
talking about bunch of time and, apparently, didn't get to a conclusion.
For each of the kernels, libvirt labels them (with both DAC and selinux
labels), then proceeds to launching qemu. If this is done parallel, the
race is pretty obvious. Could you remind me why you couldn't use
<seclabel model='none'/> or <seclabel relabel='no'/> or
something that
would mitigate this? If we cannot use this, then we need to implement
the <seclabel/> element for kernel and initrd.
19 of the failures (4%) were of the form:
process exited while connecting to monitor: fread() failed
which I believe is a previously unknown bug. I have filed it as
https://bugzilla.redhat.com/1298122
I think even this one might be the case, maybe selinux stops qemu from
reading the kernel/initrd.
Finally there was 1 failure:
Unable to read from monitor: Connection reset by peer
which I believe is also a new bug. I have filed it as
https://bugzilla.redhat.com/1298124
This, I believe, means QEMU exited (as in the previous one), just at
different point in time.
I would be good if libvirt could routinely test the case of multiple
parallel launches of qemu:///session, since it still contains bugs
even after Cole's fixes.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
9:50 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Wed, Jan 13, 2016 at 04:25:14PM +0100, Martin Kletzander wrote:
For each of the kernels, libvirt labels them (with both DAC and
selinux
labels), then proceeds to launching qemu. If this is done parallel, the
race is pretty obvious. Could you remind me why you couldn't use
<seclabel model='none'/> or <seclabel relabel='no'/> or
something that
would mitigate this?
We value having sVirt :-)
However I'm just about to rerun the tests with <seclabel type='none'/>
to see if the problem goes away. Will let you know tomorrow once they
have run again.
If we cannot use this, then we need to implement
the <seclabel/> element for kernel and initrd.
Right, that could work for us I think.
>19 of the failures (4%) were of the form:
>
> process exited while connecting to monitor: fread() failed
>
>which I believe is a previously unknown bug. I have filed it as
>https://bugzilla.redhat.com/1298122
>
I think even this one might be the case, maybe selinux stops qemu from
reading the kernel/initrd.
>Finally there was 1 failure:
>
> Unable to read from monitor: Connection reset by peer
>
>which I believe is also a new bug. I have filed it as
>https://bugzilla.redhat.com/1298124
>
This, I believe, means QEMU exited (as in the previous one), just at
different point in time.
OK - let's see if they go away when we get the kernel
thing fixed.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3:42 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Wed, Jan 13, 2016 at 03:50:47PM +0000, Richard W.M. Jones wrote:
On Wed, Jan 13, 2016 at 04:25:14PM +0100, Martin Kletzander wrote:
> For each of the kernels, libvirt labels them (with both DAC and selinux
> labels), then proceeds to launching qemu. If this is done parallel, the
> race is pretty obvious. Could you remind me why you couldn't use
> <seclabel model='none'/> or <seclabel relabel='no'/> or
something that
> would mitigate this?
We value having sVirt :-)
However I'm just about to rerun the tests with <seclabel type='none'/>
to see if the problem goes away. Will let you know tomorrow once they
have run again.
Yes - using <seclabel type=none> does prevent all of the errors. Of
course this isn't a solution because we want to use sVirt.
> >https://bugzilla.redhat.com/1298122
> >https://bugzilla.redhat.com/1298124
I will mark these as duplicates.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
3:51 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Wed, Jan 13, 2016 at 16:25:14 +0100, Martin Kletzander wrote:
On Wed, Jan 13, 2016 at 10:18:42AM +0000, Richard W.M. Jones wrote:
>As people may know, we frequently encounter errors caused by libvirt
>when running the libguestfs appliance.
>
>I wanted to find out exactly how frequently these happen and classify
>the errors, so I ran the 'virt-df' tool overnight 1700 times. This
>tool runs several parallel qemu:///session libvirt connections both
>creating a short-lived appliance guest.
>
>Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183
>"XML-RPC error : Cannot write data: Transport endpoint is not connected"
>
>Results:
>
>The test failed 538 times (32% of the time), which is pretty dismal.
>To be fair, virt-df is aggressive about how it launches parallel
>libvirt connections. Most other virt-* tools use only a single
>libvirt connection and are consequently more reliable.
>
>Of the failures, 518 (96%) were of the form:
>
> process exited while connecting to monitor: qemu: could not load kernel
'/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission
denied
>
>which is https://bugzilla.redhat.com/921135 or maybe
>https://bugzilla.redhat.com/1269975. It's not clear to me if these
>bugs have different causes, but if they do then potentially we're
>seeing a mix of both since my test has no way to distinguish them.
>
It looks to me as the same problem. And as the same problem we were
talking about bunch of time and, apparently, didn't get to a conclusion.
For each of the kernels, libvirt labels them (with both DAC and selinux
labels), then proceeds to launching qemu. If this is done parallel, the
race is pretty obvious. Could you remind me why you couldn't use
<seclabel model='none'/> or <seclabel relabel='no'/> or
something that
would mitigate this? If we cannot use this, then we need to implement
the <seclabel/> element for kernel and initrd.
Hmm, can't we just label kernel and initrd files the same way we label
<shareable/> disk images, i.e., non-exclusive label so that all QEMU
process can access them and avoid removing the label once a domain
disappears?
Jirka
4:12 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Thu, Jan 14, 2016 at 10:51:47AM +0100, Jiri Denemark wrote:
On Wed, Jan 13, 2016 at 16:25:14 +0100, Martin Kletzander wrote:
> On Wed, Jan 13, 2016 at 10:18:42AM +0000, Richard W.M. Jones wrote:
> >As people may know, we frequently encounter errors caused by libvirt
> >when running the libguestfs appliance.
> >
> >I wanted to find out exactly how frequently these happen and classify
> >the errors, so I ran the 'virt-df' tool overnight 1700 times. This
> >tool runs several parallel qemu:///session libvirt connections both
> >creating a short-lived appliance guest.
> >
> >Note that I have added Cole's patch to fix
https://bugzilla.redhat.com/1271183
> >"XML-RPC error : Cannot write data: Transport endpoint is not
connected"
> >
> >Results:
> >
> >The test failed 538 times (32% of the time), which is pretty dismal.
> >To be fair, virt-df is aggressive about how it launches parallel
> >libvirt connections. Most other virt-* tools use only a single
> >libvirt connection and are consequently more reliable.
> >
> >Of the failures, 518 (96%) were of the form:
> >
> > process exited while connecting to monitor: qemu: could not load kernel
'/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission
denied
> >
> >which is https://bugzilla.redhat.com/921135 or maybe
> >https://bugzilla.redhat.com/1269975. It's not clear to me if these
> >bugs have different causes, but if they do then potentially we're
> >seeing a mix of both since my test has no way to distinguish them.
> >
>
> It looks to me as the same problem. And as the same problem we were
> talking about bunch of time and, apparently, didn't get to a conclusion.
>
> For each of the kernels, libvirt labels them (with both DAC and selinux
> labels), then proceeds to launching qemu. If this is done parallel, the
> race is pretty obvious. Could you remind me why you couldn't use
> <seclabel model='none'/> or <seclabel relabel='no'/> or
something that
> would mitigate this? If we cannot use this, then we need to implement
> the <seclabel/> element for kernel and initrd.
Hmm, can't we just label kernel and initrd files the same way we label
<shareable/> disk images, i.e., non-exclusive label so that all QEMU
process can access them and avoid removing the label once a domain
disappears?
We actually should treat it in the same way as <readonly/> disks,
and give it a shared read-only label. And indeed we *do* that.
The difference comes in the restore step - where we blow away the
readonly label and put it back to the original. For disks we never
restore readonly/shared labels, but for kernels we do. If we just
kill the restore step for kernels too, we should be fine AFAICT.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4:24 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Thu, Jan 14, 2016 at 10:12:30AM +0000, Daniel P. Berrange wrote:
The difference comes in the restore step - where we blow away the
readonly label and put it back to the original. For disks we never
restore readonly/shared labels, but for kernels we do. If we just
kill the restore step for kernels too, we should be fine AFAICT.
Works for me - I can try a patch, or if you can point me at the code I
should comment out I'll do that.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
4:42 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Thu, Jan 14, 2016 at 10:24:45AM +0000, Richard W.M. Jones wrote:
On Thu, Jan 14, 2016 at 10:12:30AM +0000, Daniel P. Berrange wrote:
> The difference comes in the restore step - where we blow away the
> readonly label and put it back to the original. For disks we never
> restore readonly/shared labels, but for kernels we do. If we just
> kill the restore step for kernels too, we should be fine AFAICT.
Works for me - I can try a patch, or if you can point me at the code I
should comment out I'll do that.
Is security_selinux.c in the virSecuritySELinuxRestoreAllLabel method,
trying commenting out the lines related to kernel + initrd at the
end.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
9:35 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On 01/14/2016 05:12 AM, Daniel P. Berrange wrote:
On Thu, Jan 14, 2016 at 10:51:47AM +0100, Jiri Denemark wrote:
> On Wed, Jan 13, 2016 at 16:25:14 +0100, Martin Kletzander wrote:
>> On Wed, Jan 13, 2016 at 10:18:42AM +0000, Richard W.M. Jones wrote:
>>> As people may know, we frequently encounter errors caused by libvirt
>>> when running the libguestfs appliance.
>>>
>>> I wanted to find out exactly how frequently these happen and classify
>>> the errors, so I ran the 'virt-df' tool overnight 1700 times. This
>>> tool runs several parallel qemu:///session libvirt connections both
>>> creating a short-lived appliance guest.
>>>
>>> Note that I have added Cole's patch to fix
https://bugzilla.redhat.com/1271183
>>> "XML-RPC error : Cannot write data: Transport endpoint is not
connected"
>>>
>>> Results:
>>>
>>> The test failed 538 times (32% of the time), which is pretty dismal.
>>> To be fair, virt-df is aggressive about how it launches parallel
>>> libvirt connections. Most other virt-* tools use only a single
>>> libvirt connection and are consequently more reliable.
>>>
>>> Of the failures, 518 (96%) were of the form:
>>>
>>> process exited while connecting to monitor: qemu: could not load kernel
'/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission
denied
>>>
>>> which is https://bugzilla.redhat.com/921135 or maybe
>>> https://bugzilla.redhat.com/1269975. It's not clear to me if these
>>> bugs have different causes, but if they do then potentially we're
>>> seeing a mix of both since my test has no way to distinguish them.
>>>
>>
>> It looks to me as the same problem. And as the same problem we were
>> talking about bunch of time and, apparently, didn't get to a conclusion.
>>
>> For each of the kernels, libvirt labels them (with both DAC and selinux
>> labels), then proceeds to launching qemu. If this is done parallel, the
>> race is pretty obvious. Could you remind me why you couldn't use
>> <seclabel model='none'/> or <seclabel relabel='no'/>
or something that
>> would mitigate this? If we cannot use this, then we need to implement
>> the <seclabel/> element for kernel and initrd.
>
> Hmm, can't we just label kernel and initrd files the same way we label
> <shareable/> disk images, i.e., non-exclusive label so that all QEMU
> process can access them and avoid removing the label once a domain
> disappears?
We actually should treat it in the same way as <readonly/> disks,
and give it a shared read-only label. And indeed we *do* that.
The difference comes in the restore step - where we blow away the
readonly label and put it back to the original. For disks we never
restore readonly/shared labels, but for kernels we do. If we just
kill the restore step for kernels too, we should be fine AFAICT.
Indeed I forgot we don't restore labels on readonly/shareable disks..
certainly kernel/initrd should match that.
- Cole
9:54 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
FYI I'm testing the attached patch (written by Jiri Denemark). The
tests usually take a day to run.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
3:48 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Thu, Jan 14, 2016 at 03:54:29PM +0000, Richard W.M. Jones wrote:
FYI I'm testing the attached patch (written by Jiri Denemark). The
tests usually take a day to run.
Yes this patch appears to fix all the problems. Can it
be applied to libvirt?
Rich.
diff --git i/src/security/security_dac.c
w/src/security/security_dac.c
index 80709fe..378b922 100644
--- i/src/security/security_dac.c
+++ w/src/security/security_dac.c
@@ -1128,14 +1128,6 @@ virSecurityDACRestoreAllLabel(virSecurityManagerPtr mgr,
virSecurityDACRestoreFileLabel(priv, def->os.loader->nvram) < 0)
rc = -1;
- if (def->os.kernel &&
- virSecurityDACRestoreFileLabel(priv, def->os.kernel) < 0)
- rc = -1;
-
- if (def->os.initrd &&
- virSecurityDACRestoreFileLabel(priv, def->os.initrd) < 0)
- rc = -1;
-
if (def->os.dtb &&
virSecurityDACRestoreFileLabel(priv, def->os.dtb) < 0)
rc = -1;
diff --git i/src/security/security_selinux.c w/src/security/security_selinux.c
index 721c451..475cdbc 100644
--- i/src/security/security_selinux.c
+++ w/src/security/security_selinux.c
@@ -2034,14 +2034,6 @@ virSecuritySELinuxRestoreAllLabel(virSecurityManagerPtr mgr,
virSecuritySELinuxRestoreFileLabel(mgr, def->os.loader->nvram) < 0)
rc = -1;
- if (def->os.kernel &&
- virSecuritySELinuxRestoreFileLabel(mgr, def->os.kernel) < 0)
- rc = -1;
-
- if (def->os.initrd &&
- virSecuritySELinuxRestoreFileLabel(mgr, def->os.initrd) < 0)
- rc = -1;
-
if (def->os.dtb &&
virSecuritySELinuxRestoreFileLabel(mgr, def->os.dtb) < 0)
rc = -1;
_______________________________________________
Libguestfs mailing list
Libguestfs(a)redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3:52 a.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On Fri, Jan 15, 2016 at 09:48:01 +0000, Richard W.M. Jones wrote:
On Thu, Jan 14, 2016 at 03:54:29PM +0000, Richard W.M. Jones wrote:
>
> FYI I'm testing the attached patch (written by Jiri Denemark). The
> tests usually take a day to run.
Yes this patch appears to fix all the problems. Can it
be applied to libvirt?
Thanks for testing it. I'll make it a proper patch and send it for
review to be applied before 1.3.2 gets out.
Jirka
5:45 p.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On 01/13/2016 05:18 AM, Richard W.M. Jones wrote:
As people may know, we frequently encounter errors caused by libvirt
when running the libguestfs appliance.
I wanted to find out exactly how frequently these happen and classify
the errors, so I ran the 'virt-df' tool overnight 1700 times. This
tool runs several parallel qemu:///session libvirt connections both
creating a short-lived appliance guest.
Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183
"XML-RPC error : Cannot write data: Transport endpoint is not connected"
Results:
The test failed 538 times (32% of the time), which is pretty dismal.
To be fair, virt-df is aggressive about how it launches parallel
libvirt connections. Most other virt-* tools use only a single
libvirt connection and are consequently more reliable.
Of the failures, 518 (96%) were of the form:
process exited while connecting to monitor: qemu: could not load kernel
'/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission
denied
which is https://bugzilla.redhat.com/921135 or maybe
https://bugzilla.redhat.com/1269975. It's not clear to me if these
bugs have different causes, but if they do then potentially we're
seeing a mix of both since my test has no way to distinguish them.
I just experimented with this, I think it's the issue I suggested at:
https://bugzilla.redhat.com/show_bug.cgi?id=1269975#c4
I created two VMs, kernel1 and kernel2, just booting off a kernel in
$HOME/session-kernel/vmlinuz. Then I added this patch:
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index f083f3f..5d9f0fa 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -4901,6 +4901,13 @@ qemuProcessLaunch(virConnectPtr conn,
incoming ? incoming->path : NULL) < 0)
goto cleanup;
+ if (STREQ(vm->def->name, "kernel1")) {
+ for (int z = 0; z < 30; z++) {
+ printf("kernel1: sleeping %d of 30\n", z + 1);
+ sleep(1);
+ }
+ }
+
/* Security manager labeled all devices, therefore
* if any operation from now on fails, we need to ask the caller to
* restore labels.
Which is right after selinux labels are set on VM startup. This is then easy
to reproduce with:
virsh start kernel1 (sleeps)
virsh start kernel2 && virsh destroy kernel2
The shared vmlinuz is reset to user_home_t after kernel2 is shut down, so
kernel1 fails to start after the patch's timeout
When we detect similar issues with <disk> devices, like when the media already
has the expected label, we encode 'relabel=no' in the disk XML, which tells
libvirt not to run restorecon on the disks path when the VM is shutdown.
However kernel/initrd XML doesn't have support for this XML, so it won't work
there. Adding that could be one fix.
But I think there's longer term plans for this type of issue by using ACLs, or
virtlockd or something, Michal had patches but I don't know the specifics.
Unfortunately even hardlinks share selinux labels so I don't think there's any
workaround on the libguestfs side short of using a separate copy of the
appliance kernel for each VM
- Cole
5:48 p.m.
New subject: [libvirt] Quantifying libvirt errors in launching the libguestfs appliance
On 01/13/2016 06:45 PM, Cole Robinson wrote:
On 01/13/2016 05:18 AM, Richard W.M. Jones wrote:
> As people may know, we frequently encounter errors caused by libvirt
> when running the libguestfs appliance.
>
> I wanted to find out exactly how frequently these happen and classify
> the errors, so I ran the 'virt-df' tool overnight 1700 times. This
> tool runs several parallel qemu:///session libvirt connections both
> creating a short-lived appliance guest.
>
> Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183
> "XML-RPC error : Cannot write data: Transport endpoint is not connected"
>
> Results:
>
> The test failed 538 times (32% of the time), which is pretty dismal.
> To be fair, virt-df is aggressive about how it launches parallel
> libvirt connections. Most other virt-* tools use only a single
> libvirt connection and are consequently more reliable.
>
> Of the failures, 518 (96%) were of the form:
>
> process exited while connecting to monitor: qemu: could not load kernel
'/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission
denied
>
> which is https://bugzilla.redhat.com/921135 or maybe
> https://bugzilla.redhat.com/1269975. It's not clear to me if these
> bugs have different causes, but if they do then potentially we're
> seeing a mix of both since my test has no way to distinguish them.
>
I just experimented with this, I think it's the issue I suggested at:
https://bugzilla.redhat.com/show_bug.cgi?id=1269975#c4
I created two VMs, kernel1 and kernel2, just booting off a kernel in
$HOME/session-kernel/vmlinuz. Then I added this patch:
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index f083f3f..5d9f0fa 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -4901,6 +4901,13 @@ qemuProcessLaunch(virConnectPtr conn,
incoming ? incoming->path : NULL) < 0)
goto cleanup;
+ if (STREQ(vm->def->name, "kernel1")) {
+ for (int z = 0; z < 30; z++) {
+ printf("kernel1: sleeping %d of 30\n", z + 1);
+ sleep(1);
+ }
+ }
+
/* Security manager labeled all devices, therefore
* if any operation from now on fails, we need to ask the caller to
* restore labels.
Which is right after selinux labels are set on VM startup. This is then easy
to reproduce with:
virsh start kernel1 (sleeps)
virsh start kernel2 && virsh destroy kernel2
The shared vmlinuz is reset to user_home_t after kernel2 is shut down, so
kernel1 fails to start after the patch's timeout
When we detect similar issues with <disk> devices, like when the media already
has the expected label, we encode 'relabel=no' in the disk XML, which tells
libvirt not to run restorecon on the disks path when the VM is shutdown.
However kernel/initrd XML doesn't have support for this XML, so it won't work
there. Adding that could be one fix.
But I think there's longer term plans for this type of issue by using ACLs, or
virtlockd or something, Michal had patches but I don't know the specifics.
Unfortunately even hardlinks share selinux labels so I don't think there's any
workaround on the libguestfs side short of using a separate copy of the
appliance kernel for each VM
Whoops, should have checked my libvirt mail first, you guys already came to
this conclusion elsewhere in the thread :)
- Cole
3215
days inactive
3217
days old
15 comments
7 participants
participants (7)
-
Cole Robinson -
Daniel P. Berrange -
Jiri Denemark -
Justin Clift -
Martin Kletzander -
Richard W.M. Jones -
Yaniv Kaul