Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
10:45 a.m.
On Mon, Oct 12, 2020 at 9:05 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
Hi all,
On Mon, Oct 12, 2020 at 5:17 AM <jenkins(a)jenkins.phx.ovirt.org> wrote:
>
> Project:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
> Build:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
Above failed with:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
vdsm.log has:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
2020-10-11 22:05:14,695-0400 INFO (jsonrpc/1) [api.host] FINISH
getJobs return={'jobs': {'05eaea44-7e4c-4442-9926-2bcb696520f1':
{'id': '05eaea44-7e4c-4442-9926-2bcb696520f1', 'status':
'failed',
'description': 'sparsify_volume', 'job_type': 'storage',
'error':
{'code': 100, 'message': 'General Exception: (\'Command
[\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\',
\\\'--in-place\\\',
\\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67\\\']
failed with rc=1 out=b\\\'3/12\\\\n{ "message": "libguestfs error:
guestfs_launch failed.\\\\\\\\nThis usually means the libguestfs
appliance failed to start or crashed.\\\\\\\\nDo:\\\\\\\\n export
LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\\\\\\nand run the command
again. For further information, read:\\\\\\\\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\\\\\nYou
can also run \\\\\\\'libguestfs-test-tool\\\\\\\' and post the
*complete* output\\\\\\\\ninto a bug report or message to the
libguestfs mailing list.", "timestamp":
"2020-10-11T22:05:08.397538670-04:00", "type": "error"
}\\\\n\\\'
err=b"virt-sparsify: error: libguestfs error: guestfs_launch
failed.\\\\nThis usually means the libguestfs appliance failed to
start or crashed.\\\\nDo:\\\\n export LIBGUESTFS_DEBUG=1
LIBGUESTFS_TRACE=1\\\\nand run the command again. For further
information, read:\\\\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\nYou
can also run \\\'libguestfs-test-tool\\\' and post the *complete*
output\\\\ninto a bug report or message to the libguestfs mailing
list.\\\\n\\\\nIf reporting bugs, run virt-sparsify with debugging
enabled and include the \\\\ncomplete output:\\\\n\\\\n virt-sparsify
-v -x [...]\\\\n"\',)'}}}, 'status': {'code': 0,
'message': 'Done'}}
from=::ffff:192.168.201.4,43318,
flow_id=365642f4-2fe2-45df-937a-f4ca435eea38 (api:54)
2020-10-11 22:05:14,695-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer]
Return 'Host.getJobs' in bridge with
{'05eaea44-7e4c-4442-9926-2bcb696520f1': {'id':
'05eaea44-7e4c-4442-9926-2bcb696520f1', 'status': 'failed',
'description': 'sparsify_volume', 'job_type': 'storage',
'error':
{'code': 100, 'message': 'General Exception: (\'Command
[\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\',
\\\'--in-place\\\',
\\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67\\\']
failed with rc=1 out=b\\\'3/12\\\\n{ "message": "libguestfs error:
guestfs_launch failed.\\\\\\\\nThis usually means the libguestfs
appliance failed to start or crashed.\\\\\\\\nDo:\\\\\\\\n export
LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\\\\\\nand run the command
again. For further information, read:\\\\\\\\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\\\\\nYou
can also run \\\\\\\'libguestfs-test-tool\\\\\\\' and post the
*complete* output\\\\\\\\ninto a bug report or message to the
libguestfs mailing list.", "timestamp":
"2020-10-11T22:05:08.397538670-04:00", "type": "error"
}\\\\n\\\'
err=b"virt-sparsify: error: libguestfs error: guestfs_launch
failed.\\\\nThis usually means the libguestfs appliance failed to
start or crashed.\\\\nDo:\\\\n export LIBGUESTFS_DEBUG=1
LIBGUESTFS_TRACE=1\\\\nand run the command again. For further
information, read:\\\\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\\nYou
can also run \\\'libguestfs-test-tool\\\' and post the *complete*
output\\\\ninto a bug report or message to the libguestfs mailing
list.\\\\n\\\\nIf reporting bugs, run virt-sparsify with debugging
enabled and include the \\\\ncomplete output:\\\\n\\\\n virt-sparsify
-v -x [...]\\\\n"\',)'}}} (__init__:356)
/var/log/messages has:
Oct 11 22:04:51 lago-basic-suite-master-host-0 kvm[80601]: 1 guest now active
Oct 11 22:05:06 lago-basic-suite-master-host-0 journal[80557]: Domain
id=1 name='guestfs-hl0ntvn92rtkk2u0'
uuid=05ea5a53-562f-49f8-a8ca-76b45c5325b4 is tainted: custom-argv
Oct 11 22:05:06 lago-basic-suite-master-host-0 journal[80557]: Domain
id=1 name='guestfs-hl0ntvn92rtkk2u0'
uuid=05ea5a53-562f-49f8-a8ca-76b45c5325b4 is tainted: host-cpu
Oct 11 22:05:06 lago-basic-suite-master-host-0 kvm[80801]: 2 guests now active
Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]:
internal error: End of file from qemu monitor
Oct 11 22:05:08 lago-basic-suite-master-host-0 kvm[80807]: 1 guest now active
Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot
resolve symlink /tmp/libguestfseTG8xF/console.sock: No such file or
directoryKN<F3>L^?
Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot
resolve symlink /tmp/libguestfseTG8xF/guestfsd.sock: No such file or
directoryKN<F3>L^?
Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot
lookup default selinux label for /tmp/libguestfs1WkcF7/overlay1.qcow2
Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot
lookup default selinux label for
/var/tmp/.guestfs-36/appliance.d/kernel
Oct 11 22:05:08 lago-basic-suite-master-host-0 journal[80557]: cannot
lookup default selinux label for
/var/tmp/.guestfs-36/appliance.d/initrd
Oct 11 22:05:08 lago-basic-suite-master-host-0 vdsm[74096]: ERROR Job
'05eaea44-7e4c-4442-9926-2bcb696520f1' failed#012Traceback (most
recent call last):#012 File
"/usr/lib/python3.6/site-packages/vdsm/jobs.py", line 159, in run#012
self._run()#012 File
"/usr/lib/python3.6/site-packages/vdsm/storage/sdm/api/sparsify_volume.py",
line 57, in _run#012
virtsparsify.sparsify_inplace(self._vol_info.path)#012 File
"/usr/lib/python3.6/site-packages/vdsm/virtsparsify.py", line 40, in
sparsify_inplace#012 commands.run(cmd)#012 File
"/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 101,
in run#012 raise cmdutils.Error(args, p.returncode, out,
err)#012vdsm.common.cmdutils.Error: Command ['/usr/bin/virt-sparsify',
'--machine-readable', '--in-place',
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67']
failed with rc=1 out=b'3/12\n{ "message": "libguestfs error:
guestfs_launch failed.\\nThis usually means the libguestfs appliance
failed to start or crashed.\\nDo:\\n export LIBGUESTFS_DEBUG=1
LIBGUESTFS_TRACE=1\\nand run the command again. For further
information, read:\\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\nYou
can also run \'libguestfs-test-tool\' and post the *complete*
output\\ninto a bug report or message to the libguestfs mailing
list.", "timestamp": "2020-10-11T22:05:08.397538670-04:00",
"type":
"error" }\n' err=b"virt-sparsify: error: libguestfs error:
guestfs_launch failed.\nThis usually means the libguestfs appliance
failed to start or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1
LIBGUESTFS_TRACE=1\nand run the command again. For further
information, read:\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can
also run 'libguestfs-test-tool' and post the *complete* output\ninto a
bug report or message to the libguestfs mailing list.\n\nIf reporting
bugs, run virt-sparsify with debugging enabled and include the
\ncomplete output:\n\n virt-sparsify -v -x [...]\n"
The next run of the job (480) did finish successfully. No idea if it
was already fixed by a patch, or is simply a random/env issue.
I think this is env issue, we run on overloaded vms with small amount of memory.
I have seen such radnom failures before.
Is it possible to pass LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 without
patching vdsm? Not sure. In any case, if this does not cause too much
excess debug logging, perhaps better always pass it, to help
retroactively analyze such failures.
Unfortunately using LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 creates
huge amount of logs so it is not possible to include these log for every run.
Maybe we can show the logs only if the command failed, but vdsm infrastructure
make this hard to do since we always log commnad stderr, even if the command
was successful.
Also I'm not sure this is a good idea to always run virt-sparsify in
debug level.
I think I posted a patch to run libguestfs-test-tool after virt-something errors
for collecting data, but I cannot find it. Maybe this can be useful,
so after every
virt-something error we will have enough info about the issue in vdsm log.
Or, patch
virt-sparsify/libguestfs/whatever to always log at least enough
information on failure even without passing these.
I think this is the right solution - when virt-something tool fails,
it should log the
reason for the failure - the error that caused the tool to fail. I'm
not sure this is
easy to do as the failing code run inside a special VM. Maybe the code running
in the VM should log the output in a machine readable way, so once an error
is detected virt-something can report the error as the reason, without running
in debug mode.
Nir
11:15 a.m.
New subject: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
I think this is the right solution - when virt-something tool fails,
it should log the reason for the failure - the error that caused the
tool to fail. I'm not sure this is easy to do as the failing code
run inside a special VM. Maybe the code running in the VM should log
the output in a machine readable way, so once an error is detected
virt-something can report the error as the reason, without running
in debug mode.
All the virt-* tools that I've written have a non-zero exit code and
print an error message on stderr when they fail. Errors from inside
the appliance are propagated to the library and thence to the tool
correctly.
I think the best thing to do is:
- spool up stdout + stderr from the tool
- if the exit code != 0, save the spooled output for analysis
- if the exit code == 0, discard it (or keep it if you like)
Without LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE stdout + stderr should be
quite minimal for all tools.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
11:56 a.m.
New subject: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> I think this is the right solution - when virt-something tool fails,
> it should log the reason for the failure - the error that caused the
> tool to fail. I'm not sure this is easy to do as the failing code
> run inside a special VM. Maybe the code running in the VM should log
> the output in a machine readable way, so once an error is detected
> virt-something can report the error as the reason, without running
> in debug mode.
All the virt-* tools that I've written have a non-zero exit code and
print an error message on stderr when they fail. Errors from inside
the appliance are propagated to the library and thence to the tool
correctly.
I think the best thing to do is:
- spool up stdout + stderr from the tool
- if the exit code != 0, save the spooled output for analysis
- if the exit code == 0, discard it (or keep it if you like)
This is what we already do, and the result is not helpful. If you look
at the log message in the previous message, basically the only
info about the error is:
libguestfs error: guestfs_launch failed
I don't see what we can do with this error message.
If this is an error running qemu, the error message from qemu can help
to understand why qemu failed to start.
Without LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE stdout + stderr should be
quite minimal for all tools.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
12:40 p.m.
New subject: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
>
> On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > I think this is the right solution - when virt-something tool fails,
> > it should log the reason for the failure - the error that caused the
> > tool to fail. I'm not sure this is easy to do as the failing code
> > run inside a special VM. Maybe the code running in the VM should log
> > the output in a machine readable way, so once an error is detected
> > virt-something can report the error as the reason, without running
> > in debug mode.
>
> All the virt-* tools that I've written have a non-zero exit code and
> print an error message on stderr when they fail. Errors from inside
> the appliance are propagated to the library and thence to the tool
> correctly.
>
> I think the best thing to do is:
>
> - spool up stdout + stderr from the tool
>
> - if the exit code != 0, save the spooled output for analysis
>
> - if the exit code == 0, discard it (or keep it if you like)
This is what we already do, and the result is not helpful. If you look
at the log message in the previous message, basically the only
info about the error is:
libguestfs error: guestfs_launch failed
I don't see what we can do with this error message.
Right, so in this particular instance the error message would tell us
that you should run libguestfs-test-tool because your qemu/kernel/etc
is broken in some way :-/
There's not a particularly good answer here if you don't want to ever
use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
libguestfs-test-tool if you see any error which matches the substring
/guestfs_launch/ ?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
12:54 a.m.
New subject: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
> >
> > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > I think this is the right solution - when virt-something tool fails,
> > > it should log the reason for the failure - the error that caused the
> > > tool to fail. I'm not sure this is easy to do as the failing code
> > > run inside a special VM. Maybe the code running in the VM should log
> > > the output in a machine readable way, so once an error is detected
> > > virt-something can report the error as the reason, without running
> > > in debug mode.
> >
> > All the virt-* tools that I've written have a non-zero exit code and
> > print an error message on stderr when they fail. Errors from inside
> > the appliance are propagated to the library and thence to the tool
> > correctly.
> >
> > I think the best thing to do is:
> >
> > - spool up stdout + stderr from the tool
> >
> > - if the exit code != 0, save the spooled output for analysis
> >
> > - if the exit code == 0, discard it (or keep it if you like)
>
> This is what we already do, and the result is not helpful. If you look
> at the log message in the previous message, basically the only
> info about the error is:
>
> libguestfs error: guestfs_launch failed
>
> I don't see what we can do with this error message.
Right, so in this particular instance the error message would tell us
that you should run libguestfs-test-tool because your qemu/kernel/etc
is broken in some way :-/
There's not a particularly good answer here if you don't want to ever
use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
libguestfs-test-tool if you see any error which matches the substring
/guestfs_launch/ ?
Another (orthogonal?) option:
Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
(e.g. some other file descriptor, or to a file passed via env or whatever).
This way, it might make sense for vdsm to always pass these vars, continue
logging all stdout/err, and log/keep debug/trace logs only on errors.
Best regards,
--
Didi
1:16 a.m.
New subject: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
>
> On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones <rjones(a)redhat.com>
wrote:
> > >
> > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > I think this is the right solution - when virt-something tool fails,
> > > > it should log the reason for the failure - the error that caused the
> > > > tool to fail. I'm not sure this is easy to do as the failing
code
> > > > run inside a special VM. Maybe the code running in the VM should log
> > > > the output in a machine readable way, so once an error is detected
> > > > virt-something can report the error as the reason, without running
> > > > in debug mode.
> > >
> > > All the virt-* tools that I've written have a non-zero exit code and
> > > print an error message on stderr when they fail. Errors from inside
> > > the appliance are propagated to the library and thence to the tool
> > > correctly.
> > >
> > > I think the best thing to do is:
> > >
> > > - spool up stdout + stderr from the tool
> > >
> > > - if the exit code != 0, save the spooled output for analysis
> > >
> > > - if the exit code == 0, discard it (or keep it if you like)
> >
> > This is what we already do, and the result is not helpful. If you look
> > at the log message in the previous message, basically the only
> > info about the error is:
> >
> > libguestfs error: guestfs_launch failed
> >
> > I don't see what we can do with this error message.
>
> Right, so in this particular instance the error message would tell us
> that you should run libguestfs-test-tool because your qemu/kernel/etc
> is broken in some way :-/
>
> There's not a particularly good answer here if you don't want to ever
> use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> libguestfs-test-tool if you see any error which matches the substring
> /guestfs_launch/ ?
Another (orthogonal?) option:
Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
(e.g. some other file descriptor, or to a file passed via env or whatever).
This way, it might make sense for vdsm to always pass these vars, continue
logging all stdout/err, and log/keep debug/trace logs only on errors.
This now happened again:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
2020-11-09 01:05:42,031-0500 INFO (jsonrpc/4) [api.host] FINISH
getAllVmIoTunePolicies return={'status': {'code': 0, 'message':
'Done'}, 'io_tune_policies_dict':
{'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
'current_values': [{'name': 'vda', 'path':
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}}} from=::1,34002 (api:54)
2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer]
Return 'Host.getAllVmIoTunePolicies' in bridge with
{'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
'current_values': [{'name': 'vda', 'path':
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}} (__init__:360)
2020-11-09 01:05:42,435-0500 DEBUG (tasks/3) [common.commands] FAILED:
<err> = b"virt-sparsify: error: libguestfs error: guestfs_launch
failed.\nThis usually means the libguestfs appliance failed to start
or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand
run the command again. For further information, read:\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can
also run 'libguestfs-test-tool' and post the *complete* output\ninto a
bug report or message to the libguestfs mailing list.\n\nIf reporting
bugs, run virt-sparsify with debugging enabled and include the
\ncomplete output:\n\n virt-sparsify -v -x [...]\n"; <rc> = 1
(commands:98)
I suggest that if we have come to a dead-end and no-one has any clue, then
we either patch something (vdsm?) to allow getting more information if this
happens again, or open a bug for further discussion/prioritization.
Best regards,
--
Didi
2:56 a.m.
New subject: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
Hi,
Not sure if this is the same issue or not, but I opened a thread about
guestfs_launch failing not too long ago.
https://www.redhat.com/archives/libguestfs/2020-August/msg00352.html
Since we are using our own tool, we ended up retrying guestfs_launch
(and also guestfs_open, since the documentation suggests not reusing
handles) - Not sure if this solved the issue since we introduced this
retrial not too long ago.
Sam
On Mon, Nov 9, 2020 at 9:57 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
> >
> > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
> > > >
> > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > > I think this is the right solution - when virt-something tool
fails,
> > > > > it should log the reason for the failure - the error that caused
the
> > > > > tool to fail. I'm not sure this is easy to do as the failing
code
> > > > > run inside a special VM. Maybe the code running in the VM should
log
> > > > > the output in a machine readable way, so once an error is
detected
> > > > > virt-something can report the error as the reason, without
running
> > > > > in debug mode.
> > > >
> > > > All the virt-* tools that I've written have a non-zero exit code
and
> > > > print an error message on stderr when they fail. Errors from inside
> > > > the appliance are propagated to the library and thence to the tool
> > > > correctly.
> > > >
> > > > I think the best thing to do is:
> > > >
> > > > - spool up stdout + stderr from the tool
> > > >
> > > > - if the exit code != 0, save the spooled output for analysis
> > > >
> > > > - if the exit code == 0, discard it (or keep it if you like)
> > >
> > > This is what we already do, and the result is not helpful. If you look
> > > at the log message in the previous message, basically the only
> > > info about the error is:
> > >
> > > libguestfs error: guestfs_launch failed
> > >
> > > I don't see what we can do with this error message.
> >
> > Right, so in this particular instance the error message would tell us
> > that you should run libguestfs-test-tool because your qemu/kernel/etc
> > is broken in some way :-/
> >
> > There's not a particularly good answer here if you don't want to ever
> > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> > libguestfs-test-tool if you see any error which matches the substring
> > /guestfs_launch/ ?
>
> Another (orthogonal?) option:
>
> Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
> (e.g. some other file descriptor, or to a file passed via env or whatever).
> This way, it might make sense for vdsm to always pass these vars, continue
> logging all stdout/err, and log/keep debug/trace logs only on errors.
This now happened again:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
2020-11-09 01:05:42,031-0500 INFO (jsonrpc/4) [api.host] FINISH
getAllVmIoTunePolicies return={'status': {'code': 0, 'message':
'Done'}, 'io_tune_policies_dict':
{'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
'current_values': [{'name': 'vda', 'path':
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}}} from=::1,34002 (api:54)
2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer]
Return 'Host.getAllVmIoTunePolicies' in bridge with
{'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
'current_values': [{'name': 'vda', 'path':
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}} (__init__:360)
2020-11-09 01:05:42,435-0500 DEBUG (tasks/3) [common.commands] FAILED:
<err> = b"virt-sparsify: error: libguestfs error: guestfs_launch
failed.\nThis usually means the libguestfs appliance failed to start
or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand
run the command again. For further information, read:\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can
also run 'libguestfs-test-tool' and post the *complete* output\ninto a
bug report or message to the libguestfs mailing list.\n\nIf reporting
bugs, run virt-sparsify with debugging enabled and include the
\ncomplete output:\n\n virt-sparsify -v -x [...]\n"; <rc> = 1
(commands:98)
I suggest that if we have come to a dead-end and no-one has any clue, then
we either patch something (vdsm?) to allow getting more information if this
happens again, or open a bug for further discussion/prioritization.
Best regards,
--
Didi
_______________________________________________
Libguestfs mailing list
Libguestfs(a)redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
3:03 a.m.
New subject: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Mon, Nov 09, 2020 at 09:16:33AM +0200, Yedidyah Bar David wrote:
On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
>
> On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
> >
> > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
> > > >
> > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > > I think this is the right solution - when virt-something tool
fails,
> > > > > it should log the reason for the failure - the error that caused
the
> > > > > tool to fail. I'm not sure this is easy to do as the failing
code
> > > > > run inside a special VM. Maybe the code running in the VM should
log
> > > > > the output in a machine readable way, so once an error is
detected
> > > > > virt-something can report the error as the reason, without
running
> > > > > in debug mode.
> > > >
> > > > All the virt-* tools that I've written have a non-zero exit code
and
> > > > print an error message on stderr when they fail. Errors from inside
> > > > the appliance are propagated to the library and thence to the tool
> > > > correctly.
> > > >
> > > > I think the best thing to do is:
> > > >
> > > > - spool up stdout + stderr from the tool
> > > >
> > > > - if the exit code != 0, save the spooled output for analysis
> > > >
> > > > - if the exit code == 0, discard it (or keep it if you like)
> > >
> > > This is what we already do, and the result is not helpful. If you look
> > > at the log message in the previous message, basically the only
> > > info about the error is:
> > >
> > > libguestfs error: guestfs_launch failed
> > >
> > > I don't see what we can do with this error message.
> >
> > Right, so in this particular instance the error message would tell us
> > that you should run libguestfs-test-tool because your qemu/kernel/etc
> > is broken in some way :-/
> >
> > There's not a particularly good answer here if you don't want to ever
> > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> > libguestfs-test-tool if you see any error which matches the substring
> > /guestfs_launch/ ?
>
> Another (orthogonal?) option:
>
> Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
> (e.g. some other file descriptor, or to a file passed via env or whatever).
> This way, it might make sense for vdsm to always pass these vars, continue
> logging all stdout/err, and log/keep debug/trace logs only on errors.
This now happened again:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
2020-11-09 01:05:42,031-0500 INFO (jsonrpc/4) [api.host] FINISH
getAllVmIoTunePolicies return={'status': {'code': 0, 'message':
'Done'}, 'io_tune_policies_dict':
{'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
'current_values': [{'name': 'vda', 'path':
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}}} from=::1,34002 (api:54)
2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer]
Return 'Host.getAllVmIoTunePolicies' in bridge with
{'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
'current_values': [{'name': 'vda', 'path':
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}} (__init__:360)
2020-11-09 01:05:42,435-0500 DEBUG (tasks/3) [common.commands] FAILED:
<err> = b"virt-sparsify: error: libguestfs error: guestfs_launch
failed.\nThis usually means the libguestfs appliance failed to start
or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand
run the command again. For further information, read:\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can
also run 'libguestfs-test-tool' and post the *complete* output\ninto a
bug report or message to the libguestfs mailing list.\n\nIf reporting
bugs, run virt-sparsify with debugging enabled and include the
\ncomplete output:\n\n virt-sparsify -v -x [...]\n"; <rc> = 1
(commands:98)
I suggest that if we have come to a dead-end and no-one has any clue, then
we either patch something (vdsm?) to allow getting more information if this
happens again, or open a bug for further discussion/prioritization.
You definitely need to run libguestfs-test-tool from vdsm if we're
going to get any further with this.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
3:29 a.m.
New subject: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Mon, Nov 9, 2020 at 11:03 AM Richard W.M. Jones <rjones(a)redhat.com> wrote:
On Mon, Nov 09, 2020 at 09:16:33AM +0200, Yedidyah Bar David wrote:
> On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
> >
> > On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones <rjones(a)redhat.com>
wrote:
> > >
> > > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
> > > > >
> > > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > > > I think this is the right solution - when virt-something
tool fails,
> > > > > > it should log the reason for the failure - the error that
caused the
> > > > > > tool to fail. I'm not sure this is easy to do as the
failing code
> > > > > > run inside a special VM. Maybe the code running in the VM
should log
> > > > > > the output in a machine readable way, so once an error is
detected
> > > > > > virt-something can report the error as the reason, without
running
> > > > > > in debug mode.
> > > > >
> > > > > All the virt-* tools that I've written have a non-zero exit
code and
> > > > > print an error message on stderr when they fail. Errors from
inside
> > > > > the appliance are propagated to the library and thence to the
tool
> > > > > correctly.
> > > > >
> > > > > I think the best thing to do is:
> > > > >
> > > > > - spool up stdout + stderr from the tool
> > > > >
> > > > > - if the exit code != 0, save the spooled output for analysis
> > > > >
> > > > > - if the exit code == 0, discard it (or keep it if you like)
> > > >
> > > > This is what we already do, and the result is not helpful. If you
look
> > > > at the log message in the previous message, basically the only
> > > > info about the error is:
> > > >
> > > > libguestfs error: guestfs_launch failed
> > > >
> > > > I don't see what we can do with this error message.
> > >
> > > Right, so in this particular instance the error message would tell us
> > > that you should run libguestfs-test-tool because your qemu/kernel/etc
> > > is broken in some way :-/
> > >
> > > There's not a particularly good answer here if you don't want to
ever
> > > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> > > libguestfs-test-tool if you see any error which matches the substring
> > > /guestfs_launch/ ?
> >
> > Another (orthogonal?) option:
> >
> > Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
> > (e.g. some other file descriptor, or to a file passed via env or whatever).
> > This way, it might make sense for vdsm to always pass these vars, continue
> > logging all stdout/err, and log/keep debug/trace logs only on errors.
>
> This now happened again:
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
>
>
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
>
> 2020-11-09 01:05:42,031-0500 INFO (jsonrpc/4) [api.host] FINISH
> getAllVmIoTunePolicies return={'status': {'code': 0,
'message':
> 'Done'}, 'io_tune_policies_dict':
> {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
> 'current_values': [{'name': 'vda', 'path':
>
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
> 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
> 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec':
0,
> 'read_iops_sec': 0}}]}}} from=::1,34002 (api:54)
> 2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer]
> Return 'Host.getAllVmIoTunePolicies' in bridge with
> {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
> 'current_values': [{'name': 'vda', 'path':
>
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
> 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
> 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec':
0,
> 'read_iops_sec': 0}}]}} (__init__:360)
> 2020-11-09 01:05:42,435-0500 DEBUG (tasks/3) [common.commands] FAILED:
> <err> = b"virt-sparsify: error: libguestfs error: guestfs_launch
> failed.\nThis usually means the libguestfs appliance failed to start
> or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand
> run the command again. For further information, read:\n
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can
> also run 'libguestfs-test-tool' and post the *complete* output\ninto a
> bug report or message to the libguestfs mailing list.\n\nIf reporting
> bugs, run virt-sparsify with debugging enabled and include the
> \ncomplete output:\n\n virt-sparsify -v -x [...]\n"; <rc> = 1
> (commands:98)
>
> I suggest that if we have come to a dead-end and no-one has any clue, then
> we either patch something (vdsm?) to allow getting more information if this
> happens again, or open a bug for further discussion/prioritization.
You definitely need to run libguestfs-test-tool from vdsm if we're
going to get any further with this.
Now filed [1] on vdsm. Feel free to comment there about what else we
should do to get more information. Thanks!
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1895843
Best regards,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
--
Didi
1288
days inactive
1315
days old
8 comments
4 participants
participants (4)
-
Nir Soffer -
Richard W.M. Jones -
Sam Eiderman -
Yedidyah Bar David