3:02 a.m.
François, thanks for the patches so far.
This morning I pushed:
https://github.com/libguestfs/nbdkit/commit/e26fb6cea3fdba13b5472daf6767e...
build: add missing #include "byte-swapping.h"
https://github.com/libguestfs/nbdkit/commit/b962272a561bfed087316d24b7d51...
build: more fallbacks for mkostemp
The iconv patch makes iconv a requirement. I don't know how many
systems actually lack iconv -- probably not very many -- but even so I
don't want nbdkit to require a library just to compile a single
plugin, as a feature of nbdkit is that the minimal set of
requirements. So I have modified your patch (see attachment) to make
the iconv dependency optional. Can you let me know if this works on
Haiku or if I broke anything?
The final patch was:
https://github.com/mmuman/nbdkit/commit/50ee315525258b0defae06f6e29da841d...
build: avoid hardcoding /var/tmp by default, use P_tmpdir
Unfortunately this won't work well on some Linux systems which use the
/tmp-on-tmpdir misfeature. On these systems, /tmp has only a small,
limited amount of space and is backed by memory, and we must use
/var/tmp for large files. I don't know how to fix this. Perhaps a
simple:
AS_CASE([$host_os],
[haiku*], [LARGE_TMPDIR=/tmp]
[LARGE_TMPDIR=/var/tmp]
)
AC_DEFINE_UNQUOTED([LARGE_TMPDIR],["$LARGE_TMPDIR"],
[Temporary directory for large files])
would be enough for now?
Finally about the list of failing tests:
https://pastebin.com/1PFG85DS
Mostly these are because of the old qemu-io binary. The qemu-io -f
option was added 4 years ago :-/ so it could be time to upgrade.
This error:
./test-shebang-perl.sh: PID file was not created
rect ELF header
FAIL test-shebang-perl.sh (exit status: 1)
(I think the error message is "incorrect ELF header") could be
something to do with the way we use shebangs (#!). IIRC this also
fails on FreeBSD so probably nbdkit is doing something which is not
correct POSIX.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
6:44 p.m.
Hi,
Le 06/11/2018 à 10:02, Richard W.M. Jones a écrit :
François, thanks for the patches so far.
This morning I pushed:
https://github.com/libguestfs/nbdkit/commit/e26fb6cea3fdba13b5472daf6767e...
build: add missing #include "byte-swapping.h"
https://github.com/libguestfs/nbdkit/commit/b962272a561bfed087316d24b7d51...
build: more fallbacks for mkostemp
The iconv patch makes iconv a requirement. I don't know how many
systems actually lack iconv -- probably not very many -- but even so I
don't want nbdkit to require a library just to compile a single
plugin, as a feature of nbdkit is that the minimal set of
requirements. So I have modified your patch (see attachment) to make
the iconv dependency optional. Can you let me know if this works on
Haiku or if I broke anything?
Seems to work.
The final patch was:
https://github.com/mmuman/nbdkit/commit/50ee315525258b0defae06f6e29da841d...
build: avoid hardcoding /var/tmp by default, use P_tmpdir
Unfortunately this won't work well on some Linux systems which use the
/tmp-on-tmpdir misfeature. On these systems, /tmp has only a small,
limited amount of space and is backed by memory, and we must use
/var/tmp for large files. I don't know how to fix this. Perhaps a
simple:
AS_CASE([$host_os],
[haiku*], [LARGE_TMPDIR=/tmp]
[LARGE_TMPDIR=/var/tmp]
)
AC_DEFINE_UNQUOTED([LARGE_TMPDIR],["$LARGE_TMPDIR"],
[Temporary directory for large files])
would be enough for now?
That should do.
I made a patch here:
https://github.com/mmuman/nbdkit/commits/tmpfix
Also added one for the loader lib search path environment, which is
different for us.
Finally about the list of failing tests:
https://pastebin.com/1PFG85DS
Mostly these are because of the old qemu-io binary. The qemu-io -f
option was added 4 years ago :-/ so it could be time to upgrade.
Well, our fork is quite old, and QEMU is fast-moving…
This error:
./test-shebang-perl.sh: PID file was not created
rect ELF header
FAIL test-shebang-perl.sh (exit status: 1)
(I think the error message is "incorrect ELF header") could be
something to do with the way we use shebangs (#!). IIRC this also
fails on FreeBSD so probably nbdkit is doing something which is not
correct POSIX.
Yes I believe it gets rejected because the path is relative.
I think FreeBSD also probably rejects non-absolute interpreter paths for
security reasons.
François.
2:48 a.m.
On Fri, Nov 09, 2018 at 01:44:38AM +0100, François Revol wrote:
Le 06/11/2018 à 10:02, Richard W.M. Jones a écrit :
I made a patch here:
https://github.com/mmuman/nbdkit/commits/tmpfix
Also added one for the loader lib search path environment, which is
different for us.
For the tmpdir patch, I changed plugins/example3/example3.c to a
simpler patch which doesn't require the variable stack allocation
(but also doesn't use TMPDIR -- it's only an example program so
I chose to keep the code simple).
I pushed the LIBRARY_PATH patch unchanged.
> This error:
>
> ./test-shebang-perl.sh: PID file was not created
> rect ELF header
> FAIL test-shebang-perl.sh (exit status: 1)
>
> (I think the error message is "incorrect ELF header") could be
> something to do with the way we use shebangs (#!). IIRC this also
> fails on FreeBSD so probably nbdkit is doing something which is not
> correct POSIX.
Yes I believe it gets rejected because the path is relative.
I think FreeBSD also probably rejects non-absolute interpreter paths for
security reasons.
Interesting - I guess this wouldn't affect normal uses of shebangs,
only the tests. This is probably fixable with a bit of effort but my
FreeBSD VM died recently so I have to get that working first.
Thanks,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
7:40 a.m.
On Tue, Nov 06, 2018 at 09:02:17AM +0000, Richard W.M. Jones wrote:
Finally about the list of failing tests:
https://pastebin.com/1PFG85DS
Mostly these are because of the old qemu-io binary. The qemu-io -f
option was added 4 years ago :-/ so it could be time to upgrade.
FWIW this turns out to be a problem in RHEL 7 as well :-( I worked
around it (in RHEL 7 only) by patching the affected tests like this:
# Ancient qemu-io in RHEL 7 doesn't support -f FORMAT option. However
# we can just omit it and the tests still work fine.
for f in tests/*.sh; do
sed -i -e 's/qemu-io -f raw/qemu-io/g' $f
done
This error:
./test-shebang-perl.sh: PID file was not created
rect ELF header
FAIL test-shebang-perl.sh (exit status: 1)
(I think the error message is "incorrect ELF header") could be
something to do with the way we use shebangs (#!). IIRC this also
fails on FreeBSD so probably nbdkit is doing something which is not
correct POSIX.
I reproduced this on FreeBSD. The path used in the tests is indeed
relative:
#!../nbdkit
However replacing this with an absolute path did *not* fix the issue.
FreeBSD seems to not like running a shell script from a shebang,
probably because this is (was?) insecure. However because of the way
our test harness works we really need to run the shell script. As
there seems to be no simple way to fix this for now, I left the bug in
1.8.0.
Finally the way that iconv detection was implemented broke FreeBSD. I
added an interim replacement for this which at least fixes FreeBSD &
Linux, however of couse I did not test Haiku:
https://github.com/libguestfs/nbdkit/commit/acbe7ad89e75efa8eea41d5891bca...
So hopefully nbdkit 1.8.0 works for you, or at least is not completely
broken. If there are any problems then let us know.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
8:09 a.m.
Le 13/11/2018 à 14:40, Richard W.M. Jones a écrit :
On Tue, Nov 06, 2018 at 09:02:17AM +0000, Richard W.M. Jones wrote:
> Finally about the list of failing tests:
>
> https://pastebin.com/1PFG85DS
>
> Mostly these are because of the old qemu-io binary. The qemu-io -f
> option was added 4 years ago :-/ so it could be time to upgrade.
FWIW this turns out to be a problem in RHEL 7 as well :-( I worked
around it (in RHEL 7 only) by patching the affected tests like this:
# Ancient qemu-io in RHEL 7 doesn't support -f FORMAT option. However
# we can just omit it and the tests still work fine.
for f in tests/*.sh; do
sed -i -e 's/qemu-io -f raw/qemu-io/g' $f
done
Possibly use something like this?
qemuioargs="-f raw"
qemu-io -h | grep -- -f > /dev/null 2>&1 || qemuioargs=""
qemu-io $qemuioargs ...
Seems to work here.
I reproduced this on FreeBSD. The path used in the tests is indeed
relative:
#!../nbdkit
However replacing this with an absolute path did *not* fix the issue.
FreeBSD seems to not like running a shell script from a shebang,
probably because this is (was?) insecure. However because of the way
our test harness works we really need to run the shell script. As
there seems to be no simple way to fix this for now, I left the bug in
1.8.0.
Possibly there are other restrictions, like having the interpreter
root-owned, dunno.
Looks like their ports attempt to fix those:
https://www.freebsd.org/doc/en/books/porters-handbook/uses-shebangfix.html
But the page doesn't list the restrictions.
Finally the way that iconv detection was implemented broke FreeBSD. I
added an interim replacement for this which at least fixes FreeBSD &
Linux, however of couse I did not test Haiku:
https://github.com/libguestfs/nbdkit/commit/acbe7ad89e75efa8eea41d5891bca...
Well it doesn't work, because you don't link with the library, so the
floppy plugin will be skipped.
So hopefully nbdkit 1.8.0 works for you, or at least is not
completely
broken. If there are any problems then let us know.
François.
8:18 a.m.
On Tue, Nov 13, 2018 at 03:09:16PM +0100, François Revol wrote:
Le 13/11/2018 à 14:40, Richard W.M. Jones a écrit :
> On Tue, Nov 06, 2018 at 09:02:17AM +0000, Richard W.M. Jones wrote:
>> Finally about the list of failing tests:
>>
>> https://pastebin.com/1PFG85DS
>>
>> Mostly these are because of the old qemu-io binary. The qemu-io -f
>> option was added 4 years ago :-/ so it could be time to upgrade.
>
> FWIW this turns out to be a problem in RHEL 7 as well :-( I worked
> around it (in RHEL 7 only) by patching the affected tests like this:
>
> # Ancient qemu-io in RHEL 7 doesn't support -f FORMAT option. However
> # we can just omit it and the tests still work fine.
> for f in tests/*.sh; do
> sed -i -e 's/qemu-io -f raw/qemu-io/g' $f
> done
>
Possibly use something like this?
qemuioargs="-f raw"
qemu-io -h | grep -- -f > /dev/null 2>&1 || qemuioargs=""
qemu-io $qemuioargs ...
Seems to work here.
Yes we could do something like that, but perhaps with a better name
such as ‘$qemu_io_format_raw_args’.
> I reproduced this on FreeBSD. The path used in the tests is
indeed
> relative:
>
> #!../nbdkit
>
> However replacing this with an absolute path did *not* fix the issue.
> FreeBSD seems to not like running a shell script from a shebang,
> probably because this is (was?) insecure. However because of the way
> our test harness works we really need to run the shell script. As
> there seems to be no simple way to fix this for now, I left the bug in
> 1.8.0.
Possibly there are other restrictions, like having the interpreter
root-owned, dunno.
Looks like their ports attempt to fix those:
https://www.freebsd.org/doc/en/books/porters-handbook/uses-shebangfix.html
But the page doesn't list the restrictions.
>
> Finally the way that iconv detection was implemented broke FreeBSD. I
> added an interim replacement for this which at least fixes FreeBSD &
> Linux, however of couse I did not test Haiku:
>
>
https://github.com/libguestfs/nbdkit/commit/acbe7ad89e75efa8eea41d5891bca...
Well it doesn't work, because you don't link with the library, so the
floppy plugin will be skipped.
Right, that was expected. It's a small regression for Haiku though so
if you can work out a better way then let me know.
However the problem is it's not easy to fix. On FreeBSD by default
they have two <iconv.h> headers, and two iconv libraries, and if they
are mixed up then it all breaks.
Rich.
> So hopefully nbdkit 1.8.0 works for you, or at least is not
completely
> broken. If there are any problems then let us know.
François.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
10:24 a.m.
On Tue, Nov 13, 2018 at 02:18:34PM +0000, Richard W.M. Jones wrote:
On Tue, Nov 13, 2018 at 03:09:16PM +0100, François Revol wrote:
> Possibly use something like this?
>
> qemuioargs="-f raw"
> qemu-io -h | grep -- -f > /dev/null 2>&1 || qemuioargs=""
>
> qemu-io $qemuioargs ...
>
> Seems to work here.
Yes we could do something like that, but perhaps with a better name
such as ‘$qemu_io_format_raw_args’.
I looked into this further. Unfortunately although we are able to
work around the missing ‘-f’ flag using this technique, we are still
no closer to having the tests pass. This old qemu does not support
various other features we need, at least:
- newstyle protocol
- --image-opts parameter
- TLS client
It looks a bit hopeless trying to get the nbdkit tests to work with
this version of qemu, although of course this is only a test and does
not stop nbdkit itself from working (eg. if you use a non-qemu client
or qemu client is running on another machine).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
10:42 a.m.
On 11/13/18 10:24 AM, Richard W.M. Jones wrote:
On Tue, Nov 13, 2018 at 02:18:34PM +0000, Richard W.M. Jones wrote:
> On Tue, Nov 13, 2018 at 03:09:16PM +0100, François Revol wrote:
>> Possibly use something like this?
>>
>> qemuioargs="-f raw"
>> qemu-io -h | grep -- -f > /dev/null 2>&1 || qemuioargs=""
>>
>> qemu-io $qemuioargs ...
>>
>> Seems to work here.
>
> Yes we could do something like that, but perhaps with a better name
> such as ‘$qemu_io_format_raw_args’.
I looked into this further. Unfortunately although we are able to
work around the missing ‘-f’ flag using this technique, we are still
no closer to having the tests pass. This old qemu does not support
various other features we need, at least:
- newstyle protocol
- --image-opts parameter
- TLS client
It looks a bit hopeless trying to get the nbdkit tests to work with
this version of qemu, although of course this is only a test and does
not stop nbdkit itself from working (eg. if you use a non-qemu client
or qemu client is running on another machine).
I'd lean towards skipping tests if qemu-io is too old for how we intend
to use it. You're right that it doesn't affect nbdkit itself, but
skipping is better than failing if we know the test environment is not
going to work.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
9:16 a.m.
On 11/13/18 7:40 AM, Richard W.M. Jones wrote:
> (I think the error message is "incorrect ELF header")
could be
> something to do with the way we use shebangs (#!). IIRC this also
> fails on FreeBSD so probably nbdkit is doing something which is not
> correct POSIX.
POSIX says that #! shebangs are undefined behavior. You're at the mercy
of each operating system, because there is no portable solution across
all systems.
I reproduced this on FreeBSD. The path used in the tests is indeed
relative:
#!../nbdkit
However replacing this with an absolute path did *not* fix the issue.
FreeBSD seems to not like running a shell script from a shebang,
probably because this is (was?) insecure. However because of the way
our test harness works we really need to run the shell script. As
there seems to be no simple way to fix this for now, I left the bug in
1.8.0.
Can you use '#!/bin/sh /path/to/nbdkit'? That way, the shebang is
running a binary (not a script), where the binary in turn runs the
desired script? Or does that run into other limitations (for example,
#! can only portably used with exactly one argument after the
interpreter name; BSD and more recently coreutils have added 'env -S' to
work around that limitation, although it is not yet widespread
portable). But yes, I do seem to recall that #! must point to a binary
interpreter rather than a script of an unspecified language (the fact
that Linux will happily run a script under /bin/sh is not universal).
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
9:37 a.m.
On Tue, Nov 13, 2018 at 09:16:33AM -0600, Eric Blake wrote:
On 11/13/18 7:40 AM, Richard W.M. Jones wrote:
>>(I think the error message is "incorrect ELF header") could be
>>something to do with the way we use shebangs (#!). IIRC this also
>>fails on FreeBSD so probably nbdkit is doing something which is not
>>correct POSIX.
>
POSIX says that #! shebangs are undefined behavior. You're at the
mercy of each operating system, because there is no portable
solution across all systems.
>I reproduced this on FreeBSD. The path used in the tests is indeed
>relative:
>
> #!../nbdkit
>
>However replacing this with an absolute path did *not* fix the issue.
>FreeBSD seems to not like running a shell script from a shebang,
>probably because this is (was?) insecure. However because of the way
>our test harness works we really need to run the shell script. As
>there seems to be no simple way to fix this for now, I left the bug in
>1.8.0.
Can you use '#!/bin/sh /path/to/nbdkit'?
I didn't reproduce the full shebang above, but it's actually:
#!../nbdkit perl
(https://github.com/libguestfs/nbdkit/blob/90c0de1c2518510ae0fd8ce6aa41f2f...)
So I suppose adding the extra parameter will not work.
That way, the shebang is
running a binary (not a script), where the binary in turn runs the
desired script? Or does that run into other limitations (for
example, #! can only portably used with exactly one argument after
the interpreter name; BSD and more recently coreutils have added
'env -S' to work around that limitation, although it is not yet
widespread portable).
This might work. I'll check and get back.
But yes, I do seem to recall that #! must
point to a binary interpreter rather than a script of an unspecified
language (the fact that Linux will happily run a script under
/bin/sh is not universal).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
10:02 a.m.
Either of the following changes works on FreeBSD (the second one seems
to me to be simpler actually):
--- a/tests/shebang.pl
+++ b/tests/shebang.pl
@@ -1,4 +1,4 @@
-#!../nbdkit perl
+#!/usr/bin/env -S bash ../nbdkit perl
--- a/tests/shebang.pl
+++ b/tests/shebang.pl
@@ -1,4 +1,4 @@
-#!../nbdkit perl
+#!../src/nbdkit ../plugins/perl/.libs/nbdkit-perl-plugin.so
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
2200
days inactive
2207
days old
10 comments
3 participants
participants (3)
-
Eric Blake -
François Revol -
Richard W.M. Jones