10:44 a.m.
The gcs plugin uses the google cloud module and can't work without it.
Skip the test if the module is unavailable.
Signed-off-by: Thomas Weißschuh <thomas(a)t-8ch.de>
---
tests/test-gcs.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh
index ed823377dc5a..9b165e8159be 100755
--- a/tests/test-gcs.sh
+++ b/tests/test-gcs.sh
@@ -40,6 +40,7 @@ requires $PYTHON --version
requires_nbdcopy
requires_plugin python
skip_if_valgrind "because Python code leaks memory"
+requires $PYTHON -c 'import google.cloud'
# Skip this test if the real google-cloud-storage module is installed,
# since that module abuses pth files to insert itself into the module
base-commit: ffa62d30fe6dcd29ae8e20df5c86d48505550614
--
2.44.0
11:09 a.m.
On Sun, Apr 28, 2024 at 05:44:53PM +0200, Thomas Weißschuh wrote:
The gcs plugin uses the google cloud module and can't work
without it.
Skip the test if the module is unavailable.
Signed-off-by: Thomas Weißschuh <thomas(a)t-8ch.de>
---
tests/test-gcs.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh
index ed823377dc5a..9b165e8159be 100755
--- a/tests/test-gcs.sh
+++ b/tests/test-gcs.sh
@@ -40,6 +40,7 @@ requires $PYTHON --version
requires_nbdcopy
requires_plugin python
skip_if_valgrind "because Python code leaks memory"
+requires $PYTHON -c 'import google.cloud'
It shouldn't do (I think?) ... Did you see:
https://gitlab.com/nbdkit/nbdkit/-/commit/aca30111632091bbc91096f19638b9e...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
11:22 a.m.
On 2024-04-28 17:09:06+0000, Richard W.M. Jones wrote:
On Sun, Apr 28, 2024 at 05:44:53PM +0200, Thomas Weißschuh wrote:
> The gcs plugin uses the google cloud module and can't work without it.
> Skip the test if the module is unavailable.
>
> Signed-off-by: Thomas Weißschuh <thomas(a)t-8ch.de>
> ---
> tests/test-gcs.sh | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh
> index ed823377dc5a..9b165e8159be 100755
> --- a/tests/test-gcs.sh
> +++ b/tests/test-gcs.sh
> @@ -40,6 +40,7 @@ requires $PYTHON --version
> requires_nbdcopy
> requires_plugin python
> skip_if_valgrind "because Python code leaks memory"
> +requires $PYTHON -c 'import google.cloud'
It shouldn't do (I think?) ... Did you see:
https://gitlab.com/nbdkit/nbdkit/-/commit/aca30111632091bbc91096f19638b9e...
I do have that commit, and I don't have that module installed.
The actual problem is the import of "google.api_core", which itself is
part of google-api-core which I don't have installed.
That is a very narrow band of platforms this test can run on.
Thomas
11:34 a.m.
On Sun, Apr 28, 2024 at 06:22:43PM +0200, Thomas Weißschuh wrote:
On 2024-04-28 17:09:06+0000, Richard W.M. Jones wrote:
> On Sun, Apr 28, 2024 at 05:44:53PM +0200, Thomas Weißschuh wrote:
> > The gcs plugin uses the google cloud module and can't work without it.
> > Skip the test if the module is unavailable.
> >
> > Signed-off-by: Thomas Weißschuh <thomas(a)t-8ch.de>
> > ---
> > tests/test-gcs.sh | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh
> > index ed823377dc5a..9b165e8159be 100755
> > --- a/tests/test-gcs.sh
> > +++ b/tests/test-gcs.sh
> > @@ -40,6 +40,7 @@ requires $PYTHON --version
> > requires_nbdcopy
> > requires_plugin python
> > skip_if_valgrind "because Python code leaks memory"
> > +requires $PYTHON -c 'import google.cloud'
>
> It shouldn't do (I think?) ... Did you see:
>
> https://gitlab.com/nbdkit/nbdkit/-/commit/aca30111632091bbc91096f19638b9e...
I do have that commit, and I don't have that module installed.
The actual problem is the import of "google.api_core", which itself is
part of google-api-core which I don't have installed.
That is a very narrow band of platforms this test can run on.
It should run without any google.* modules installed, and also if any
combination of google.* modules are installed. In all cases it ought
to load the mocked module from tests/test-gcs and ignore the installed
google.* modules, or if it can't do that (see my commit message above)
it should skip the test.
For me, that all works.
However I found last week that there's something seriously weird about
the google.* modules (not to mention Python module loading in
general), so I can believe the test might not work in some
combination.
Nevertheless, the commit as proposed is definitely wrong. The gcs
test does not (or should not) use any installed google.* module, so if
it does then that's a bug of some kind.
What the full log (tests/test-gcs.sh.log) when it fails?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
12:26 p.m.
On 2024-04-28 17:34:42+0000, Richard W.M. Jones wrote:
On Sun, Apr 28, 2024 at 06:22:43PM +0200, Thomas Weißschuh wrote:
> On 2024-04-28 17:09:06+0000, Richard W.M. Jones wrote:
> > On Sun, Apr 28, 2024 at 05:44:53PM +0200, Thomas Weißschuh wrote:
> > > The gcs plugin uses the google cloud module and can't work without
it.
> > > Skip the test if the module is unavailable.
> > >
> > > Signed-off-by: Thomas Weißschuh <thomas(a)t-8ch.de>
> > > ---
> > > tests/test-gcs.sh | 1 +
> > > 1 file changed, 1 insertion(+)
> > >
> > > diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh
> > > index ed823377dc5a..9b165e8159be 100755
> > > --- a/tests/test-gcs.sh
> > > +++ b/tests/test-gcs.sh
> > > @@ -40,6 +40,7 @@ requires $PYTHON --version
> > > requires_nbdcopy
> > > requires_plugin python
> > > skip_if_valgrind "because Python code leaks memory"
> > > +requires $PYTHON -c 'import google.cloud'
> >
> > It shouldn't do (I think?) ... Did you see:
> >
> >
https://gitlab.com/nbdkit/nbdkit/-/commit/aca30111632091bbc91096f19638b9e...
>
> I do have that commit, and I don't have that module installed.
>
> The actual problem is the import of "google.api_core", which itself is
> part of google-api-core which I don't have installed.
>
> That is a very narrow band of platforms this test can run on.
It should run without any google.* modules installed, and also if any
combination of google.* modules are installed. In all cases it ought
to load the mocked module from tests/test-gcs and ignore the installed
google.* modules, or if it can't do that (see my commit message above)
it should skip the test.
For me, that all works.
I missed the tests/test-gcs/google/api_core directory.
However I found last week that there's something seriously weird
about
the google.* modules (not to mention Python module loading in
general), so I can believe the test might not work in some
combination.
Nevertheless, the commit as proposed is definitely wrong. The gcs
test does not (or should not) use any installed google.* module, so if
it does then that's a bug of some kind.
Agree.
I narrowed it down to python-protobuf being installed.
It creates /usr/lib/python3.12/site-packages/google .
I think this should be a namespace package, which is handled specially,
But I'm not sure if the problem is in python-protobuf, the Python import
machinery or the setup by test nbdkit testsuite.
Will investigate.
What the full log (tests/test-gcs.sh.log) when it fails?
FAIL: test-gcs.sh
=================
+ requires_run
+ is_windows
++ nbdkit --dump-config
++ grep '^host_os'
++ cut -d= -f2
+ host_os=linux-gnu
+ [[ linux-gnu =~ mingw|msys ]]
+ requires hexdump --version
+ requires /usr/bin/python3 --version
+ requires_nbdcopy
+ requires nbdcopy --version
+ requires_nbdsh_uri
+ requires nbdsh -c 'exit(not h.supports_uri())'
+ requires_plugin python
+ requires nbdkit python --version
+ skip_if_valgrind 'because Python code leaks memory'
+ '[' '' = 1 ']'
+ requires_not /usr/bin/python3 -c 'import google.cloud.storage'
+ requires test -d ../../tests/test-gcs
+ prepend PYTHONPATH ../../tests/test-gcs
+ eval 'PYTHONPATH=../../tests/test-gcs${PYTHONPATH:+:$PYTHONPATH}'
++ PYTHONPATH=../../tests/test-gcs
+ export PYTHONPATH
+ file=gcs.out
+ rm -f gcs.out
+ cleanup_fn rm -f gcs.out
+ local _hook=_cleanup_hook0
+ declare -ag _cleanup_hook0
+ eval '_cleanup_hook0=("$@")'
++ _cleanup_hook0=("$@")
+ nbdkit -v gcs json-credentials=TEST_JSON_CREDENTIALS bucket=MY_FILES key=MY_KEY --run
'nbdcopy "$uri" gcs.out'
/home/t-8ch/src/nbdkit/build/server/nbdkit -v "--run=nbdcopy \"\$uri\"
gcs.out" -- /home/t-8ch/src/nbdkit/build/plugins/python/.libs/nbdkit-python-plugin.so
/home/t-8ch/src/nbdkit/build/plugins/gcs/nbdkit-gcs-plugin
json-credentials=TEST_JSON_CREDENTIALS bucket=MY_FILES key=MY_KEY
nbdkit: debug: nbdkit 1.39.4
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: debug: service mode: unix socket
nbdkit: debug: NBD URI: nbd+unix://?socket=/tmp/nbdkitQvp5XP/socket
nbdkit: debug: registering plugin
/home/t-8ch/src/nbdkit/build/plugins/python/.libs/nbdkit-python-plugin.so
nbdkit: debug: registered plugin
/home/t-8ch/src/nbdkit/build/plugins/python/.libs/nbdkit-python-plugin.so (name python)
nbdkit: debug: python: load
nbdkit: debug: python: config key=script,
value=/home/t-8ch/src/nbdkit/build/plugins/gcs/nbdkit-gcs-plugin
Traceback (most recent call last):
File "/home/t-8ch/src/nbdkit/build/plugins/gcs/nbdkit-gcs-plugin", line 46, in
<module>
from google.api_core.exceptions import NotFound, GatewayTimeout, \
ModuleNotFoundError: No module named 'google.api_core'
nbdkit: error: /home/t-8ch/src/nbdkit/build/plugins/gcs/nbdkit-gcs-plugin: error running
this script
++ _run_cleanup_hooks
++ local _status=1 _i
++ set +e
++ trap '' INT QUIT TERM EXIT ERR
++ echo ../../tests/test-gcs.sh: run cleanup hooks: exit code 1
../../tests/test-gcs.sh: run cleanup hooks: exit code 1
++ (( _i = 0 ))
++ (( _i < 1 ))
++ eval '"${_cleanup_hook0[@]}"'
+++ rm -f gcs.out
++ (( ++_i ))
++ (( _i < 1 ))
++ exit 1
FAIL test-gcs.sh (exit status: 1)
Thomas
12:40 p.m.
On 2024-04-28 19:26:00+0000, Thomas Weißschuh wrote:
On 2024-04-28 17:34:42+0000, Richard W.M. Jones wrote:
[..]
> It should run without any google.* modules installed, and also
if any
> combination of google.* modules are installed. In all cases it ought
> to load the mocked module from tests/test-gcs and ignore the installed
> google.* modules, or if it can't do that (see my commit message above)
> it should skip the test.
>
> For me, that all works.
I missed the tests/test-gcs/google/api_core directory.
> However I found last week that there's something seriously weird about
> the google.* modules (not to mention Python module loading in
> general), so I can believe the test might not work in some
> combination.
>
> Nevertheless, the commit as proposed is definitely wrong. The gcs
> test does not (or should not) use any installed google.* module, so if
> it does then that's a bug of some kind.
Agree.
I narrowed it down to python-protobuf being installed.
It creates /usr/lib/python3.12/site-packages/google .
I think this should be a namespace package, which is handled specially,
But I'm not sure if the problem is in python-protobuf, the Python import
machinery or the setup by test nbdkit testsuite.
Will investigate.
PEBKAC...
I first ran the testsuite against v1.38.1 where the gcs test failed.
Then I went to master and tried to fix it, blindly creating
tests/test-gcs/google/__init__.py. This broke the namespace package
import logic and with it the test on master again.
So please just disregard all of this noise.
Thomas
1:38 p.m.
On Sun, Apr 28, 2024 at 07:40:43PM +0200, Thomas Weißschuh wrote:
On 2024-04-28 19:26:00+0000, Thomas Weißschuh wrote:
> On 2024-04-28 17:34:42+0000, Richard W.M. Jones wrote:
> [..]
> > It should run without any google.* modules installed, and also if any
> > combination of google.* modules are installed. In all cases it ought
> > to load the mocked module from tests/test-gcs and ignore the installed
> > google.* modules, or if it can't do that (see my commit message above)
> > it should skip the test.
> >
> > For me, that all works.
>
> I missed the tests/test-gcs/google/api_core directory.
>
> > However I found last week that there's something seriously weird about
> > the google.* modules (not to mention Python module loading in
> > general), so I can believe the test might not work in some
> > combination.
> >
> > Nevertheless, the commit as proposed is definitely wrong. The gcs
> > test does not (or should not) use any installed google.* module, so if
> > it does then that's a bug of some kind.
>
> Agree.
>
> I narrowed it down to python-protobuf being installed.
> It creates /usr/lib/python3.12/site-packages/google .
>
> I think this should be a namespace package, which is handled specially,
> But I'm not sure if the problem is in python-protobuf, the Python import
> machinery or the setup by test nbdkit testsuite.
>
> Will investigate.
PEBKAC...
I first ran the testsuite against v1.38.1 where the gcs test failed.
Then I went to master and tried to fix it, blindly creating
tests/test-gcs/google/__init__.py. This broke the namespace package
import logic and with it the test on master again.
So please just disregard all of this noise.
No problem. I'll need to backport the gcs fixes to the stable branch
once we're sure they're good.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
59
days inactive
59
days old
6 comments
2 participants
participants (2)
-
Richard W.M. Jones -
Thomas Weißschuh