[PATCH nbdkit] tests: gcs: skip tests without google cloud module
The gcs plugin uses the google cloud module and can't work without it. Skip the test if the module is unavailable. Signed-off-by: Thomas Weißschuh <thomas@t-8ch.de> --- tests/test-gcs.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh index ed823377dc5a..9b165e8159be 100755 --- a/tests/test-gcs.sh +++ b/tests/test-gcs.sh @@ -40,6 +40,7 @@ requires $PYTHON --version requires_nbdcopy requires_plugin python skip_if_valgrind "because Python code leaks memory" +requires $PYTHON -c 'import google.cloud' # Skip this test if the real google-cloud-storage module is installed, # since that module abuses pth files to insert itself into the module base-commit: ffa62d30fe6dcd29ae8e20df5c86d48505550614 -- 2.44.0
On Sun, Apr 28, 2024 at 05:44:53PM +0200, Thomas Weißschuh wrote:
The gcs plugin uses the google cloud module and can't work without it. Skip the test if the module is unavailable.
Signed-off-by: Thomas Weißschuh <thomas@t-8ch.de> --- tests/test-gcs.sh | 1 + 1 file changed, 1 insertion(+)
diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh index ed823377dc5a..9b165e8159be 100755 --- a/tests/test-gcs.sh +++ b/tests/test-gcs.sh @@ -40,6 +40,7 @@ requires $PYTHON --version requires_nbdcopy requires_plugin python skip_if_valgrind "because Python code leaks memory" +requires $PYTHON -c 'import google.cloud'
It shouldn't do (I think?) ... Did you see: https://gitlab.com/nbdkit/nbdkit/-/commit/aca30111632091bbc91096f19638b9e1df... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On 2024-04-28 17:09:06+0000, Richard W.M. Jones wrote:
On Sun, Apr 28, 2024 at 05:44:53PM +0200, Thomas Weißschuh wrote:
The gcs plugin uses the google cloud module and can't work without it. Skip the test if the module is unavailable.
Signed-off-by: Thomas Weißschuh <thomas@t-8ch.de> --- tests/test-gcs.sh | 1 + 1 file changed, 1 insertion(+)
diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh index ed823377dc5a..9b165e8159be 100755 --- a/tests/test-gcs.sh +++ b/tests/test-gcs.sh @@ -40,6 +40,7 @@ requires $PYTHON --version requires_nbdcopy requires_plugin python skip_if_valgrind "because Python code leaks memory" +requires $PYTHON -c 'import google.cloud'
It shouldn't do (I think?) ... Did you see:
https://gitlab.com/nbdkit/nbdkit/-/commit/aca30111632091bbc91096f19638b9e1df...
I do have that commit, and I don't have that module installed. The actual problem is the import of "google.api_core", which itself is part of google-api-core which I don't have installed. That is a very narrow band of platforms this test can run on. Thomas
On Sun, Apr 28, 2024 at 06:22:43PM +0200, Thomas Weißschuh wrote:
On 2024-04-28 17:09:06+0000, Richard W.M. Jones wrote:
On Sun, Apr 28, 2024 at 05:44:53PM +0200, Thomas Weißschuh wrote:
The gcs plugin uses the google cloud module and can't work without it. Skip the test if the module is unavailable.
Signed-off-by: Thomas Weißschuh <thomas@t-8ch.de> --- tests/test-gcs.sh | 1 + 1 file changed, 1 insertion(+)
diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh index ed823377dc5a..9b165e8159be 100755 --- a/tests/test-gcs.sh +++ b/tests/test-gcs.sh @@ -40,6 +40,7 @@ requires $PYTHON --version requires_nbdcopy requires_plugin python skip_if_valgrind "because Python code leaks memory" +requires $PYTHON -c 'import google.cloud'
It shouldn't do (I think?) ... Did you see:
https://gitlab.com/nbdkit/nbdkit/-/commit/aca30111632091bbc91096f19638b9e1df...
I do have that commit, and I don't have that module installed.
The actual problem is the import of "google.api_core", which itself is part of google-api-core which I don't have installed.
That is a very narrow band of platforms this test can run on.
It should run without any google.* modules installed, and also if any combination of google.* modules are installed. In all cases it ought to load the mocked module from tests/test-gcs and ignore the installed google.* modules, or if it can't do that (see my commit message above) it should skip the test. For me, that all works. However I found last week that there's something seriously weird about the google.* modules (not to mention Python module loading in general), so I can believe the test might not work in some combination. Nevertheless, the commit as proposed is definitely wrong. The gcs test does not (or should not) use any installed google.* module, so if it does then that's a bug of some kind. What the full log (tests/test-gcs.sh.log) when it fails? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On 2024-04-28 17:34:42+0000, Richard W.M. Jones wrote:
On Sun, Apr 28, 2024 at 06:22:43PM +0200, Thomas Weißschuh wrote:
On 2024-04-28 17:09:06+0000, Richard W.M. Jones wrote:
On Sun, Apr 28, 2024 at 05:44:53PM +0200, Thomas Weißschuh wrote:
The gcs plugin uses the google cloud module and can't work without it. Skip the test if the module is unavailable.
Signed-off-by: Thomas Weißschuh <thomas@t-8ch.de> --- tests/test-gcs.sh | 1 + 1 file changed, 1 insertion(+)
diff --git a/tests/test-gcs.sh b/tests/test-gcs.sh index ed823377dc5a..9b165e8159be 100755 --- a/tests/test-gcs.sh +++ b/tests/test-gcs.sh @@ -40,6 +40,7 @@ requires $PYTHON --version requires_nbdcopy requires_plugin python skip_if_valgrind "because Python code leaks memory" +requires $PYTHON -c 'import google.cloud'
It shouldn't do (I think?) ... Did you see:
https://gitlab.com/nbdkit/nbdkit/-/commit/aca30111632091bbc91096f19638b9e1df...
I do have that commit, and I don't have that module installed.
The actual problem is the import of "google.api_core", which itself is part of google-api-core which I don't have installed.
That is a very narrow band of platforms this test can run on.
It should run without any google.* modules installed, and also if any combination of google.* modules are installed. In all cases it ought to load the mocked module from tests/test-gcs and ignore the installed google.* modules, or if it can't do that (see my commit message above) it should skip the test.
For me, that all works.
I missed the tests/test-gcs/google/api_core directory.
However I found last week that there's something seriously weird about the google.* modules (not to mention Python module loading in general), so I can believe the test might not work in some combination.
Nevertheless, the commit as proposed is definitely wrong. The gcs test does not (or should not) use any installed google.* module, so if it does then that's a bug of some kind.
Agree. I narrowed it down to python-protobuf being installed. It creates /usr/lib/python3.12/site-packages/google . I think this should be a namespace package, which is handled specially, But I'm not sure if the problem is in python-protobuf, the Python import machinery or the setup by test nbdkit testsuite. Will investigate.
What the full log (tests/test-gcs.sh.log) when it fails?
FAIL: test-gcs.sh ================= + requires_run + is_windows ++ nbdkit --dump-config ++ grep '^host_os' ++ cut -d= -f2 + host_os=linux-gnu + [[ linux-gnu =~ mingw|msys ]] + requires hexdump --version + requires /usr/bin/python3 --version + requires_nbdcopy + requires nbdcopy --version + requires_nbdsh_uri + requires nbdsh -c 'exit(not h.supports_uri())' + requires_plugin python + requires nbdkit python --version + skip_if_valgrind 'because Python code leaks memory' + '[' '' = 1 ']' + requires_not /usr/bin/python3 -c 'import google.cloud.storage' + requires test -d ../../tests/test-gcs + prepend PYTHONPATH ../../tests/test-gcs + eval 'PYTHONPATH=../../tests/test-gcs${PYTHONPATH:+:$PYTHONPATH}' ++ PYTHONPATH=../../tests/test-gcs + export PYTHONPATH + file=gcs.out + rm -f gcs.out + cleanup_fn rm -f gcs.out + local _hook=_cleanup_hook0 + declare -ag _cleanup_hook0 + eval '_cleanup_hook0=("$@")' ++ _cleanup_hook0=("$@") + nbdkit -v gcs json-credentials=TEST_JSON_CREDENTIALS bucket=MY_FILES key=MY_KEY --run 'nbdcopy "$uri" gcs.out' /home/t-8ch/src/nbdkit/build/server/nbdkit -v "--run=nbdcopy \"\$uri\" gcs.out" -- /home/t-8ch/src/nbdkit/build/plugins/python/.libs/nbdkit-python-plugin.so /home/t-8ch/src/nbdkit/build/plugins/gcs/nbdkit-gcs-plugin json-credentials=TEST_JSON_CREDENTIALS bucket=MY_FILES key=MY_KEY nbdkit: debug: nbdkit 1.39.4 nbdkit: debug: TLS disabled: could not load TLS certificates nbdkit: debug: service mode: unix socket nbdkit: debug: NBD URI: nbd+unix://?socket=/tmp/nbdkitQvp5XP/socket nbdkit: debug: registering plugin /home/t-8ch/src/nbdkit/build/plugins/python/.libs/nbdkit-python-plugin.so nbdkit: debug: registered plugin /home/t-8ch/src/nbdkit/build/plugins/python/.libs/nbdkit-python-plugin.so (name python) nbdkit: debug: python: load nbdkit: debug: python: config key=script, value=/home/t-8ch/src/nbdkit/build/plugins/gcs/nbdkit-gcs-plugin Traceback (most recent call last): File "/home/t-8ch/src/nbdkit/build/plugins/gcs/nbdkit-gcs-plugin", line 46, in <module> from google.api_core.exceptions import NotFound, GatewayTimeout, \ ModuleNotFoundError: No module named 'google.api_core' nbdkit: error: /home/t-8ch/src/nbdkit/build/plugins/gcs/nbdkit-gcs-plugin: error running this script ++ _run_cleanup_hooks ++ local _status=1 _i ++ set +e ++ trap '' INT QUIT TERM EXIT ERR ++ echo ../../tests/test-gcs.sh: run cleanup hooks: exit code 1 ../../tests/test-gcs.sh: run cleanup hooks: exit code 1 ++ (( _i = 0 )) ++ (( _i < 1 )) ++ eval '"${_cleanup_hook0[@]}"' +++ rm -f gcs.out ++ (( ++_i )) ++ (( _i < 1 )) ++ exit 1 FAIL test-gcs.sh (exit status: 1) Thomas
On 2024-04-28 19:26:00+0000, Thomas Weißschuh wrote:
On 2024-04-28 17:34:42+0000, Richard W.M. Jones wrote:
[..]
It should run without any google.* modules installed, and also if any combination of google.* modules are installed. In all cases it ought to load the mocked module from tests/test-gcs and ignore the installed google.* modules, or if it can't do that (see my commit message above) it should skip the test.
For me, that all works.
I missed the tests/test-gcs/google/api_core directory.
However I found last week that there's something seriously weird about the google.* modules (not to mention Python module loading in general), so I can believe the test might not work in some combination.
Nevertheless, the commit as proposed is definitely wrong. The gcs test does not (or should not) use any installed google.* module, so if it does then that's a bug of some kind.
Agree.
I narrowed it down to python-protobuf being installed. It creates /usr/lib/python3.12/site-packages/google .
I think this should be a namespace package, which is handled specially, But I'm not sure if the problem is in python-protobuf, the Python import machinery or the setup by test nbdkit testsuite.
Will investigate.
PEBKAC... I first ran the testsuite against v1.38.1 where the gcs test failed. Then I went to master and tried to fix it, blindly creating tests/test-gcs/google/__init__.py. This broke the namespace package import logic and with it the test on master again. So please just disregard all of this noise. Thomas
On Sun, Apr 28, 2024 at 07:40:43PM +0200, Thomas Weißschuh wrote:
On 2024-04-28 19:26:00+0000, Thomas Weißschuh wrote:
On 2024-04-28 17:34:42+0000, Richard W.M. Jones wrote:
[..]
It should run without any google.* modules installed, and also if any combination of google.* modules are installed. In all cases it ought to load the mocked module from tests/test-gcs and ignore the installed google.* modules, or if it can't do that (see my commit message above) it should skip the test.
For me, that all works.
I missed the tests/test-gcs/google/api_core directory.
However I found last week that there's something seriously weird about the google.* modules (not to mention Python module loading in general), so I can believe the test might not work in some combination.
Nevertheless, the commit as proposed is definitely wrong. The gcs test does not (or should not) use any installed google.* module, so if it does then that's a bug of some kind.
Agree.
I narrowed it down to python-protobuf being installed. It creates /usr/lib/python3.12/site-packages/google .
I think this should be a namespace package, which is handled specially, But I'm not sure if the problem is in python-protobuf, the Python import machinery or the setup by test nbdkit testsuite.
Will investigate.
PEBKAC...
I first ran the testsuite against v1.38.1 where the gcs test failed.
Then I went to master and tried to fix it, blindly creating tests/test-gcs/google/__init__.py. This broke the namespace package import logic and with it the test on master again.
So please just disregard all of this noise.
No problem. I'll need to backport the gcs fixes to the stable branch once we're sure they're good. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
participants (2)
-
Richard W.M. Jones -
Thomas Weißschuh