10:40 a.m.
Patch 2 certainly allows you to set the thread model. However patch 3
shows that if you set it to nbdkit.THREAD_MODEL_PARALLEL it will
crash.
If you look closely at the stack trace (attached below) you can see
that ignoring threads which are in parts of nbdkit unrelated to
Python:
Thread 4: In pread, waiting in time.sleep(). This thread has released
the GIL.
Thread 2: Started to process a pread call but didn't reach Python code yet.
Thread 1: In pread, segfaults when checking if pread() is defined
in the Python code.
My understanding is this should all be OK and there's no reason for
Python to crash here. I wonder if it's because we're calling "down"
into Python from C, rather than the usual way of calling from Python
into C.
Rich.
Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -v -P
test-python-thread-model.pid -U /tmp/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fc4b943e97c in find_name_in_mro (type=<optimized out>,
name=0x7fc4aba406b0, error=0x7fc4aa21e7d4)
at /usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Python/errors.c:221
221 PyErr_Occurred(void)
[Current thread is 1 (Thread 0x7fc4aa21f640 (LWP 109870))]
glibc-2.31.9000-21.fc33.x86_64 gmp-6.1.2-12.fc32.x86_64 libidn2-2.3.0-1.fc32.x86_64
libselinux-3.1-1.fc33.x86_64 libtasn1-4.15.0-1.fc32.x86_64 p11-kit-0.23.18.1-1.fc32.x86_64
pcre2-10.34-4.fc32.x86_64
(gdb) t a a bt
Thread 9 (Thread 0x7fc4a9a1e640 (LWP 109871)):
#0 0x00007fc4b9d32e5b in __lll_lock_wait_private () from /lib64/libpthread.so.0
#1 0x00007fc4b9d34115 in flockfile () from /lib64/libpthread.so.0
#2 0x000000000040a053 in nbdkit_debug (fs=0x41bada "starting worker thread %s")
at debug.c:91
#3 0x00000000004081b7 in connection_worker (data=0x9ee9ff0) at connections.c:116
#4 0x00007fc4b9d293f9 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc4b9c55b23 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fc4aaa20640 (LWP 109869)):
#0 0x00007fc4b9d32e5b in __lll_lock_wait_private () from /lib64/libpthread.so.0
#1 0x00007fc4b9d34115 in flockfile () from /lib64/libpthread.so.0
#2 0x000000000040a053 in nbdkit_debug (fs=0x41b3ea "%s: pread count=%u
offset=%lu") at debug.c:91
#3 0x0000000000406507 in backend_pread (b=0x9f07040, buf=0x9f80dc0, count=512, offset=0,
flags=0, err=0x7fc4aaa1fa78) at backend.c:482
#4 0x0000000000411f7e in handle_request (cmd=0, flags=0, offset=0, count=512,
buf=0x9f80dc0, extents=0x0) at protocol.c:241
#5 0x00000000004131c9 in protocol_recv_request_send_reply () at protocol.c:713
#6 0x00000000004081e7 in connection_worker (data=0x9ee9ff0) at connections.c:123
#7 0x00007fc4b9d293f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fc4b9c55b23 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fc4aba22640 (LWP 109867)):
#0 0x00007fc4b9c5079b in mprotect () from /lib64/libc.so.6
#1 0x00007fc4b9d2a0e1 in pthread_create@(a)GLIBC_2.2.5 () from /lib64/libpthread.so.0
#2 0x0000000000408491 in handle_single_connection (sockin=8, sockout=8) at
connections.c:204
#3 0x0000000000418824 in start_thread (datav=0x9ee9b10) at sockets.c:337
#4 0x00007fc4b9d293f9 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc4b9c55b23 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fc4b96bb200 (LWP 109858)):
#0 0x00007fc4b9c4aa2f in poll () from /lib64/libc.so.6
#1 0x0000000000418af3 in check_sockets_and_quit_fd (socks=0x7ffe7d9b3b80) at
sockets.c:447
#2 0x0000000000418bd7 in accept_incoming_connections (socks=0x7ffe7d9b3b80) at
sockets.c:475
#3 0x000000000040f6c9 in start_serving () at main.c:974
#4 0x000000000040ef8b in main (argc=9, argv=0x7ffe7d9b3de8) at main.c:736
Thread 5 (Thread 0x7fc4a921d640 (LWP 109872)):
#0 0x00007fc4b9c464ef in write () from /lib64/libc.so.6
#1 0x00007fc4b9bd5e5d in _IO_file_write@(a)GLIBC_2.2.5 () from /lib64/libc.so.6
#2 0x00007fc4b9bd5196 in new_do_write () from /lib64/libc.so.6
#3 0x00007fc4b9bd6f49 in __GI__IO_do_write () from /lib64/libc.so.6
#4 0x00007fc4b9bd73b3 in __GI__IO_file_overflow () from /lib64/libc.so.6
#5 0x00007fc4b9bd2423 in fputc () from /lib64/libc.so.6
#6 0x000000000040a0b9 in nbdkit_debug (fs=0x41bada "starting worker thread %s")
at debug.c:98
#7 0x00000000004081b7 in connection_worker (data=0x9ee9cb0) at connections.c:116
#8 0x00007fc4b9d293f9 in start_thread () from /lib64/libpthread.so.0
#9 0x00007fc4b9c55b23 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fc4ab221640 (LWP 109868)):
#0 0x00007fc4b9c4d1bb in select () from /lib64/libc.so.6
#1 0x00007fc4b95321e9 in pysleep (secs=<optimized out>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Modules/timemodule.c:1909
#2 time_sleep (self=<optimized out>, obj=<optimized out>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Modules/timemodule.c:341
#3 0x00007fc4b9451a62 in cfunction_vectorcall_O (func=<built-in method sleep of module
object at remote 0x7fc4abc41540>, args=0x7fc4abc3ffd0, nargsf=<optimized out>,
kwnames=<optimized out>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/methodobject.c:510
#4 0x00007fc4b944a0a7 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized
out>, args=0x7fc4abc3ffd0, callable=<built-in method sleep of module object at
remote 0x7fc4abc41540>, tstate=0x9f0a610) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Include/cpython/abstract.h:118
#5 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fc4abc3ffd0,
callable=<built-in method sleep of module object at remote 0x7fc4abc41540>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Include/cpython/abstract.h:127
#6 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic
pointer>, tstate=0x9f0a610) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Python/ceval.c:5044
#7 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>,
throwflag=<optimized out>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Python/ceval.c:3459
#8 0x00007fc4b945270b in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fc4abc3fe40, for file
./python-thread-model.py, line 49, in pread (h={}, count=512, offset=0), tstate=0x9f0a610)
at /usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Include/internal/pycore_ceval.h:40
#9 function_code_fastcall (tstate=0x9f0a610, co=<optimized out>, args=<optimized
out>, nargs=3, globals=<optimized out>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/call.c:329
#10 0x00007fc4b9455276 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized
out>, args=0x7fc4ab220790, callable=<function at remote 0x7fc4aba34a60>,
tstate=0x9f0a610) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Include/cpython/abstract.h:81
#11 _PyObject_CallFunctionVa (tstate=0x9f0a610, callable=<function at remote
0x7fc4aba34a60>, format=<optimized out>, va=<optimized out>,
is_size_t=<optimized out>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/call.c:542
#12 0x00007fc4b94c8762 in _PyObject_CallFunction_SizeT (callable=<optimized out>,
format=<optimized out>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/call.c:596
#13 0x00007fc4b9f7b840 in py_pread (handle=0x9ee9f20, buf=0x9f98ca0, count=512, offset=0,
flags=0) at python.c:619
#14 0x0000000000410d89 in plugin_pread (b=0x9f07040, handle=0x9ee9f20, buf=0x9f98ca0,
count=512, offset=0, flags=0, err=0x7fc4ab220a78) at plugins.c:524
#15 0x000000000040653f in backend_pread (b=0x9f07040, buf=0x9f98ca0, count=512, offset=0,
flags=0, err=0x7fc4ab220a78) at backend.c:485
#16 0x0000000000411f7e in handle_request (cmd=0, flags=0, offset=0, count=512,
buf=0x9f98ca0, extents=0x0) at protocol.c:241
#17 0x00000000004131c9 in protocol_recv_request_send_reply () at protocol.c:713
#18 0x00000000004081e7 in connection_worker (data=0x9ee9cb0) at connections.c:123
#19 0x00007fc4b9d293f9 in start_thread () from /lib64/libpthread.so.0
#20 0x00007fc4b9c55b23 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fc4a821b640 (LWP 109874)):
#0 0x00007fc4b9c55b15 in clone () from /lib64/libc.so.6
#1 0x00007fc4b9d29310 in annobin_start_thread.start () from /lib64/libpthread.so.0
#2 0x00007fc4a821b640 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fc4a8a1c640 (LWP 109873)):
#0 0x00007fc4b9d32e5b in __lll_lock_wait_private () from /lib64/libpthread.so.0
#1 0x00007fc4b9d34115 in flockfile () from /lib64/libpthread.so.0
#2 0x000000000040a053 in nbdkit_debug (fs=0x41b3ea "%s: pread count=%u
offset=%lu") at debug.c:91
#3 0x0000000000406507 in backend_pread (b=0x9f07040, buf=0x9f7cc40, count=512, offset=0,
flags=0, err=0x7fc4a8a1ba78) at backend.c:482
#4 0x0000000000411f7e in handle_request (cmd=0, flags=0, offset=0, count=512,
buf=0x9f7cc40, extents=0x0) at protocol.c:241
#5 0x00000000004131c9 in protocol_recv_request_send_reply () at protocol.c:713
#6 0x00000000004081e7 in connection_worker (data=0x9eea670) at connections.c:123
#7 0x00007fc4b9d293f9 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fc4b9c55b23 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fc4aa21f640 (LWP 109870)):
#0 0x00007fc4b943e97c in find_name_in_mro (type=<optimized out>,
name='pread', error=0x7fc4aa21e7d4) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Python/errors.c:221
#1 0x00007fc4b943e7e3 in _PyType_Lookup (type=0x7fc4b966c760 <PyModule_Type>,
name='pread') at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/typeobject.c:3232
#2 0x00007fc4b94512bd in _PyObject_GenericGetAttrWithDict (obj=<module at remote
0x7fc4abbfabd0>, name='pread', dict=0x0, suppress=0) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/object.c:1194
#3 0x00007fc4b9455194 in PyObject_GenericGetAttr (name='pread', obj=<module at
remote 0x7fc4abbfabd0>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/object.c:1278
#4 module_getattro (m=0x7fc4abbfabd0, name='pread') at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/moduleobject.c:717
#5 0x00007fc4b94c2014 in PyObject_GetAttrString (v=<module at remote
0x7fc4abbfabd0>, name=<optimized out>) at
/usr/src/debug/python3.9-3.9.0~b3-1.fc33.x86_64/Objects/object.c:795
#6 0x00007fc4b9f7a5e8 in callback_defined (name=0x7fc4b9f7d4b6 "pread",
obj_rtn=0x7fc4aa21e940) at python.c:77
#7 0x00007fc4b9f7b7c8 in py_pread (handle=0x9ee9f20, buf=0x9f88890, count=512, offset=0,
flags=0) at python.c:610
#8 0x0000000000410d89 in plugin_pread (b=0x9f07040, handle=0x9ee9f20, buf=0x9f88890,
count=512, offset=0, flags=0, err=0x7fc4aa21ea78) at plugins.c:524
#9 0x000000000040653f in backend_pread (b=0x9f07040, buf=0x9f88890, count=512, offset=0,
flags=0, err=0x7fc4aa21ea78) at backend.c:485
#10 0x0000000000411f7e in handle_request (cmd=0, flags=0, offset=0, count=512,
buf=0x9f88890, extents=0x0) at protocol.c:241
#11 0x00000000004131c9 in protocol_recv_request_send_reply () at protocol.c:713
#12 0x00000000004081e7 in connection_worker (data=0x9ee9cb0) at connections.c:123
#13 0x00007fc4b9d293f9 in start_thread () from /lib64/libpthread.so.0
#14 0x00007fc4b9c55b23 in clone () from /lib64/libc.so.6
10:40 a.m.
New subject: [PATCH nbdkit 1/3] python: Make last_error into a thread-local variable.
While with the current thread model this makes no difference, it is
required if we want to allow more parallel thread models.
---
plugins/python/python.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/plugins/python/python.c b/plugins/python/python.c
index f21a1602..9fa89964 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -63,7 +63,7 @@ static const char *script;
static PyObject *module;
static int py_api_version = 1;
-static int last_error;
+static __thread int last_error;
/* Is a callback defined? */
static int
--
2.27.0
10:40 a.m.
New subject: [PATCH nbdkit 2/3] python: Allow thread model to be set from Python code.
Thanks: Nir Soffer.
---
plugins/python/nbdkit-python-plugin.pod | 26 ++++++++++++++++-------
plugins/python/python.c | 28 ++++++++++++++++++++++++-
2 files changed, 45 insertions(+), 9 deletions(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod
b/plugins/python/nbdkit-python-plugin.pod
index 852366f0..edd56d13 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -163,6 +163,15 @@ There are no arguments or return value.
There are no arguments or return value.
+=item C<thread_model>
+
+(Optional, nbdkit E<ge> 1.22)
+
+ def thread_model():
+ return nbdkit.THEAD_MODEL_SERIALIZE_ALL_REQUESTS
+
+See L</Threads> below.
+
=item C<get_ready>
(Optional)
@@ -367,10 +376,6 @@ optionally using C<nbdkit.set_error> first.
These are not needed because you can just use ordinary Python
constructs.
-=item Missing: C<thread_model>
-
-See L</Threads> below.
-
=item Missing:
C<name>,
C<version>,
@@ -387,10 +392,15 @@ These are not yet supported.
=head2 Threads
-The thread model for Python callbacks currently cannot be set from
-Python. It is hard-coded in the C part to
-C<NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS>. This may change or be
-settable in future.
+The thread model for Python callbacks defaults to
+C<NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS>. Since S<nbdkit 1.22>
+it is possible to set this by implementing a C<thread_model> function
+which returns one of the constants C<nbdkit.THREAD_MODEL_*>.
+
+The Python Global Interpreter Lock (GIL) usually means that you cannot
+execute Python code in parallel, but Python code which calls into
+libraries which block (eg. to make HTTP requests) might be executed in
+parallel.
=head1 FILES
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 9fa89964..aa19475b 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -472,6 +472,27 @@ py_config_complete (void)
return 0;
}
+static int
+py_thread_model (void)
+{
+ PyObject *fn;
+ PyObject *r;
+ int ret = NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS;
+
+ if (callback_defined ("thread_model", &fn)) {
+ PyErr_Clear ();
+
+ r = PyObject_CallObject (fn, NULL);
+ Py_DECREF (fn);
+ if (check_python_failure ("thread_model") == -1)
+ return -1;
+ ret = PyLong_AsLong (r);
+ Py_DECREF (r);
+ }
+
+ return ret;
+}
+
static int
py_get_ready (void)
{
@@ -961,7 +982,11 @@ py_can_cache (void *handle)
"script=<FILENAME> (required) The Python plugin to run.\n" \
"[other arguments may be used by the plugin that you load]"
-#define THREAD_MODEL NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS
+/* This is the maximum possible, but the default for plugins is
+ * NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS. Plugins can override
+ * that by providing a thread_model() function.
+ */
+#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL
static struct nbdkit_plugin plugin = {
.name = "python",
@@ -975,6 +1000,7 @@ static struct nbdkit_plugin plugin = {
.config_complete = py_config_complete,
.config_help = py_config_help,
+ .thread_model = py_thread_model,
.get_ready = py_get_ready,
.open = py_open,
--
2.27.0
10:40 a.m.
New subject: [PATCH nbdkit 3/3] python: Test the parallel thread model.
---
tests/Makefile.am | 3 ++
tests/test-python-thread-model.sh | 84 +++++++++++++++++++++++++++++++
tests/python-thread-model.py | 50 ++++++++++++++++++
3 files changed, 137 insertions(+)
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 79be5639..2aa9e14c 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -1020,15 +1020,18 @@ TESTS += \
test-python.sh \
test-python-exception.sh \
test-python-export-name.sh \
+ test-python-thread-model.sh \
test-shebang-python.sh \
$(NULL)
EXTRA_DIST += \
python-exception.py \
python-export-name.py \
+ python-thread-model.py \
shebang.py \
test-python-exception.sh \
test-python-export-name.sh \
test-python-plugin.py \
+ test-python-thread-model.sh \
test-python.sh \
test-shebang-python.sh \
test_python.py \
diff --git a/tests/test-python-thread-model.sh b/tests/test-python-thread-model.sh
new file mode 100755
index 00000000..35bb5160
--- /dev/null
+++ b/tests/test-python-thread-model.sh
@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2018-2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+source ./functions.sh
+set -e
+set -x
+
+SCRIPT="$SRCDIR/python-thread-model.py"
+if ! test -d "$SRCDIR" || ! test -f "$SCRIPT"; then
+ echo "$0: could not locate python-thread-model.py"
+ exit 1
+fi
+
+# Python has proven very difficult to valgrind, therefore it is disabled.
+if [ "$NBDKIT_VALGRIND" = "1" ]; then
+ echo "$0: skipping Python test under valgrind."
+ exit 77
+fi
+
+requires nbdsh --version
+
+out=test-python-thread-model.out
+pid=test-python-thread-model.pid
+sock=`mktemp -u`
+files="$out $pid $sock"
+rm -f $files
+cleanup_fn rm -f $files
+
+# Check the plugin is loadable and the effective thread model is parallel.
+nbdkit python $SCRIPT --dump-plugin >$out
+grep "^thread_model=parallel" $out
+
+start_nbdkit -P $pid -U $sock python $SCRIPT
+
+export sock
+nbdsh -c '
+import os
+import time
+
+h.connect_unix (os.environ["sock"])
+
+# We should be able to issue multiple requests in parallel,
+# and the total time taken should not be much more than 10 seconds
+# because all sleeps in the plugin should happen in parallel.
+start_t = time.time()
+for i in range (10):
+ buf = nbd.Buffer (512)
+ h.aio_pread (buf, 0)
+
+while h.aio_in_flight() > 0:
+ h.poll(-1)
+end_t = time.time()
+
+print (end_t - start_t)
+'
diff --git a/tests/python-thread-model.py b/tests/python-thread-model.py
new file mode 100644
index 00000000..67879e04
--- /dev/null
+++ b/tests/python-thread-model.py
@@ -0,0 +1,50 @@
+# nbdkit
+# Copyright (C) 2018-2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+# Python plugin which uses the parallel thread model. Note that the
+# cpython implementation of time.sleep releases the GIL (see
+# Modules/timemodule.c:pysleep)
+
+import nbdkit
+import time
+
+def thread_model():
+ return nbdkit.THREAD_MODEL_PARALLEL
+
+def open(readonly):
+ return {}
+
+def get_size(h):
+ return 512
+
+def pread(h, count, offset):
+ time.sleep(10)
+ return bytearray(count)
--
2.27.0
10:46 a.m.
Nevermind, I discovered: https://docs.python.org/3/c-api/init.html
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
10:50 a.m.
On Wed, Aug 05, 2020 at 04:40:26PM +0100, Richard W.M. Jones wrote:
Patch 2 certainly allows you to set the thread model. However patch
3
shows that if you set it to nbdkit.THREAD_MODEL_PARALLEL it will
crash.
If you look closely at the stack trace (attached below) you can see
that ignoring threads which are in parts of nbdkit unrelated to
Python:
Thread 4: In pread, waiting in time.sleep(). This thread has released
the GIL.
Has it - I'm not seeing any call to PyEval_SaveThread()/RestoreThread
which is needed to release the GIL when going from Python to C.
Thread 2: Started to process a pread call but didn't reach Python code yet.
Thread 1: In pread, segfaults when checking if pread() is defined
in the Python code.
My understanding is this should all be OK and there's no reason for
Python to crash here. I wonder if it's because we're calling "down"
into Python from C, rather than the usual way of calling from Python
into C.
I'm also not seeing any coodes to PyGILState_Ensure()/Release which is
needed call into Python from C.
Without these calls, random crashes are certainly expected.
What libvirt-python does is
eg
PyThreadState *_save = NULL;
if (PyEval_ThreadsInitialized())
_save = PyEval_SaveThread();
...call C functions...
if (PyEval_ThreadsInitialized())
PyEval_RestoreThread(_save);
and
PyGILState_STATE _save = PyGILState_UNLOCKED;
if (PyEval_ThreadsInitialized())
_save = PyGILState_Ensure();
... call Python functions...
if (PyEval_ThreadsInitialized())
PyGILState_Release(_save);
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1552
days inactive
1552
days old
5 comments
2 participants
participants (2)
-
Daniel P. Berrangé -
Richard W.M. Jones