[PATCH v3 0/2] python: Avoid leaking py_array and py_args in event callbacks
Version 1 was here: https://listman.redhat.com/archives/libguestfs/2023-February/030732.html (Ignore version 2 which had a mistake, this is version 3) Following Eric's suggestion here: https://listman.redhat.com/archives/libguestfs/2023-February/030746.html let's decrement the reference of py_array right after adding it to the args. (This works even if args fails to be built). However the other part of Eric's suggestion is wrong as it ends up calling Py_DECREF (args) when args == NULL along the error path. This lead me to look more closely at this patch: https://listman.redhat.com/archives/libguestfs/2019-January/020346.html which I believe is wrong (at least, the part that fiddles with the reference to args). I cannot reproduce the original problem, nor can I find any justification by looking at the documentation of PyObject_CallObject. So we start by reverting that commit. This means we are only decrementing args once, and not allow the error path. I tested this with: - with the reproducer - with the reproducer and bbf396fc5 reverted - normal Python test suite Rich.
This reverts commit 85235aec837716f1ddb2926b9a59a02543195500. --- python/handle.c | 1 - 1 file changed, 1 deletion(-) diff --git a/python/handle.c b/python/handle.c index bf639b5789..8eeabe60a7 100644 --- a/python/handle.c +++ b/python/handle.c @@ -139,7 +139,6 @@ guestfs_int_py_event_callback_wrapper (guestfs_h *g, goto out; } - Py_INCREF (args); py_r = PyObject_CallObject (py_callback, args); Py_DECREF (args); if (py_r != NULL) -- 2.39.0
On 2/17/23 18:45, Richard W.M. Jones wrote:
This reverts commit 85235aec837716f1ddb2926b9a59a02543195500. --- python/handle.c | 1 - 1 file changed, 1 deletion(-)
diff --git a/python/handle.c b/python/handle.c index bf639b5789..8eeabe60a7 100644 --- a/python/handle.c +++ b/python/handle.c @@ -139,7 +139,6 @@ guestfs_int_py_event_callback_wrapper (guestfs_h *g, goto out; }
- Py_INCREF (args); py_r = PyObject_CallObject (py_callback, args); Py_DECREF (args); if (py_r != NULL)
The patch looks good, but I think the commit's subject line, and the commit message, should be a bit more clear / verbose: - we're only reverting a part of commit 85235aec837716f1ddb2926b9a59a02543195500 - the reason for doing this is that Py_BuildValue() always returns a new reference <https://docs.python.org/3/c-api/arg.html?highlight=py_buildvalue#c.Py_BuildValue> (when it succeeds). With those notes: Reviewed-by: Laszlo Ersek <lersek@redhat.com>
See also: https://listman.redhat.com/archives/libguestfs/2023-February/030730.html https://listman.redhat.com/archives/libguestfs/2023-February/030745.html https://listman.redhat.com/archives/libguestfs/2023-February/030746.html Fixes: commit 6ef5837e2d8c5d4d83eff51c0201eb2e08f719de Thanks: Laszlo Ersek, Eric Blake --- python/handle.c | 1 + 1 file changed, 1 insertion(+) diff --git a/python/handle.c b/python/handle.c index 8eeabe60a7..85089e6bce 100644 --- a/python/handle.c +++ b/python/handle.c @@ -134,6 +134,7 @@ guestfs_int_py_event_callback_wrapper (guestfs_h *g, args = Py_BuildValue ("(Kiy#O)", (unsigned PY_LONG_LONG) event, event_handle, buf, buf_len, py_array); + Py_DECREF (py_array); if (args == NULL) { PyErr_PrintEx (0); goto out; -- 2.39.0
On 2/17/23 18:45, Richard W.M. Jones wrote:
See also:
https://listman.redhat.com/archives/libguestfs/2023-February/030730.html https://listman.redhat.com/archives/libguestfs/2023-February/030745.html https://listman.redhat.com/archives/libguestfs/2023-February/030746.html
Fixes: commit 6ef5837e2d8c5d4d83eff51c0201eb2e08f719de Thanks: Laszlo Ersek, Eric Blake --- python/handle.c | 1 + 1 file changed, 1 insertion(+)
diff --git a/python/handle.c b/python/handle.c index 8eeabe60a7..85089e6bce 100644 --- a/python/handle.c +++ b/python/handle.c @@ -134,6 +134,7 @@ guestfs_int_py_event_callback_wrapper (guestfs_h *g, args = Py_BuildValue ("(Kiy#O)", (unsigned PY_LONG_LONG) event, event_handle, buf, buf_len, py_array); + Py_DECREF (py_array); if (args == NULL) { PyErr_PrintEx (0); goto out;
I *think* doing it like this is safe in this instance. I got concerned for a minute because of: https://docs.python.org/3/c-api/refcounting.html?highlight=py_decref#c.Py_DE... Warning The deallocation function can cause arbitrary Python code to be invoked (e.g. when a class instance with a __del__() method is deallocated). While exceptions in such code are not propagated, the executed code has free access to all Python global variables. This means that any object that is reachable from a global variable should be in a consistent state before Py_DECREF() is invoked. For example, code to delete an object from a list should copy a reference to the deleted object in a temporary variable, update the list data structure, and then call Py_DECREF() for the temporary variable. and: https://docs.python.org/3/c-api/exceptions.html?highlight=pyerr_printex#c.Py... Call this function only when the error indicator is set. Otherwise it will cause a fatal error! So I had two concerns: - can Py_DECREF() clear the error indicator? - can Py_DECREF() lead to the exception (pending form Py_BuildValue()) being replaced (masked?) with a different exception? I think the code is OK: - My reading of the docs implies that the error indicator only gets cleared with explicit PyErr_Clear() or PyErr_PrintEx() calls. - We're only releasing a simple list of longs, so no particular exception should be expected while destructing that. And the testing described in the cover letter confirms this. Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Thanks for the detailed review. This is now upstream in these two commits: e07304de86defb8d3a09902b5f9f7593fa9863ac afa4d64fca1672734ec474d6e8a1eeed4b34c6f4 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
participants (2)
-
Laszlo Ersek -
Richard W.M. Jones