Repository navigation
Conversation
|
|
||
| // Acquire PEP 788 finalization guard before switching or running. | ||
| // If finalization has already started, TryAcquire will fail safely. | ||
| if (_PyInterpreterGuard_TryAcquire(interp, &session->guard) < 0) { |
There was a problem hiding this comment.
We should try to use the public API (PyInterpreterGuard_FromCurrent) and store the result as a pointer. That would also remove the need for the extra has_guard field, because NULL could act as has_guard = 0.
There was a problem hiding this comment.
I tried using the PyInterpreterGuard_FromCurrent() but that resulted in a deadlock in _PyThreadState_Attach, reason could be the behaviour of _enter_session ( Not super sure though).
Instead, since PyInterpreterGuard wraps PyInterpreterState *interp (NULL when zero-initialized), I removed has_guard by checking session->guard.interp != NULL while keeping _PyInterpreterGuard_TryAcquire before _enter_session().
There was a problem hiding this comment.
Please determine the source of the deadlock. I think it's a bad idea to rely on guard internals here.
There was a problem hiding this comment.
This is the Python and C stack trace (I symbolised it a little bit because it's easier to follow this way):
test_concurrent_run_and_destroy (test.test__interpreters.DestroyTests.test_concurrent_run_and_destroy) ...
=== C Stack Trace on Freed tstate->interp ===
#01 _PyThreadState_CheckConsistency (Python\pystate.c:3371)
#02 take_gil (Python\ceval_gil.c:411)
#03 _PyEval_AcquireLock (Python\ceval_gil.c:593)
#04 _PyThreadState_Attach (Python\pystate.c:2405)
#05 _PyThreadState_Swap (Python\pystate.c:2801)
#06 PyThreadState_Swap (Python\pystate.c:2810)
#07 _enter_session (Python\crossinterp.c:2603)
#08 _PyXI_Enter (Python\crossinterp.c:2731)
#09 _run_in_interpreter (Modules\_interpretersmodule.c:676)
#10 _interpreters_run_string_impl (Modules\_interpretersmodule.c:1199)
#11 _interpreters_run_string (Modules\clinic\_interpretersmodule.c.h:583)
Assertion failed: !_PyMem_IsPtrFreed(tstate->interp), file Python\pystate.c, line 3371
Fatal Python error: Aborted
Current thread 0x00003e8c [Thread-1 (runner)] (most recent call first):
File "Lib\test\test__interpreters.py", line 478 in runner
File "Lib\threading.py", line 1161 in run
File "Lib\threading.py", line 1219 in _bootstrap_inner
File "Lib\threading.py", line 1181 in _bootstrap
Based on the further analysis I found two things related to public API:
- It takes a
voidargument, that leads to it guarding the caller's interpreter not the sub-interpreter we require. - Calling it after
_enter_session()leavesPyThreadState_Swap()unguarded—so a concurrentdestroy()freesinterpor marks the new thread stateSHUTTING_DOWN, hanging in_PyThreadState_HangThread()insidetake_gil().
There was a problem hiding this comment.
That's an abort, not a deadlock. Can you show the diff?
There was a problem hiding this comment.
I'm sorry for mixing it up, I'm sharing the diff:
diff --git a/Python/crossinterp.c b/Python/crossinterp.c
index b8dd5f11c2a..e998fb627da 100644
--- a/Python/crossinterp.c
+++ b/Python/crossinterp.c
@@ -2546,6 +2546,8 @@ struct xi_session {
// once the session exits. Do not access this directly; use
// _PyXI_Preserve() and _PyXI_GetPreserved() instead;
PyObject *_preserved;
+
+ PyInterpreterGuard *guard;
};
_PyXI_session *
@@ -2641,6 +2643,11 @@ _exit_session(_PyXI_session *session)
assert(!session->own_init_tstate);
}
+ if (session->guard != NULL) {
+ PyInterpreterGuard_Close(session->guard);
+ session->guard = NULL;
+ }
+
*session = (_PyXI_session){0};
}
@@ -2717,6 +2724,11 @@ _PyXI_Enter(_PyXI_session *session,
// Switch to the requested interpreter (if necessary).
_enter_session(session, interp);
_PyXI_failure override = XI_FAILURE_INIT;
override.code = _PyXI_ERR_UNCAUGHT_EXCEPTION;
#ifndef NDEBUG
tstate = _PyThreadState_GET();
#endif
+ session->guard = PyInterpreterGuard_FromCurrent();
+ if (session->guard == NULL) {
+ goto error;
+ }
+
// Ensure this thread owns __main__.
if (_PyInterpreterState_SetRunningMain(interp) < 0) {
There was a problem hiding this comment.
Ah, right, we need to attach to an interpreter to use PyInterpreterGuard_FromCurrent, so that won't work
But, I think the current approach is thread-unsafe as currently written, because there's nothing stopping an interpreter from being deleted during the call to _PyInterpreterGuard_TryAcquire. PEP 788 comes with interpreter views for this exact use-case, so let's try those. Something like this should work:
- Switch the
PyInterpreterState *interpparameter of_PyXI_Enterto aPyInterpreterView *view. - Call
PyInterpreterGuard_FromViewbefore trying to attach. - Release the guard after attaching to the interpreter (we don't want to block finalization for all of the subinterpreter's execution; just the part where it can be potentially deleted by another thread).
Based on the feedback I have utilized the PEP 788 finalization guard API (
_PyInterpreterGuard_TryAcquire / _PyInterpreterGuard_Release, added in #128640) inside_PyXI_session, as follows:Before
_enter_session(): Call_PyInterpreterGuard_TryAcquire(). If destruction has already started, fail cleanly withInterpreterError("interpreter is being destroyed").At the end of
_exit_session(): Release the guard after thread-state cleanup and swap-back, soPy_EndInterpreter()waits for any in-flight session to exit before finalizing._interpreters.destroy#127041