Closed
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-featureA feature request or enhancementA feature request or enhancement
Description
Feature or enhancement
Proposal:
This is an interpretation of the feature ask in: https://discuss.python.org/t/cancel-running-work-in-processpoolexecutor/58605/1. It would be a way to stop all the workers running in a ProcessPoolExecutor
p = ProcessPoolExecutor()
# use p
# i know i want p to die at this point no matter what
p.terminate_workers()Previously the way to do this was to use to loop through the ._processes of the ProcessPoolExecutor though, preferably this should be possible without accessing implementation details.
Has this already been discussed elsewhere?
I have already discussed this feature proposal on Discourse
Links to previous discussion of this feature:
https://discuss.python.org/t/cancel-running-work-in-processpoolexecutor/58605/1
Linked PRs
- gh-128041: Add a terminate_workers method to ProcessPoolExecutor #128043
- gh-128041: Try to fix transiency of test_force_shutdown_workers #130812
- Revert "gh-128041: Add
terminate_workersandkill_workersmethods to ProcessPoolExecutor (GH-128043)" #130838 - gh-128041: Add
terminate_workersandkill_workersmethods to ProcessPoolExecutor #130849 - gh-128041: Fix incorrect bullet placement #130900
Metadata
Metadata
Assignees
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-featureA feature request or enhancementA feature request or enhancement
Activity
pythongh-128041 - Add a terminate_workers method to ProcessPoolExecutor
gh-128041: Add `terminate_workers` and `kill_workers` methods to Proc…
gpshead commentedon Mar 3, 2025
thanks for the feature contribution!
picnixz commentedon Mar 3, 2025
Re-opening because we have some build bots failing (I'm not sure whether the macOS failure is related but the other seems so)
https://github.com/python/cpython/actions/runs/13631446353/job/38099943353#step:11:686
https://buildbot.python.org/#/builders/568/builds/8398
csm10495 commentedon Mar 3, 2025
Opened #130812 to try to fix the transiency. I can't seem to add the skip-news label, but it should get that.
Revert "pythongh-128041: Add `terminate_workers` and `kill_workers` m…
Revert "gh-128041: Add `terminate_workers` and `kill_workers` methods…
colesbury commentedon Mar 4, 2025
@csm10495 - I reverted #128043 due to CI failures. Please open a new PR with the changes and fixes when you have a chance. I can help with testing if you'd like.
csm10495 commentedon Mar 4, 2025
Thanks! @colesbury. I'll dig more to figure out the issue before opening a fresh pr.
csm10495 commentedon Mar 4, 2025
Current issue:
Interestingly enough it was suggested that commenting out
makes the test pass. It does.. most of the time. It still transiently fails the same way with a leaked child process (though much rarer).
So I guess the timing changed caused by that line make it more likely to happen, but we're still dealing with a race condition.
csm10495 commentedon Mar 4, 2025
I think I figured it out. Will open a PR in a few minutes.
pythongh-128041: Add `terminate_workers` and `kill_workers` methods t…
csm10495 commentedon Mar 4, 2025
Opened a new PR: #130849. I believe the transient issues are resolved.
gpshead commentedon Mar 4, 2025
FWIW on the original PR while I suspected we might have flakiness issues... I was way underestimating that apparently. I should've does the buildbot tests before merging. sorry for the churn! :)
gh-128041: Add `terminate_workers` and `kill_workers` methods to Proc…
gpshead commentedon Mar 5, 2025
merged again with the improvements. 🤞🏾
csm10495 commentedon Mar 5, 2025
Tiny related fix: #130900 (docs only)
gh-128041: Fix incorrect bullet placement in "What's new" (GH-130900)
pythongh-130895: fix multiprocessing.Process join/wait/poll races
pythongh-130895: fix multiprocessing.Process join/wait/poll races