Repository navigation
Open
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
Bug description:
I am running a small processor that handles large tasks. When a task encounters an exception I want to store all errors (including other tasks errors) and cancel any other queued work.
However, it seems like the ProcessPoolExecutor's _ExecutorManagerThread does not notify any waiting threads of the future cancelations: making the idiom as_completed hang indefinitely.
Minimal Reproducable Example
import concurrent.futures
import time
def task(n: int) -> int:
if n == 2:
raise Exception("Not gonna do it")
else:
time.sleep(0.1)
return n
def main() -> None:
with concurrent.futures.ProcessPoolExecutor(
max_workers=2,
) as executor:
futures = [executor.submit(task, i) for i in range(1, 16)]
for future in concurrent.futures.as_completed(futures):
try:
result = future.result()
except Exception as e:
print(f"Exception: {e}")
executor.shutdown(wait=False, cancel_futures=True)
continue
print(f"Result: {result}")
# Look for 'CANCELLED' here:
print("Other futures states:", [f._state for f in futures])
if __name__ == "__main__":
main()Workaround
A workaround is to break after the first future that completed with a failure and do a post hoc gathering of other exceptions of non-cancelled futures.
Suggested Fix
In
cpython/Lib/concurrent/futures/process.py
Lines 520 to 522 in a68ddea
| for work_id, work_item in self.pending_work_items.items(): | |
| if not work_item.future.cancel(): | |
| new_pending_work_items[work_id] = work_item |
One could replace it with:
for work_id, work_item in self.pending_work_items.items():
canceled = work_item.future.cancel()
if canceled:
work_item.future.set_running_or_notify_cancel()
else:
new_pending_work_items[work_id] = work_itemSimilar like:
CPython versions tested on:
3.13
Operating systems tested on:
macOS
Linked PRs
Metadata
Metadata
Assignees
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Projects
- StatusShow more project fieldsNo status
Activity
duaneg commentedon Jul 15, 2025
Yep, looks like the same sort of issue as #109934. The unit test I added for that can't be used directly for process pool, but I think I've managed to come up with one that works reliably based on your repro. It will fail for the thread pool executor without the fix for #109934, so I'll base the PR on top of that branch.
pythongh-136655: ensure cancelled futures are notified on process poo…
YvesDup commentedon Oct 4, 2025
Except if I missed something, you should open a PR including your fix and a test.
duaneg commentedon Oct 5, 2025
I added the fix for this issue to #134618, which is the PR I opened to fix #109934. I did it that way rather than open a separate PR as otherwise the test will fail on the thread pool executor due to that bug, and I thought it likely a maintainer/reviewer would prefer to deal with them together.
If it is helpful I'd be happy to open a separate PR just for this issue, though. We can skip the test for the thread pool executor, with a TODO to re-enable it once #109934 is fixed.
YvesDup commentedon Oct 11, 2025
I suggest opening separately 2 PRs with tests.
pythongh-136655: ensure cancelled futures are notified on process poo…
duaneg commentedon Oct 13, 2025
I've not updated the PR for the other issue yet, but would be happy to do so in whatever way is most convenient for reviewers/maintainers.