You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
All benchmark executions are hanging with the below error. This hang is not occurring in the CI. It is occurring, most likely, when the benchmarks are executed on a GPU. Disabling use of a separate process to run benchmark fixes the hang. Further investigation needed on executing device code on separate process.
ERROR:root:Benchmark execution failed due to the following error:
Traceback (most recent call last):
File "/home/ac.ayoga/work/dpbench/dpbench/runner.py", line 101, in run_benchmark
results = bench.run(
File "/home/ac.ayoga/work/dpbench/dpbench/infrastructure/benchmark.py", line 916, in run
runner = BenchmarkRunner(
File "/home/ac.ayoga/work/dpbench/dpbench/infrastructure/benchmark.py", line 491, in init
self.results.error_state = results_dict["error_state"]
File "", line 2, in getitem
File "/home/ac.ayoga/.conda/envs/dpbench/lib/python3.9/multiprocessing/managers.py", line 825, in _callmethod
raise convert_to_error(kind, result)
KeyError: 'error_state'
It might be related to us mixing Thread and MultiProcess. The exit_after decorator launches a new thread and we are calling it inside a new process. Ivan's PR #165 addresses it. Can you try removing the use of exit_after in benchmarks.py or just try PR #165?
It might be related to us mixing Thread and MultiProcess. The exit_after decorator launches a new thread and we are calling it inside a new process. Ivan's PR #165 addresses it. Can you try removing the use of exit_after in benchmarks.py or just try PR #165?
The failure/hang persists even after removing exit_after.
Activity
diptorupd commentedon Nov 16, 2022
It might be related to us mixing
ThreadandMultiProcess. Theexit_afterdecorator launches a new thread and we are calling it inside a new process. Ivan's PR #165 addresses it. Can you try removing the use ofexit_afterin benchmarks.py or just try PR #165?adarshyoga commentedon Nov 16, 2022
The failure/hang persists even after removing
exit_after.adarshyoga commentedon May 18, 2023
Tested on discrete GPU. Did not encounter any hangs. Issue resolved.