Skip to content

Benchmarks hang when executed on GPU #166

Closed

Description

@adarshyoga

All benchmark executions are hanging with the below error. This hang is not occurring in the CI. It is occurring, most likely, when the benchmarks are executed on a GPU. Disabling use of a separate process to run benchmark fixes the hang. Further investigation needed on executing device code on separate process.


ERROR:root:Benchmark execution failed due to the following error:
Traceback (most recent call last):
File "/home/ac.ayoga/work/dpbench/dpbench/runner.py", line 101, in run_benchmark
results = bench.run(
File "/home/ac.ayoga/work/dpbench/dpbench/infrastructure/benchmark.py", line 916, in run
runner = BenchmarkRunner(
File "/home/ac.ayoga/work/dpbench/dpbench/infrastructure/benchmark.py", line 491, in init
self.results.error_state = results_dict["error_state"]
File "", line 2, in getitem
File "/home/ac.ayoga/.conda/envs/dpbench/lib/python3.9/multiprocessing/managers.py", line 825, in _callmethod
raise convert_to_error(kind, result)
KeyError: 'error_state'

Activity

  1. diptorupd commented on Nov 16, 2022

    @diptorupd

    It might be related to us mixing Thread and MultiProcess. The exit_after decorator launches a new thread and we are calling it inside a new process. Ivan's PR #165 addresses it. Can you try removing the use of exit_after in benchmarks.py or just try PR #165?

  2. adarshyoga commented on Nov 16, 2022

    @adarshyoga
    ContributorAuthor

    It might be related to us mixing Thread and MultiProcess. The exit_after decorator launches a new thread and we are calling it inside a new process. Ivan's PR #165 addresses it. Can you try removing the use of exit_after in benchmarks.py or just try PR #165?

    The failure/hang persists even after removing exit_after.

  3. adarshyoga commented on May 18, 2023

    @adarshyoga
    ContributorAuthor

    Tested on discrete GPU. Did not encounter any hangs. Issue resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions