Repository navigation
Kill hung tensor tests with pytest-timeout and lower the Windows retry timeout in CI - #3152
Merged
Merged
Conversation
…y timeout in CI The tensor tests sporadically hang at 99% on both Linux and Windows (IntelPython/dpctl#1954). Add a 120 s per-test timeout with the thread method, so a hung xdist worker is killed and the test is reported by name, and use the 15 min tensor retry timeout on Windows too, since healthy Windows runs take 4-6 min.
antonwolfy
requested review from
ndgrigorian and
vlad-perevezentsev
as code owners
November 3, 2026 18:57
There was a problem hiding this comment.
🟢 Approval recommended
The CI tensor-test timeout changes have no active review issues.
No open issues 🎉
What changed in this PR
This PR reduces time lost to sporadic tensor-test hangs in the package-testing CI workflows.
Changes:
- Installs
pytest-timeoutand applies a 120-second, thread-based timeout to tensor tests on Linux and Windows. - Lowers the Windows tensor-test retry timeout from 20 to 15 minutes, matching Linux.
| File | Description |
|---|---|
.github/workflows/conda-package.yml |
Adds per-test timeouts to direct and retried tensor runs and unifies the retry timeout. |
.github/workflows/conda-build-test.yml |
Applies the same timeout settings to the reusable conda build-and-test workflow. |
Review effort: Balanced
Contributor
|
View rendered docs @ https://intelpython.github.io/dpnp/index.html |
Contributor
|
Array API standard conformance tests for dpnp=0.22.0dev0=np2py314h8d9cdd5_25 ran successfully. |
Collaborator
ndgrigorian
approved these changes
Nov 4, 2026
github-actions Bot
added a commit
that referenced
this pull request
Nov 4, 2026
…y timeout in CI (#3152) Lower the Windows tensor-tests retry timeout from 20 to 15 minutes and add a per-test timeout via `pytest-timeout` to the tensor tests in `conda-package.yml` and `conda-build-test.yml` (used by `conda-package-cf.yml`). The `dpnp.tests.tensor` run sporadically hangs at 99% (IntelPython/dpctl#1954), on both Linux and Windows. Since October 1, in `conda-package.yml`, 48 of 885 Linux and 25 of 871 Windows tensor steps hit it. The log shows no progress for the rest of the attempt until the retry timeout kills it, and the next attempt passes. | Change | Before | After | |---|---|---| | Windows tensor retry timeout | 20 min (`rerun-tensor-tests-timeout-win`) | 15 min, the same `rerun-tensor-tests-timeout` as Linux | | Per-test timeout (Linux and Windows) | none | `--timeout=120 --timeout-method=thread` | Healthy Windows tensor steps now take 4-6 min, and the slowest took 10-11 min since October 16 (`conda-package-cf.yml`: 4-9 min), so 15 min leaves room without the 20 min tail kept by #3132. `--timeout-method=thread` is set explicitly because the hung main thread waits in native code (`SyclQueue.wait()`), where the default `signal` method on Linux would never fire. The thread method ends the stuck pytest-xdist worker: xdist reports the test as failed with its name and stack, starts a new worker and finishes the run, so a hang fails within ~2 min and the retry reruns the step. This was checked locally with a test that blocks in native code under `-n 2 --timeout=5 --timeout-method=thread`: `worker 'gw1' crashed while running 'tests/test_hang.py::test_hang'`, `1 failed, 2 passed in 5.58s`. 58f4749
This was referenced Nov 8, 2026
antonwolfy
added a commit
that referenced
this pull request
Nov 11, 2026
…3177) #3152 added a per-test timeout to the tensor tests of the conda package workflows, for the hang at 99% (IntelPython/dpctl#1954). `check-onemath.yaml` hits the same hang but was left out, because it runs the tests without xdist. Measured on `check-onemath.yaml` runs since 2026-10-01, from the retry annotations: 55 of 577 test jobs hit the 10 min timeout of the tensor step, 8 of them on two attempts, so about 10.5 h of runner time went to waiting. A healthy tensor step takes a median of 291 s (37 recent jobs). This PR: - runs the tensor tests with `--timeout=120 --timeout-method=thread`, as `conda-package.yml` does; - adds `-n auto` to them: without xdist the thread method exits the whole pytest process at the first timeout, so the run stops there and the retry starts from scratch. With xdist only the stuck worker is killed and replaced, the run finishes, and the hung test is reported by name; - adds `pytest-timeout` and `pytest-xdist` to `environments/test.yml`. `generate_coverage.yaml` also merges that file but passes neither option, so its run doesn't change. Checked locally with a test that hangs: a serial run with the thread method ends at the hang, the tests after it don't run and no cache is written, while with `-n 2` the worker is replaced and the other tests all run. Expected effect: a hang fails one test after 2 min instead of stalling the step for 10 min, and xdist should also shorten the tensor step; this PR's CI shows by how much.
github-actions Bot
added a commit
that referenced
this pull request
Nov 11, 2026
…3177) #3152 added a per-test timeout to the tensor tests of the conda package workflows, for the hang at 99% (IntelPython/dpctl#1954). `check-onemath.yaml` hits the same hang but was left out, because it runs the tests without xdist. Measured on `check-onemath.yaml` runs since 2026-10-01, from the retry annotations: 55 of 577 test jobs hit the 10 min timeout of the tensor step, 8 of them on two attempts, so about 10.5 h of runner time went to waiting. A healthy tensor step takes a median of 291 s (37 recent jobs). This PR: - runs the tensor tests with `--timeout=120 --timeout-method=thread`, as `conda-package.yml` does; - adds `-n auto` to them: without xdist the thread method exits the whole pytest process at the first timeout, so the run stops there and the retry starts from scratch. With xdist only the stuck worker is killed and replaced, the run finishes, and the hung test is reported by name; - adds `pytest-timeout` and `pytest-xdist` to `environments/test.yml`. `generate_coverage.yaml` also merges that file but passes neither option, so its run doesn't change. Checked locally with a test that hangs: a serial run with the thread method ends at the hang, the tests after it don't run and no cache is written, while with `-n 2` the worker is replaced and the other tests all run. Expected effect: a hang fails one test after 2 min instead of stalling the step for 10 min, and xdist should also shorten the tensor step; this PR's CI shows by how much.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lower the Windows tensor-tests retry timeout from 20 to 15 minutes and add a per-test timeout via
pytest-timeoutto the tensor tests inconda-package.ymlandconda-build-test.yml(used byconda-package-cf.yml).The
dpnp.tests.tensorrun sporadically hangs at 99% (IntelPython/dpctl#1954), on both Linux and Windows. Since October 1, inconda-package.yml, 48 of 885 Linux and 25 of 871 Windows tensor steps hit it. The log shows no progress for the rest of the attempt until the retry timeout kills it, and the next attempt passes.rerun-tensor-tests-timeout-win)rerun-tensor-tests-timeoutas Linux--timeout=120 --timeout-method=threadHealthy Windows tensor steps now take 4-6 min, and the slowest took 10-11 min since October 16 (
conda-package-cf.yml: 4-9 min), so 15 min leaves room without the 20 min tail kept by #3132.--timeout-method=threadis set explicitly because the hung main thread waits in native code (SyclQueue.wait()), where the defaultsignalmethod on Linux would never fire. The thread method ends the stuck pytest-xdist worker: xdist reports the test as failed with its name and stack, starts a new worker and finishes the run, so a hang fails within ~2 min and the retry reruns the step. This was checked locally with a test that blocks in native code under-n 2 --timeout=5 --timeout-method=thread:worker 'gw1' crashed while running 'tests/test_hang.py::test_hang',1 failed, 2 passed in 5.58s.