Skip to content

Kill hung tensor tests with pytest-timeout and lower the Windows retry timeout in CI - #3152

Merged
antonwolfy merged 2 commits into
masterfrom
lower-win-tensor-tests-timeout
Nov 4, 2026
Merged

antonwolfy merged 2 commits into
masterfrom
lower-win-tensor-tests-timeout

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Lower the Windows tensor-tests retry timeout from 20 to 15 minutes and add a per-test timeout via pytest-timeout to the tensor tests in conda-package.yml and conda-build-test.yml (used by conda-package-cf.yml).

The dpnp.tests.tensor run sporadically hangs at 99% (IntelPython/dpctl#1954), on both Linux and Windows. Since October 1, in conda-package.yml, 48 of 885 Linux and 25 of 871 Windows tensor steps hit it. The log shows no progress for the rest of the attempt until the retry timeout kills it, and the next attempt passes.

Change Before After
Windows tensor retry timeout 20 min (rerun-tensor-tests-timeout-win) 15 min, the same rerun-tensor-tests-timeout as Linux
Per-test timeout (Linux and Windows) none --timeout=120 --timeout-method=thread

Healthy Windows tensor steps now take 4-6 min, and the slowest took 10-11 min since October 16 (conda-package-cf.yml: 4-9 min), so 15 min leaves room without the 20 min tail kept by #3132.

--timeout-method=thread is set explicitly because the hung main thread waits in native code (SyclQueue.wait()), where the default signal method on Linux would never fire. The thread method ends the stuck pytest-xdist worker: xdist reports the test as failed with its name and stack, starts a new worker and finishes the run, so a hang fails within ~2 min and the retry reruns the step. This was checked locally with a test that blocks in native code under -n 2 --timeout=5 --timeout-method=thread: worker 'gw1' crashed while running 'tests/test_hang.py::test_hang', 1 failed, 2 passed in 5.58s.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

…y timeout in CI

The tensor tests sporadically hang at 99% on both Linux and Windows
(IntelPython/dpctl#1954). Add a 120 s per-test
timeout with the thread method, so a hung xdist worker is killed and the
test is reported by name, and use the 15 min tensor retry timeout on
Windows too, since healthy Windows runs take 4-6 min.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The CI tensor-test timeout changes have no active review issues.

No open issues 🎉

What changed in this PR

This PR reduces time lost to sporadic tensor-test hangs in the package-testing CI workflows.

Changes:

  • Installs pytest-timeout and applies a 120-second, thread-based timeout to tensor tests on Linux and Windows.
  • Lowers the Windows tensor-test retry timeout from 20 to 15 minutes, matching Linux.
File Description
.github/​workflows/​conda-package.yml Adds per-test timeouts to direct and retried tensor runs and unifies the retry timeout.
.github/​workflows/​conda-build-test.yml Applies the same timeout settings to the reusable conda build-and-test workflow.

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/index.html

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev0=np2py314h8d9cdd5_25 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

coveralls commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.376%. remained the same — lower-win-tensor-tests-timeout into master

@antonwolfy
antonwolfy merged commit 58f4749 into master Nov 4, 2026
87 of 92 checks passed
@antonwolfy
antonwolfy deleted the lower-win-tensor-tests-timeout branch November 4, 2026 08:33
github-actions Bot added a commit that referenced this pull request Nov 4, 2026
…y timeout in CI (#3152)

Lower the Windows tensor-tests retry timeout from 20 to 15 minutes and
add a per-test timeout via `pytest-timeout` to the tensor tests in
`conda-package.yml` and `conda-build-test.yml` (used by
`conda-package-cf.yml`).

The `dpnp.tests.tensor` run sporadically hangs at 99%
(IntelPython/dpctl#1954), on both Linux and
Windows. Since October 1, in `conda-package.yml`, 48 of 885 Linux and 25
of 871 Windows tensor steps hit it. The log shows no progress for the
rest of the attempt until the retry timeout kills it, and the next
attempt passes.

| Change | Before | After |
|---|---|---|
| Windows tensor retry timeout | 20 min
(`rerun-tensor-tests-timeout-win`) | 15 min, the same
`rerun-tensor-tests-timeout` as Linux |
| Per-test timeout (Linux and Windows) | none | `--timeout=120
--timeout-method=thread` |

Healthy Windows tensor steps now take 4-6 min, and the slowest took
10-11 min since October 16 (`conda-package-cf.yml`: 4-9 min), so 15 min
leaves room without the 20 min tail kept by #3132.

`--timeout-method=thread` is set explicitly because the hung main thread
waits in native code (`SyclQueue.wait()`), where the default `signal`
method on Linux would never fire. The thread method ends the stuck
pytest-xdist worker: xdist reports the test as failed with its name and
stack, starts a new worker and finishes the run, so a hang fails within
~2 min and the retry reruns the step. This was checked locally with a
test that blocks in native code under `-n 2 --timeout=5
--timeout-method=thread`: `worker 'gw1' crashed while running
'tests/test_hang.py::test_hang'`, `1 failed, 2 passed in 5.58s`. 58f4749
antonwolfy added a commit that referenced this pull request Nov 11, 2026
…3177)

#3152 added a per-test timeout to the tensor tests of the conda package
workflows, for the hang at 99%
(IntelPython/dpctl#1954). `check-onemath.yaml`
hits the same hang but was left out, because it runs the tests without
xdist.

Measured on `check-onemath.yaml` runs since 2026-10-01, from the retry
annotations: 55 of 577 test jobs hit the 10 min timeout of the tensor
step, 8 of them on two attempts, so about 10.5 h of runner time went to
waiting. A healthy tensor step takes a median of 291 s (37 recent jobs).

This PR:
- runs the tensor tests with `--timeout=120 --timeout-method=thread`, as
`conda-package.yml` does;
- adds `-n auto` to them: without xdist the thread method exits the
whole pytest process at the first timeout, so the run stops there and
the retry starts from scratch. With xdist only the stuck worker is
killed and replaced, the run finishes, and the hung test is reported by
name;
- adds `pytest-timeout` and `pytest-xdist` to `environments/test.yml`.
`generate_coverage.yaml` also merges that file but passes neither
option, so its run doesn't change.

Checked locally with a test that hangs: a serial run with the thread
method ends at the hang, the tests after it don't run and no cache is
written, while with `-n 2` the worker is replaced and the other tests
all run.

Expected effect: a hang fails one test after 2 min instead of stalling
the step for 10 min, and xdist should also shorten the tensor step; this
PR's CI shows by how much.
github-actions Bot added a commit that referenced this pull request Nov 11, 2026
…3177)

#3152 added a per-test timeout to the tensor tests of the conda package
workflows, for the hang at 99%
(IntelPython/dpctl#1954). `check-onemath.yaml`
hits the same hang but was left out, because it runs the tests without
xdist.

Measured on `check-onemath.yaml` runs since 2026-10-01, from the retry
annotations: 55 of 577 test jobs hit the 10 min timeout of the tensor
step, 8 of them on two attempts, so about 10.5 h of runner time went to
waiting. A healthy tensor step takes a median of 291 s (37 recent jobs).

This PR:
- runs the tensor tests with `--timeout=120 --timeout-method=thread`, as
`conda-package.yml` does;
- adds `-n auto` to them: without xdist the thread method exits the
whole pytest process at the first timeout, so the run stops there and
the retry starts from scratch. With xdist only the stuck worker is
killed and replaced, the run finishes, and the hung test is reported by
name;
- adds `pytest-timeout` and `pytest-xdist` to `environments/test.yml`.
`generate_coverage.yaml` also merges that file but passes neither
option, so its run doesn't change.

Checked locally with a test that hangs: a serial run with the thread
method ends at the hang, the tests after it don't run and no cache is
written, while with `-n 2` the worker is replaced and the other tests
all run.

Expected effect: a hang fails one test after 2 min instead of stalling
the step for 10 min, and xdist should also shorten the tensor step; this
PR's CI shows by how much.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants