Skip to content

Build and test conda packages and wheels - #3151

Open
jharlow-intel wants to merge 1 commit into
masterfrom
ci/intel-python-ci
Open

jharlow-intel wants to merge 1 commit into
masterfrom
ci/intel-python-ci

Conversation

@jharlow-intel

@jharlow-intel jharlow-intel commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Adds ci/ and .github/workflows/intel-ci.yml, which build the conda packages and wheels for Linux and Windows on every declared Python, and test each in a clean environment.

  • conda-recipe/ is now the only recipe: the conda-forge one that was conda-recipe-cf/. It builds against conda-forge, with dpctl's dev builds from dppy/label/dev after it. The Intel-channel recipe, with rattler_recipe.yaml, conda-package.yml, conda-package-cf.yml and conda-build-test.yml are removed; the Array API conformance and pybind11 examples jobs move to intel-ci.yml.
  • The dppy/label/dev uploads and their cleanup move into intel-ci.yml, for pushes to master and tags only. Each upload has every declared version: the conda packages for Python 3.11 to 3.15 and 3.14t, which need conda-forge's runtimes at least at the versions they were built with (such as libstdcxx >=15 and mkl >=2026.1) and ocl-icd-system on Linux, and the wheels for cp311 to cp315 and cp314t, which require the oneAPI runtime wheels from PyPI. Against what the label has today, 3.11 and 3.12 are new and nothing stops; a push adds more files, so the cleanup keeps fewer past builds.
  • Wheels build with cibuildwheel and icx 2026.2 against dpctl's dev wheels from the dppy/label/dev index, which environments/dpctl_wheel_build.txt limits to dpctl, and their tests take the same; [build-system] adds the oneAPI runtime wheels that dpctl's import needs at build time, which its dev wheels don't declare. They require the oneAPI runtime wheels (dpcpp-cpp-rt, intel-sycl-rt, mkl, onemkl-sycl-*), as the PyPI wheels do, each at least the version the build used ([tool.gh-ci.pin-compatible]). cp315t is skipped, as dpctl has no wheel for it yet. The Linux wheels link the image's gcc-toolset libstdc++, and ci/build.py fails a wheel that needs a newer glibc, libstdc++ or libgcc symbol version than its manylinux_2_28 tag allows. The Windows wheels build with IPO.
  • README.md and the quick start guide install from conda-forge and PyPI.
  • ci/test.py runs dpnp.tests, then dpnp.tests.tensor, against the installed package, and fails when it finds no SYCL device: the OpenCL CPU device in Actions, or a GPU with GH_CI_DEVICE=gpu.

The required checks on master come from conda-package.yml, so they need replacing with this workflow's, such as Test Linux Python 3.11, before this can merge.

Validation: run 44577660132 built and tested the conda packages and the wheels for 3.11, the newest regular Python and 3.14t on Linux and Windows, and ran the examples and the Array API job. Not run yet: a GPU, and the uploads, which only master and tags do.

@jharlow-intel jharlow-intel self-assigned this Nov 3, 2026
@jharlow-intel jharlow-intel changed the title Build and test conda packages and wheels for Intel Python CI Build and test conda packages and wheels Nov 5, 2026
@jharlow-intel
jharlow-intel force-pushed the ci/intel-python-ci branch 2 times, most recently from 6aca46e to e9f220a Compare November 9, 2026 14:42
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/3151/index.html

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev2+7.g9c2119f18 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

coveralls commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.269%. remained the same — ci/intel-python-ci into master

Comment thread .github/workflows/intel-ci.yml Resolved
Comment thread ci/test.py Resolved
Comment thread conda-recipe/meta.yaml Resolved
Comment thread .github/workflows/conda-package-cf.yml Resolved
Comment thread ci/hooks.py Resolved
Comment thread doc/quick_start_guide.rst Resolved
Comment thread AGENTS.md Outdated
| --- | --- | --- |
| pre-commit (`Check`) | `.github/workflows/pre-commit.yml` | yes |
| conda build and tests, Linux and Windows, py3.13-3.15 and 3.14t (`Test (...)`) | `conda-package.yml` | yes |
| conda packages and wheels built and tested, Linux and Windows, py3.11-3.15 and 3.14t for conda, py3.10-3.14 and 3.14t for wheels (`Test <Platform> Python <version>`, `Test <Platform> Wheels Python <version>`); the conda matrix is `conda-recipe/conda_build_config.yaml`, the wheel matrix the Python classifiers in `pyproject.toml`; see `ci/README.md` | `intel-ci.yml` | no |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wheels are 3.11-3.14 and 3.14t: pyproject.toml has no 3.10 classifier, as #3173 dropped 3.10. Also, the old row was required on master. Is a required check planned for intel-ci.yml, or should branch protection just drop the old Test (...) checks?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the wheel range to 3.11-3.14 and 3.14t. The required Test (...) checks come from conda-package.yml, which this PR removes, so branch protection needs them swapped for intel-ci.yml's jobs before this can merge.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add1e50 has the right wheel range, thanks. If intel-ci.yml's test jobs replace the required Test (...) checks, this row should say yes in the last column, as the old one did.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they replace them, so the row says yes now. Swapping the required checks is a branch protection change for an admin.

Comment thread ci/matrix.py Resolved
Comment thread ci/config.yml Resolved
Comment thread CHANGELOG.md Resolved
@jharlow-intel
jharlow-intel force-pushed the ci/intel-python-ci branch 2 times, most recently from b53329b to c93fd5e Compare November 10, 2026 22:49
Comment thread .github/workflows/conda-package.yml Resolved
Comment thread pyproject.toml Resolved
Comment thread .github/AGENTS.md
- Every job sets `timeout-minutes`; for a job that calls a reusable workflow, set it inside the called workflow.
- `actionlint` runs in pre-commit.
- Set up conda with the local `.github/actions/setup-miniconda` action, which retries a failed setup once, instead of calling `conda-incubator/setup-miniconda` directly; a job that checks dpnp out into a subdirectory uses it from there (`./source/.github/actions/setup-miniconda`).
- `workflows/intel-ci.yml` is Intel CI: its conda matrix is `conda-recipe/conda_build_config.yaml`, its wheel matrix the Python classifiers in `pyproject.toml`, and its upload and cleanup jobs run only for pushes to `master` and tags in `IntelPython/dpnp`; see `ci/README.md`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the rebase, line 16 here and the comment at .github/dependabot.yml:21 (both from #3171) still name conda-recipe/rattler_recipe.yaml and conda-recipe-cf/meta.yaml, which this PR removes. With one recipe left, conda-recipe/meta.yaml is a dev-series pin and [build-system].requires the only release-only one. Could both places say that?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both now list conda-recipe/meta.yaml among the dev-series pins and [build-system].requires as the only release-only one.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for moving all of it into one workflow with ci/. I ran the Linux artifacts of this head (CI run 44355775222) through ci/hooks.py the way the test jobs do, with GH_CI_DEVICE=gpu, on an Intel Arc Pro B70 (Ubuntu 26.04), and checked the cp311 wheel in ubuntu:24.04 and debian:13 containers on the CPU device.

On the GPU everything passes:

Package dpnp.tests dpnp.tests.tensor
conda 3.11 83056 passed, 6.1 min 15535 passed, 1.7 min
conda 3.15 83329 passed, 6.4 min 15535 passed, 1.6 min
conda 3.14t 83330 passed, 6.6 min 15535 passed, 1.7 min
wheel 3.11 83056 passed, 6.2 min 15535 passed, 1.7 min
wheel 3.14t 83330 passed, 6.6 min 15535 passed, 1.7 min

No failures and no retries. test.py picks level_zero:gpu and runs the suites serially, and the main suite needs 6-7 min of the 60-min MAIN_TIMEOUT, so a GPU runner fits. With only opencl:cpu visible, GH_CI_DEVICE=gpu fails with "GH_CI_DEVICE asks for a GPU, and there is none", as intended.

The points are inline. The ones I'd settle before merging: the wheels building against the released dpctl (pyproject.toml:34) and the change to what dppy/label/dev serves (intel-ci.yml:427).

Outside the diff:

  • The seven dpnp/tensor Cython modules lose their ABI suffix in the Linux wheels (dpnp/tensor/_usmarray.so and so on), while master's wheels and every conda package have _usmarray.cpython-311-x86_64-linux-gnu.so. The likely cause is the second find_package(Python COMPONENTS Development.Module) in dpnp/tensor/CMakeLists.txt. They still import, so the impact is low.
  • The PR description could say outright that no public workflow builds or tests against the Intel channel any more.

Good to see:

  • The wheels now declare the oneAPI runtime wheels in Requires-Dist, with floors taken from the build (master's dev wheels declared none).
  • No absolute RPATH in the wheels, and RELRO, BIND_NOW and the stack protector everywhere.
  • about.json records the source URL and commit, and the devops-tools checkout is pinned.
  • scipy is in the test requirements now, and the retries and the tensor --timeout=120 match the removed workflow.
  • The explicit oneMKL run pins are tighter than the feedstock's.
  • The conda package loads and runs in a rockylinux:8 container with its own libstdc++.

Comment thread pyproject.toml
environment = { LD_LIBRARY_PATH = "/opt/stdcxx", CPATH = "/opt/sdk/include", CC = "/opt/intel/oneapi/compiler/2026.2/bin/icx", CXX = "/opt/intel/oneapi/compiler/2026.2/bin/icpx", CMAKE_GENERATOR = "Ninja", CMAKE_ARGS = "-DDPNP_WITH_REDIST:BOOL=ON", CMAKE_BUILD_PARALLEL_LEVEL = "4", CMAKE_PREFIX_PATH = "/opt/sdk", MKLROOT = "/opt/sdk", LDFLAGS = "-Wl,-z,relro -Wl,-z,now" }
# The SYCL runtime, oneMKL and the compiler's libraries come from wheels at run time, see [tool.gh-ci].
# libstdc++ is as new as dpctl's wheel needs, which auditwheel's policy refuses, so the tag is set here.
repair-wheel-command = "python -m pip install -q wheel && python -m wheel tags --remove --platform-tag manylinux_2_28_x86_64 {wheel} && mv \"$(dirname {wheel})\"/*-manylinux_2_28_x86_64.whl {dest_dir}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Linux wheels are tagged manylinux_2_28 but need GLIBCXX_3.4.32, which that tag doesn't allow. objdump -T gives 3.4.32 as the highest version in the cp311, cp314 and cp314t wheels of this head (_ZSt21ios_base_library_initv, in libdpnp_backend_c.so, libtensor/_vm_impl and libtensor/_lapack_impl); the conda packages of this head top out at 3.4.29. auditwheel's policy allows up to 3.4.24 for manylinux_2_28, and 3.4.32 first fits manylinux_2_39, the Ubuntu 24.04 baseline.

On Ubuntu that's fine: the cp311 wheel loads and runs BLAS, LAPACK, FFT, RNG and VM calls on the CPU device in ubuntu:24.04 and debian:13 containers (libstdc++ 3.4.33). But the tag tells pip the wheels run on any glibc 2.28 system, and one with an older libstdc++ gets them installed and fails at import. The 0.21.0 wheels on PyPI (3.4.29) have the same mismatch, so it isn't new, but nothing checks it, and the CI can't see it: the ubuntu-26.04 runner's libstdc++ has 3.4.35.

Could the tag match what the wheels need, and be checked?

  • With Ubuntu 24.04 as the oldest target, tag them manylinux_2_39. Keeping manylinux_2_28 would need the modules to link to 3.4.24 at most. My guess at why they don't: the link resolves -lstdc++ to conda-forge's libstdc++.so in /opt/sdk/lib, not to the image's gcc-toolset one, whose linker script adds libstdc++_nonshared.a for the newer symbols.
  • Fail the build when a module needs more than the tag allows, e.g. auditwheel repair --plat manylinux_2_39_x86_64 with --exclude for the oneAPI runtime libraries, or an objdump -T check in ci/build.py.

The comment at :133 and the sentence about GCC 13 in ci/README.md:24 would change with it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on the two options: I'd prefer the second, building within the manylinux_2_28 policy, over the manylinux_2_39 tag, for two reasons I left out:

  • manylinux_2_39 would shut out systems where these wheels work. SLES 15 SP7, which the 2026.2 compiler supports, has glibc 2.38 and a libstdc++ from GCC 13 or newer, so the wheels should load there, but pip won't install a manylinux_2_39 wheel on it.
  • Where pip skips the wheel, it doesn't fail. There is no dpnp sdist on PyPI, so pip quietly installs the newest release whose wheel fits (0.21.0 today), and only a pinned version fails with "No matching distribution".

Building to the tag keeps every system the wheels install on today and makes the tag true.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Built to manylinux_2_28, as you suggest: the Linux before-all moves conda-forge's libstdc++ out of /opt/sdk/lib, where -lstdc++ found it, and keeps it only for dpctl's import at build time, so the link takes gcc-toolset's linker script. ci/build.py now reads each ELF file's version needs and fails a wheel that needs a newer GLIBC, GLIBCXX, CXXABI or GCC version than its tag allows. On this head's cp311 wheel the newest GLIBCXX any module needs is 3.4.22, down from 3.4.32. The comment at :133 and ci/README.md say so now.

Comment thread pyproject.toml Outdated
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3.15",
# No 3.15: dpctl has no cp315 wheel yet, which the wheels build against

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wheels build against dpctl 0.23.0 from PyPI, but require and run on 0.24.0dev0. [build-system] has dpctl>=0.23.0 and cibuildwheel gets no extra index, so the wheel build log has Found Dpctl … (found version "0.23.0"). The wheels declare dpctl>=0.24.0dev0, and their tests install 0.24.0.dev0 from the dev index. The conda packages build against 0.24.0dev0 from dppy/label/dev. A change in dpctl 0.24's Cython types or C API would then break only the wheels, possibly without an error, since they are compiled against 0.23's headers and .pxd files. dpctl and dpnp are released together, so building against the released dpctl isn't needed here.

Could cibuildwheel get the dev index too, e.g. PIP_EXTRA_INDEX_URL = "https://pypi.anaconda.org/dppy/label/dev/simple" and PIP_PRE = "1" in the environment of [tool.cibuildwheel.linux] and [tool.cibuildwheel.windows]? The dev index has cp315 dpctl wheels, so this also brings back the cp315 wheels and the 3.15 classifier this line removes.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: cibuildwheel gets the dppy/label/dev index, and environments/dpctl_wheel_build.txt is pip's constraint, so only dpctl takes a dev build; PIP_PRE would also have let in pre-releases such as scikit-build 1.0.0rc1. The wheel builds now log Found Dpctl ... (found version "0.24.0dev0+53.gdd4f089d80"). [build-system] also adds intel-cmplr-lib-rt and intel-sycl-rt: dpctl's import at build time needs them, and its dev wheels don't declare them. The 3.15 classifier is back, with cp315t skipped, as the dev index has no cp315t dpctl wheel yet.

Comment thread conda-recipe/meta.yaml Outdated
{% set max_compiler_and_mkl_version = environ.get("MAX_BUILD_CMPL_MKL_VERSION", "2027.0a0") %}
{% set required_compiler_version = environ.get("MIN_BUILD_CMPL_VERSION", "2026.1.1") %}
{% set required_mkl_version = "2026.0" %}
{% set required_compiler_and_mkl_version = "2026.0.0" %}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lowers the minimum compiler from 2026.1.1 to 2026.0.0. Master's conda-recipe/meta.yaml had separate floors, the compiler at 2026.1.1 (#2905) and oneMKL at 2026.0, and the 0.21.0 notes in CHANGELOG.md:63 say the release requires DPC++ compiler 2026.1.1 or newer. This recipe, the former conda-recipe-cf/, sets both from one value (:32 and :47).

Could it keep two floors as master did, the compiler at 2026.1.1 and oneMKL at 2026.0.0? oneMKL doesn't have to follow the compiler: its 2026.0 packages on conda-forge need only dpcpp-cpp-rt 2026.* and intel-sycl-rt >=2026.0.0,<2027.0a0, so they install with a 2026.1.1 or 2026.2 compiler. The wheels already pin the compiler to 2026.2, so nothing else changes.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: two floors again, the compiler at 2026.1.1 and oneMKL at 2026.0.0.

Comment thread .github/workflows/intel-ci.yml Outdated
activate-environment: ""

- name: Install conda-build
run: conda install -y -q -n base --override-channels -c conda-forge conda-build "py-lief<0.17"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could conda-build come from environments/build_conda_pkg.yml again, instead of unpinned here? The PR leaves that file unused; it still pins conda-build=25.11.1 and conda-verify=3.4.2. Bumping it to conda-build=26.9.1, which this step resolves today, keeps the behaviour and lets Dependabot update it. py-lief<0.17 could move there with a comment saying why it's needed, and conda-verify can go if nothing uses it.

The same for the test requirements: ci/config.yml:8 repeats environments/test.yml, which the coverage and oneMath workflows still use. One list would keep them from drifting.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conda-build now comes from environments/build_conda_pkg.yml, at 26.9.1, and conda-verify is gone, as nothing uses it. py-lief<0.17 moved there too, with its reason: without it, conda-build's Windows linkage check fails with 'lief._lief.COFF.Binary' object has no attribute 'exported_functions'. The test lists stay apart: ci/config.yml's is what hooks.py installs for the conda packages and, next to wheel_requires, for the wheels, while environments/test.yml is a conda environment file.

Comment thread .github/workflows/intel-ci.yml Outdated
python-version: "3.14"

- name: Install cibuildwheel
run: python -m pip install --no-input cibuildwheel==4.3.5

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pin is out of Dependabot's sight. A requirements file under environments/, whose pip ecosystem Dependabot already watches, would keep cibuildwheel current.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to environments/cibuildwheel.txt, which Dependabot's pip entry watches.

Comment thread ci/test.py Outdated
MAIN_TIMEOUT = 60 * 60
TENSOR_TIMEOUT = 20 * 60
# The Python version whose main suite runs every integer type
ALL_INT_TYPES_PYTHON = "3.14"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The all-integer-types run depends on this hardcoded "3.14". On pull requests the conda packages build 3.11, 3.15 and 3.14t, so it runs only in the two 3.14 wheel jobs. Once the cp315 wheels are back (see the comment on pyproject.toml:34), the wheels build 3.11, 3.15 and 3.14t too, and no PR job runs it at all. Could it follow the matrix instead, e.g. the newest regular version, which every event builds?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It follows the matrix now: the newest regular version the package's build system declares, which is 3.15 for the conda packages and the wheels.

Comment thread .github/workflows/intel-ci.yml Outdated
path: ${{ runner.temp }}/output/*.conda
if-no-files-found: error
# Tags keep them 30 days, pull requests and pushes to master 3.
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 3 }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushes to master now keep their packages for 3 days, down from the 90-day default the removed workflow had. That's short for bisecting a regression to a master build, and the dev label keeps only a couple of pushes (see the comment on the upload job). Could master pushes keep 30-90 days, with PRs staying at 3?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushes to master and tags keep them 30 days now; pull requests stay at 3.

upload:
name: Upload to anaconda.org
needs: [test, test-wheel]
if: github.repository == 'IntelPython/dpnp' && github.event_name == 'push' && (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/'))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes what dpnp users get from dppy/label/dev, and the change should be announced and agreed before the merge.

  • Pushes to master upload every declared version: conda 3.11-3.15 and 3.14t, wheels cp311-cp314 and cp314t. Today the label has conda 3.13, 3.14, 3.14t and 3.15, and wheels cp313, cp314, cp314t and cp315. So the cp315 wheels for Linux and Windows stop; 3.11 and 3.12 are new.
  • The cp315 wheels already there go too. cleanup-old-packages.py removes the oldest dev files while dppy/dpnp is over its 1 GB quota. The label holds 38 files, 765 MB, and one push of this workflow adds 22 files of about 21.6 MB each, about 475 MB. So the cp315 wheels of 2026-11-13/14 are gone within about two pushes, and fewer past builds are kept in general.
  • Every package changes too. The conda packages now need conda-forge's libstdcxx >=15, mkl >=2026.1 and ocl-icd-system, so an environment that mixes them with Intel-channel runtimes can break. pip install of a wheel now also installs the oneAPI runtime wheels (about 3 GB), which master's dev wheels didn't declare.

Could the PR description list the new set (the Python versions for conda and pip, and the conda-forge dependencies), so it can be announced to dpnp users and agreed first? Until then, keeping the cp315 wheels (by building against the dev dpctl, see pyproject.toml:34) would avoid a gap. Are uploads to the dev label on tags wanted as well?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description now lists what a push to master uploads to dppy/label/dev, and the conda-forge runtimes the conda packages need, so it can be announced and agreed. With the cp315 wheels back, nothing the label has today stops; 3.11 and 3.12 are new.

Comment thread ci/README.md Outdated
| `test.py` | Runs the installed package's test suites with pytest: `dpnp.tests`, then `dpnp.tests.tensor` on its own, which needs a lot of memory. It sets the SYCL and OpenCL variables itself, because the test environment has none. |
| `benchmark.py` | Runs the asv suite in `../benchmarks/` against the installed package. |

The scripts need only the Python standard library.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite: hooks.py needs conda, conda-package-handling and ruamel.yaml, as its docstring says, and test.py runs pytest. Maybe "The scripts need the Python standard library and the tools each one's docstring names".

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took your wording.

Comment thread ci/matrix.py Outdated
"s390x": False,
},
}
CONDA_SKIP_RE = re.compile(r"\s*skip\s*:\s*(?:true|True|yes)\s*#\s*\[(.+)\]\s*")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this misses a skip: line with a trailing comment after the selector, or a selector that isn't a platform. That's fine for today's skip: true # [osx], but a later recipe edit could make the matrix build what the recipe skips.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: matrix.py reads skip: lines with conda-build's selector pattern, which allows a comment after the selector, and refuses a skip that isn't platform-only, as the mkl repositories' copy does.

@jharlow-intel
jharlow-intel force-pushed the ci/intel-python-ci branch 3 times, most recently from c0e07ef to 4bbd3e2 Compare November 19, 2026 17:18
Adds a ci/ directory and .github/workflows/intel-ci.yml, which build the
conda packages and the wheels for every declared platform and Python and
test each with ci/test.py. A pull request builds the oldest and the newest
regular Python and the newest free-threaded one, and every push and tag run
gets a concurrency group of its own.

conda-recipe/ is now the only recipe, the conda-forge one that was
conda-recipe-cf/. The Intel-channel recipe and conda-package.yml are
removed, and so are conda-package-cf.yml and its conda-build-test.yml,
whose builds intel-ci.yml repeats.
The dev uploads to anaconda.org and their cleanup, the Array API conformance
job and the pybind11 examples move to intel-ci.yml. The examples and the Array API job
wait for the builds, not the tests, take the newest regular and
free-threaded Linux packages, which every event builds, and install dpnp
from the built packages only.

The benchmarks take PyYAML from conda-forge, as PyPI has no Python 3.15
wheel for it yet. ci/build.py refuses two built packages with the same file
name, which would overwrite each other in its flat output directory, and
takes METADATA's line ending from its headers. conda-build records the
commit each conda package came from in info/about.json, and LDFLAGS links
the Linux wheels' libraries with full RELRO.

ci/test.py reruns a failed suite, twice for the main tests and three times
for the tensor tests, and stops a tensor test stuck for 120 s. GH_CI_DEVICE
and ONEAPI_DEVICE_SELECTOR reach the tests. The recipe ignores mkl-dpcpp's
run export, and the ci/ scripts carry the BSD-3 header.

The conda builds and test environments take dpctl's dev builds from
dppy/label/dev after conda-forge, with flexible channel priority, and the
wheel tests from its pip index after PyPI: ci/build.py and ci/hooks.py get
--channel and --extra-index-url from intel-ci.yml. The wheels build against
them too: [tool.cibuildwheel] adds the index, and
environments/dpctl_wheel_build.txt is pip's constraint, so only dpctl takes
a dev build. [build-system] adds intel-cmplr-lib-rt and intel-sycl-rt, which
dpctl's import at build time needs and its dev wheels don't declare. The
3.15 classifier is back, and [tool.cibuildwheel] skip leaves out cp315t,
which ci/matrix.py honors, as dpctl has no wheel for it yet. The examples
and the Array API job pin dpnp to the exact build.
[tool.gh-ci.pin-compatible] maps each runtime wheel to the compiler or
oneMKL package the build installed; the Linux before-all and
ci/install_icx.py print those versions and ci/build.py requires >= each, <
its next major. The upload takes each wheel's version from its file name and
fails when there is nothing to upload. Every job sets conda up with
.github/actions/setup-miniconda.

The Linux wheels link the manylinux image's gcc-toolset libstdc++: the
before-all moves conda-forge's out of /opt/sdk/lib, where -lstdc++ found it,
and keeps it for dpctl's import at build time only. ci/build.py reads each
ELF file's version needs and fails a wheel that needs a newer GLIBC,
GLIBCXX, CXXABI or GCC version than its manylinux tag allows. LDFLAGS adds
--as-needed, --disable-new-dtags and --gc-sections, as conda-forge's do, and
the Windows wheels build with IPO again. dpnp/tensor no longer looks Python
up a second time, which left its Cython modules without an ABI suffix.

The recipe has the compiler floor at 2026.1.1 and oneMKL's at 2026.0.0
again, keeps numpy's run export with numpy >=1.26.0, takes ninja on every
platform, and runs python -m dpctl -f in its test; conda_build_config.yaml
names each Python's ABI. intel-ci.yml installs conda-build from
environments/build_conda_pkg.yml and cibuildwheel from
environments/cibuildwheel.txt, which Dependabot watches, uses one Miniforge,
and keeps the packages of pushes to master 30 days. ci/test.py runs every
integer type on the newest regular Python the matrix declares, and
ci/matrix.py reads skip: lines with conda-build's selector pattern.
README.md, the quick start guide and cron-run-tests.yaml install from
conda-forge and PyPI.
@jharlow-intel

Copy link
Copy Markdown
Collaborator Author

On the tensor modules' missing ABI suffix from the review summary: the second find_package(Python COMPONENTS Development.Module) in dpnp/tensor/CMakeLists.txt was it. It's gone, and the Linux wheels have dpnp/tensor/_usmarray.cpython-311-x86_64-linux-gnu.so again.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants