Skip to content

Commit a314f6d

Browse files
committed
Build and test conda packages and wheels for Intel Python CI
Adds what Intel Python CI (gh-ci) needs: a ci/ directory and .github/workflows/intel-ci.yml. The workflow builds conda-recipe-cf with conda-build and the wheels with cibuildwheel, for every platform and Python version the recipe and pyproject.toml declare, and tests each package in a clean environment with ci/test.py, on the OpenCL CPU device in Actions and on a GPU in Jenkins. Jenkins tests them again, scans them, runs ci/benchmark.py on the newest Linux package, and publishes a release tag's tested packages as its GitHub Release. conda-recipe-cf/ is now the only recipe: conda-recipe/ and conda-package.yml, which built it against Intel's channel and copied wheels out through WHEELS_OUTPUT_FOLDER, are removed. The dev uploads to anaconda.org and the cleanup of old dev packages move into intel-ci.yml, and run for pushes to master and tags only. conda_build_config.yaml now lists the Python versions, and the recipe's tests no longer run the full suite, which ci/test.py does. The wheels need the SYCL runtime libraries from intel-sycl-rt and intel-cmplr-lib-rt, so repair leaves them out and [tool.gh-ci] wheel-requires adds the two as requirements, as the wheels on PyPI have. scikit-build is pinned below 1.0, which no longer provides what setup.py imports (#2341).
1 parent 2dfa6bb commit a314f6d

25 files changed

Lines changed: 2234 additions & 1259 deletions

‎.github/workflows/backfill-docs.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,8 @@ jobs:
6464
echo "Using DPC++ ${COMPILER_VERSION} requested via workflow input."
6565
else
6666
# build against compiler version declared in the conda recipe
67-
COMPILER_VERSION=$(grep -oP 'required_compiler_version\s*=\s*"\K[^"]+' \
68-
conda-recipe/meta.yaml 2>/dev/null | head -n 1 || true)
67+
COMPILER_VERSION=$(grep -ohP 'required_compiler_version\s*=\s*"\K[^"]+' \
68+
conda-recipe-cf/meta.yaml conda-recipe/meta.yaml 2>/dev/null | head -n 1 || true)
6969
if [[ -z "${COMPILER_VERSION}" ]]; then
7070
echo "${TAG} declares no required_compiler_version. Using the latest DPC++."
7171
echo "package=${COMPILER_PACKAGE}" >> "$GITHUB_OUTPUT"

‎.github/workflows/conda-package.yml‎

Lines changed: 0 additions & 995 deletions
This file was deleted.

‎.github/workflows/intel-ci.yml‎

Lines changed: 260 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,260 @@
1+
# Intel Python CI (gh-ci): builds and tests the conda packages and wheels for Intel's Jenkins, see ci/README.md
2+
name: Intel Python CI
3+
4+
on:
5+
push:
6+
branches: [master]
7+
tags: ['*']
8+
pull_request:
9+
10+
permissions:
11+
contents: read
12+
13+
concurrency:
14+
group: intel-ci-${{ github.ref }}
15+
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
16+
17+
jobs:
18+
matrix:
19+
name: Platforms and Python versions
20+
runs-on: ${{ vars.GH_CI_RUNNER || 'ubuntu-latest' }}
21+
timeout-minutes: 10
22+
outputs:
23+
matrix: ${{ steps.declared.outputs.matrix }}
24+
wheel: ${{ steps.declared.outputs.wheel }}
25+
steps:
26+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
27+
with:
28+
persist-credentials: false
29+
30+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
31+
with:
32+
python-version: "3.14"
33+
34+
- name: Read the declared platforms and Python versions
35+
id: declared
36+
run: |
37+
python ci/matrix.py --event "$GITHUB_EVENT_NAME" --github-output matrix
38+
python ci/matrix.py --system wheel --event "$GITHUB_EVENT_NAME" --github-output wheel
39+
40+
# The recipe and pyproject.toml declare no macOS, as SYCL has none; Jenkins ignores it anyway.
41+
build:
42+
name: ${{ matrix.name }} Python ${{ matrix.python }}
43+
needs: matrix
44+
runs-on: ${{ matrix.platform == 'windows' && (vars.GH_CI_WINDOWS_RUNNER || 'windows-2022') || matrix.platform == 'macos' && (vars.GH_CI_MACOS_RUNNER || 'macos-26-intel') || (vars.GH_CI_RUNNER || 'ubuntu-latest') }}
45+
timeout-minutes: 150
46+
strategy:
47+
fail-fast: false
48+
matrix: ${{ fromJSON(needs.matrix.outputs.matrix) }}
49+
defaults:
50+
run:
51+
shell: bash -el {0}
52+
steps:
53+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
54+
with:
55+
persist-credentials: false
56+
# The recipe takes its version from git describe.
57+
fetch-depth: 0
58+
59+
# Miniforge 26.9.4-0 fails setup-miniconda's conda init on Windows runners; 26.7.2-0 works.
60+
- uses: conda-incubator/setup-miniconda@be893c923ea9cf1cf7cd510fbdde27c7e18cbdcb # v4.1.0
61+
with:
62+
miniforge-version: 26.7.2-0
63+
activate-environment: ""
64+
conda-remove-defaults: "true"
65+
66+
- name: Install conda-build
67+
run: conda install -y -q -n base --override-channels -c conda-forge conda-build
68+
69+
- name: Build and run the recipe tests
70+
env:
71+
PYTHON_VERSION: ${{ matrix.python }}
72+
run: python ci/build.py --python "$PYTHON_VERSION" --output "$RUNNER_TEMP/output"
73+
74+
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
75+
with:
76+
name: dpctl ${{ matrix.name }} Python ${{ matrix.python }}
77+
path: ${{ runner.temp }}/output/*.conda
78+
if-no-files-found: error
79+
# Jenkins downloads them within minutes of the run; tags keep them 30 days.
80+
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 3 }}
81+
82+
test:
83+
name: Test ${{ matrix.name }} Python ${{ matrix.python }}
84+
needs: [matrix, build]
85+
runs-on: ${{ matrix.platform == 'windows' && (vars.GH_CI_WINDOWS_RUNNER || 'windows-2022') || matrix.platform == 'macos' && (vars.GH_CI_MACOS_RUNNER || 'macos-26-intel') || (vars.GH_CI_RUNNER || 'ubuntu-latest') }}
86+
timeout-minutes: 60
87+
strategy:
88+
fail-fast: false
89+
matrix: ${{ fromJSON(needs.matrix.outputs.matrix) }}
90+
defaults:
91+
run:
92+
shell: bash -el {0}
93+
steps:
94+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
95+
with:
96+
persist-credentials: false
97+
98+
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
99+
with:
100+
name: dpctl ${{ matrix.name }} Python ${{ matrix.python }}
101+
path: ${{ runner.temp }}/artifacts
102+
103+
- uses: conda-incubator/setup-miniconda@be893c923ea9cf1cf7cd510fbdde27c7e18cbdcb # v4.1.0
104+
with:
105+
miniforge-version: 26.7.2-0
106+
activate-environment: ""
107+
conda-remove-defaults: "true"
108+
109+
- name: Install the package and run ci/test.py
110+
env:
111+
PYTHON_VERSION: ${{ matrix.python }}
112+
EVENT: ${{ startsWith(github.ref, 'refs/tags/') && 'tag' || github.event_name == 'pull_request' && 'pr' || 'push' }}
113+
COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
114+
run: >-
115+
python ci/hooks.py --python "$PYTHON_VERSION" --artifacts "$RUNNER_TEMP/artifacts"
116+
--work "$RUNNER_TEMP/hooks" --event "$EVENT" --commit "$COMMIT"
117+
118+
build-wheel:
119+
name: ${{ matrix.name }} Wheels Python ${{ matrix.python }}
120+
needs: matrix
121+
runs-on: ${{ matrix.platform == 'windows' && (vars.GH_CI_WINDOWS_RUNNER || 'windows-2022') || matrix.platform == 'macos' && (vars.GH_CI_MACOS_RUNNER || 'macos-26-intel') || (vars.GH_CI_RUNNER || 'ubuntu-latest') }}
122+
timeout-minutes: 150
123+
strategy:
124+
fail-fast: false
125+
matrix: ${{ fromJSON(needs.matrix.outputs.wheel) }}
126+
defaults:
127+
run:
128+
shell: bash
129+
steps:
130+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
131+
with:
132+
persist-credentials: false
133+
# versioneer takes the version from git describe.
134+
fetch-depth: 0
135+
136+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
137+
with:
138+
python-version: "3.14"
139+
140+
# CMake builds with Ninja and icx, which need MSVC's environment.
141+
- uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 # v1.13.0
142+
if: matrix.platform == 'windows'
143+
144+
- name: Install cibuildwheel
145+
run: python -m pip install --no-input cibuildwheel==4.3.5
146+
147+
- name: Build the wheel
148+
env:
149+
PYTHON_VERSION: ${{ matrix.python }}
150+
run: python ci/build.py --system wheel --python "$PYTHON_VERSION" --output "$RUNNER_TEMP/output"
151+
152+
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
153+
with:
154+
name: dpctl ${{ matrix.name }} Wheels Python ${{ matrix.python }}
155+
path: ${{ runner.temp }}/output/*.whl
156+
if-no-files-found: error
157+
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 3 }}
158+
159+
test-wheel:
160+
name: Test ${{ matrix.name }} Wheels Python ${{ matrix.python }}
161+
needs: [matrix, build-wheel]
162+
runs-on: ${{ matrix.platform == 'windows' && (vars.GH_CI_WINDOWS_RUNNER || 'windows-2022') || matrix.platform == 'macos' && (vars.GH_CI_MACOS_RUNNER || 'macos-26-intel') || (vars.GH_CI_RUNNER || 'ubuntu-latest') }}
163+
timeout-minutes: 60
164+
strategy:
165+
fail-fast: false
166+
matrix: ${{ fromJSON(needs.matrix.outputs.wheel) }}
167+
defaults:
168+
run:
169+
shell: bash -el {0}
170+
steps:
171+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
172+
with:
173+
persist-credentials: false
174+
175+
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
176+
with:
177+
name: dpctl ${{ matrix.name }} Wheels Python ${{ matrix.python }}
178+
path: ${{ runner.temp }}/artifacts
179+
180+
- uses: conda-incubator/setup-miniconda@be893c923ea9cf1cf7cd510fbdde27c7e18cbdcb # v4.1.0
181+
with:
182+
miniforge-version: 26.7.2-0
183+
activate-environment: ""
184+
conda-remove-defaults: "true"
185+
186+
- name: Install the wheel and run ci/test.py
187+
env:
188+
PYTHON_VERSION: ${{ matrix.python }}
189+
EVENT: ${{ startsWith(github.ref, 'refs/tags/') && 'tag' || github.event_name == 'pull_request' && 'pr' || 'push' }}
190+
COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
191+
run: >-
192+
python ci/hooks.py --kind wheel --python "$PYTHON_VERSION" --artifacts "$RUNNER_TEMP/artifacts"
193+
--work "$RUNNER_TEMP/hooks" --event "$EVENT" --commit "$COMMIT"
194+
195+
# The dev channel on anaconda.org, which docs and the nightly run-tests-from-dppy-bits.yaml read.
196+
upload:
197+
name: Upload to anaconda.org
198+
needs: [test, test-wheel]
199+
if: |
200+
github.repository == 'IntelPython/dpctl' &&
201+
github.event_name == 'push' &&
202+
(github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/'))
203+
runs-on: ${{ vars.GH_CI_RUNNER || 'ubuntu-latest' }}
204+
timeout-minutes: 20
205+
defaults:
206+
run:
207+
shell: bash
208+
steps:
209+
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
210+
with:
211+
pattern: dpctl *
212+
merge-multiple: true
213+
path: ${{ runner.temp }}/packages
214+
215+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
216+
with:
217+
python-version: "3.14"
218+
219+
- name: Install anaconda-client
220+
run: python -m pip install --no-input anaconda-client
221+
222+
- name: Upload the conda packages and the wheels
223+
env:
224+
ANACONDA_TOKEN: ${{ secrets.ANACONDA_TOKEN }}
225+
PACKAGES: ${{ runner.temp }}/packages
226+
run: |
227+
cd "$PACKAGES"
228+
conda_packages=(dpctl-*.conda)
229+
version=${conda_packages[0]#dpctl-}
230+
version=${version%%-*}
231+
anaconda --token "$ANACONDA_TOKEN" upload --user dppy --label dev dpctl-*.conda
232+
anaconda --token "$ANACONDA_TOKEN" upload --user dppy --label dev dpctl-*.whl --version "$version"
233+
234+
cleanup:
235+
name: Clean up anaconda.org
236+
needs: upload
237+
if: |
238+
github.repository == 'IntelPython/dpctl' &&
239+
github.event_name == 'push' &&
240+
(github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/'))
241+
runs-on: ${{ vars.GH_CI_RUNNER || 'ubuntu-latest' }}
242+
timeout-minutes: 30
243+
steps:
244+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
245+
with:
246+
persist-credentials: false
247+
repository: IntelPython/devops-tools
248+
fetch-depth: 0
249+
250+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
251+
with:
252+
python-version: "3.11"
253+
254+
- name: Install anaconda-client
255+
run: python -m pip install --no-input anaconda-client
256+
257+
- name: Remove the old dev packages
258+
env:
259+
ANACONDA_TOKEN: ${{ secrets.ANACONDA_TOKEN }}
260+
run: python scripts/cleanup-old-packages.py --verbose --force --token "$ANACONDA_TOKEN" --package dppy/dpctl --label dev

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1818

1919
### Maintenance
2020

21+
* Added Intel Python CI: `.github/workflows/intel-ci.yml` and `ci/` build the conda packages and the wheels (with `cibuildwheel`) and test them with `ci/test.py`; `conda-recipe-cf/` is now the only conda recipe, and `conda-recipe/` and `.github/workflows/conda-package.yml` were removed [gh-2408](https://github.com/IntelPython/dpctl/pull/2408)
22+
2123
## [0.23.0] - Oct. 22, 2026
2224

2325
This release extends the SYCL interface exposed by `dpctl` in several directions: `dpctl.compiler.SyclKernelBundle` objects can now be built from SYCL source strings via the DPC++ `kernel_compiler` extension, SPIR-V specialization constants are supported, USM-device allocations can be shared between processes using the SYCL IPC memory extension, and `dpctl.SyclQueue` gained `copy`, `memset`, and `fill` methods along with their asynchronous counterparts. The package can now be used with free-threaded builds of CPython 3.14 and supports Python 3.15. The `dpctl.program` submodule is deprecated in favor of `dpctl.compiler`, which better describes its purpose.

‎ci/README.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# How Intel Python CI tests dpctl
2+
3+
Intel Python CI (gh-ci) builds and tests dpctl's conda packages and wheels in GitHub Actions. Intel's Jenkins then tests each package again on Intel machines, including GPU ones, and scans it, and on a release tag publishes the packages it tested as the tag's GitHub Release. Everything it needs from this repository is in this directory, one workflow, the recipe, and `pyproject.toml`.
4+
5+
| File | Role |
6+
|---|---|
7+
| `../conda-recipe-cf/` | The conda recipe, built against conda-forge only. `conda_build_config.yaml` lists the Python versions, which has no 3.15t as conda-forge has no NumPy for it. The recipe skips macOS in `meta.yaml`, and its own tests only import dpctl and list the devices, so they need no device. |
8+
| `../pyproject.toml` | What the wheels are built from. Its Python classifiers list the wheels' Python versions, and its Operating System classifiers their platforms. `[tool.cibuildwheel]` holds the wheel build settings, and `[tool.gh-ci]` holds the run-time requirements the wheels need on top of `[project]`'s. |
9+
| `../.github/workflows/intel-ci.yml` | One job reads both matrices. One job per platform and Python version builds and uploads `dpctl <Platform> Python <version>`, and one builds `dpctl <Platform> Wheels Python <version>`. Then one job per package tests it. On a push to `master` or a tag, two more jobs upload the tested packages to the `dev` label of `dppy` on anaconda.org and remove its old ones. |
10+
| `matrix.py` | Reads the platforms and Python versions from the recipe, or with `--system wheel` from `pyproject.toml`. A pull request gets the oldest and the newest regular versions. |
11+
| `build.py` | Builds one Python version: with conda-build, which runs the recipe's own tests, or with `--system wheel` with cibuildwheel, in a manylinux container on Linux. |
12+
| `hooks.py` | The workflow's test jobs: installs one package into a clean environment with the requirements in `config.yml`, and runs `test.py` the way Jenkins does. A conda package's environment comes from conda-forge; a wheel's has only Python from conda-forge and everything else from PyPI. |
13+
| `install_icx.py` | Installs Intel's icx, which the conda recipe builds with too, for the Windows wheel builds: `[tool.cibuildwheel.windows]` in `../pyproject.toml` runs it first. The Linux wheel builds get icx from Intel's yum repository. Both get the Level Zero headers from a pinned tag of its GitHub repository. |
14+
| `config.yml` | Tells gh-ci which workflow builds the packages, what its artifacts are called, and what the tests and benchmarks need. |
15+
| `test.py` | Runs the installed package's test suite with pytest, in Actions and again in Jenkins. |
16+
| `benchmark.py` | Runs the asv suite in `../benchmarks/` against the installed package, in Jenkins only, on a bare-metal benchmarking machine. Jenkins requires it because `../benchmarks/` exists. |
17+
18+
The scripts need only the Python standard library and are the same in every repository gh-ci tests, except that `matrix.py` here looks for `conda-recipe-cf/` first. Scanning is not the repository's to define: Jenkins runs its own scans.
19+
20+
## Devices
21+
22+
dpctl's tests need a SYCL device, and GitHub's runners have none but a CPU. `test.py` prints the devices with `python -m dpctl -f` and fails if there is none, so a run that skipped every device test cannot pass.
23+
24+
- **In Actions** the device is the OpenCL CPU runtime of `intel-opencl-rt`, which `config.yml` installs from conda-forge for a conda package and from PyPI for a wheel. `test.py` points the OpenCL loader at it with `OCL_ICD_FILENAMES`, and on Windows also registers it under `HKLM\SOFTWARE\Khronos\OpenCL\Vendors`, as the loader in `System32` reads only that, which needs the administrator rights a GitHub runner gives.
25+
- **In Jenkins** the GPU agents set `GH_CI_DEVICE=gpu`. Then `test.py` leaves the loader alone, apart from letting a Linux environment's loader see the system's ICDs too, and fails unless a GPU device is present.
26+
27+
Two tests compile an extension when they run, `test_c_headers` and `test_cython_api`. `test.py` deselects them, and says so, when setuptools finds no C and C++ compiler. A conda package's environment gets `c-compiler` and `cxx-compiler` from `config.yml`; a wheel's environment uses the machine's.
28+
29+
## Wheels
30+
31+
The wheels link the SYCL runtime that `intel-sycl-rt` and `intel-cmplr-lib-rt` ship, and find it through their RPATH, or the DLL search path on Windows, so `auditwheel` and `delvewheel` leave those libraries out. `[tool.gh-ci] wheel-requires` adds the two as requirements to each wheel, as the wheels on PyPI have. icx is pinned to 2026.2 to match.
32+
33+
## Uploads
34+
35+
A push to `master` or a tag, in `IntelPython/dpctl` only and after every test passed, uploads the conda packages and the wheels the workflow built to `dppy` on anaconda.org with the label `dev`, with the repository secret `ANACONDA_TOKEN`, and then removes the old dev packages. No other event or branch gets the secret.
36+
37+
**A pull request into `master`.** Actions builds the oldest and the newest regular Python versions on Linux and Windows, as a conda package and as a wheel, and tests each package. Jenkins notices the pull request within 3 minutes, waits for that whole run, and then tests and scans every package at once, and benchmarks the newest Linux conda package, sampling each benchmark once. Each cell posts its own status on the pull request, and `intel-ci/pr` sums them up.
38+
39+
**A release tag.** Actions builds every declared version, and Jenkins tests and scans all of them, and measures the benchmarks properly on the newest Linux conda package, before it publishes the release. A benchmark that fails fails the run; a slower one does not, as nothing compares the timings with a baseline yet.
40+
41+
To support another Python, add it to `conda-recipe-cf/conda_build_config.yaml` for the conda packages and as a classifier in `pyproject.toml` for the wheels; nothing else lists versions. Try it locally from a conda base environment with conda-build, or with cibuildwheel for wheels:
42+
43+
```bash
44+
python ci/matrix.py # every platform and Python version
45+
python ci/matrix.py --event pull_request # what a pull request builds
46+
python ci/build.py --python VERSION --output /tmp/out # VERSION: one matrix.py lists
47+
python ci/hooks.py --python VERSION --artifacts /tmp/out --work /tmp/hooks
48+
python ci/build.py --system wheel --python VERSION --output /tmp/wheels
49+
python ci/hooks.py --kind wheel --python VERSION --artifacts /tmp/wheels --work /tmp/wheel-hooks
50+
```

0 commit comments

Comments
 (0)