Skip to content

Use cibuildwheel to automatically build all the wheels, avoid manual builds#935

Open
BishopWolf wants to merge 5 commits intoOpenGATE:masterfrom
BishopWolf:test-cibuildwheel
Open

Use cibuildwheel to automatically build all the wheels, avoid manual builds#935
BishopWolf wants to merge 5 commits intoOpenGATE:masterfrom
BishopWolf:test-cibuildwheel

Conversation

@BishopWolf
Copy link
Copy Markdown
Contributor

@BishopWolf BishopWolf commented Mar 9, 2026

This PR is intended to open a discussion to address the issue #928

This changes current wheel generation. CONFIRM AND APROVE BEFORE MERGE!!!

  • Windows is producing all the wheels now, fully auto using delvewheel
  • Linux is producing all the wheels now, fully auto
  • Macos is producing all the wheels now, fully auto
  • All Tests passing
  • Add Qt to Windows. Will be made in another PR
  • cache brew for macos, this will reduce build time. Will be made in another PR

Current status:

  • Currently we build one wheel per python version and per system.
  • The selection of python versions is manual. But we start with a pinned version, then we let CIBW build the correct wheel for the desired version.
  • Total build time averages 45 minutes before tests. Each build takes 16 minutes preparation and around 15 minutes build, but not all start at the same time.

Possible improvements:

  • We could let cibuildwheel select the python version automagically. This is scalable in time as cibuildwheel knows which python versions correspond to each system.
  • We could use a single system to build the tagged wheels, this will simplify a lot the action scripts. Even for novis builds you can just use python 3.12 to build the wheel for 3.9 if needed.
  • Total build time will average 50 minutes dropping unneeded python versions. 16 minutes preparation ONCE and 10 minutes per build for each python version. In the case of macos, the delocation also takes 10 more minutes, its almost instantaneous in linux and windows.

@BishopWolf
Copy link
Copy Markdown
Contributor Author

BishopWolf commented Mar 10, 2026

@tbaudier right now we only support these os
os: [ubuntu-24.04, macos-15, windows-2025]

however, to support more we just need to add them, they will be built by default
os: [ubuntu-24.04, ubuntu-24.04-arm, macos-15, macos-15-intel, windows-2025, windows-11-arm]

ALSO: No need to further worry about python version, the variable requires-python = ">=3.9" defines everything automagically

@BishopWolf
Copy link
Copy Markdown
Contributor Author

@tbaudier This is clean and ready, you can merge if approved

@BishopWolf BishopWolf changed the title [Do not Merge] Test cibuildwheel Test cibuildwheel Mar 16, 2026
@tbaudier
Copy link
Copy Markdown
Contributor

Thank you, First we want to do the release and then I will have a look to your PR

@BishopWolf
Copy link
Copy Markdown
Contributor Author

@tbaudier windows is fully automatic now.

@BishopWolf
Copy link
Copy Markdown
Contributor Author

Windows and Linux build in less than 10 minutes per python version
Macos needs on average 16 minutes, this should be addressed

@BishopWolf
Copy link
Copy Markdown
Contributor Author

BishopWolf commented Mar 18, 2026

@tbaudier we can leave the one wheel per python version if you think the builds are too slow.
For me they are fine like this as cibuildwheel is automagically selecting the correct python versions for each system.
The change is not that hard to do, just add back the python matrix and add this to each script (it will work on all of them)

if [[ ${MATRIX_PYTHON_VERSION} == "3.10" ]]; then
  export CIBW_BUILD="cp310-*"
elif [[ ${MATRIX_PYTHON_VERSION} == "3.11" ]]; then
  export CIBW_BUILD="cp311-*"
elif [[ ${MATRIX_PYTHON_VERSION} == "3.12" ]]; then
  export CIBW_BUILD="cp312-*"
elif [[ ${MATRIX_PYTHON_VERSION} == "3.13" ]]; then
  export CIBW_BUILD="cp313-*"
elif [[ ${MATRIX_PYTHON_VERSION} == "3.14" ]]; then
  export CIBW_BUILD="cp314-*"
fi

EDIT: I can see it's definitely faster, i'll push the changes

@BishopWolf BishopWolf force-pushed the test-cibuildwheel branch 2 times, most recently from f3cb3d9 to 556613e Compare March 20, 2026 09:17
@BishopWolf BishopWolf changed the title Test cibuildwheel Use cibuildwheel to automatically build all the wheels, avoid manual builds Mar 23, 2026
@BishopWolf
Copy link
Copy Markdown
Contributor Author

@tbaudier I can add back python 3.10 support for both windows and macos, it is building fine for me.

@BishopWolf
Copy link
Copy Markdown
Contributor Author

@BishopWolf
Copy link
Copy Markdown
Contributor Author

brew qt should be fine for all macos versions accordingly to https://formulae.brew.sh/formula/qt. the cache should handle it

@BishopWolf
Copy link
Copy Markdown
Contributor Author

BishopWolf commented Mar 25, 2026

I have rebased the last working commit to current master. I can add the intended modifications in a different PR.

Current issues:
When adding QT in windows : the virtual machine can't find the conda location. https://github.com/BishopWolf/opengate/tree/windows-qt6

When using homebrew cache in mac : cibuildwheel can't find system python

@BishopWolf
Copy link
Copy Markdown
Contributor Author

@tbaudier @dsarrut @nkrah Let me know when are you available to discuss this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants