ci: tag lockstep releases and publish to PyPI with trusted publishing #8

Part of NVIDIA/OpenShell#5287, phases 5 and 6. It depends on #2 and #3 being on main and on the bot from #5 and #7, which creates the release tags.

Problem

Only NVIDIA/OpenShell's release workflow can publish openshell to PyPI today, through the pypi environment in core, and the PyPI project trusts only that workflow. The latest release is 0.1.7. sdk-python has no release workflow and no way to create release tags.

Proposal

How a release runs

Publishing runs from main, like core. Core's pypi environment allows only main, and its release workflow is dispatched on main. Here the trigger is the merge of a stable bump PR. All jobs run in one workflow run whose ref is refs/heads/main. That is what lets both environments stay limited to main. A pushed tag never triggers publishing, so a stray v* tag can't publish anything.

.github/workflows/release.yml runs on push to main, and by hand (workflow_dispatch on main) for retries and TestPyPI:

1. tag (environment sdk-bot, App token; sdk-go's tag-release.sh):
   • It runs only when the pin in OPENSHELL_CORE_REF changed to a stable core tag. Every other push to main stops here.
   • It creates the lightweight tag vX.Y.Z at the merged commit, plus a draft GitHub release whose body is the bump PR's description and "Matches OpenShell core vX.Y.Z".
   • Every step is skipped when already done, so a re-run finishes a partial release.
2. build (no id-token, no secrets):
   • Check out the tagged commit and run mise run build. That builds the sdist and wheel, verifies the wheel against the tag's version with verify-python-wheel.py, and installs both on Python 3.11. setuptools-scm derives the version from the tag.
   • Write openshell-checksums-sha256.txt and upload dist/ as an artifact.
3. publish-pypi (environment pypi, id-token: write only):
   • Download the artifact and publish with pypa/gh-action-pypi-publish, pinned by SHA.
   • Trusted publishing needs no token. The action uploads PEP 740 attestations by default.
   • A dispatch input repository: testpypi switches this job to the testpypi environment and TestPyPI.
4. github-release: attach the wheel, the sdist and the checksums to the draft, then publish it. Publishing comes last so the release never appears without its files, and so the flow already works if we turn on immutable releases (see below).

Notes on the design:

• The build job doesn't publish, and the publish job doesn't build. This keeps the OIDC token away from build steps, as PyPI recommends.
• PyPI's trusted publisher names the workflow file that contains the publish job, release.yml.
• A pre-release pin creates nothing. Core doesn't publish pre-releases to PyPI either.
• Immutable releases: decided later. Not enabled for now; it may be too constraining while the release flow is new. Once published, an immutable release's files and tag can't change (its notes can). The draft-then-publish order already supports it, so enabling it later needs only the repository setting.
• No dev releases. To try unreleased changes, install from the repository: pip install "openshell @ git+https://github.com/ai-openshell/sdk-python@main". The bindings are committed, so this needs no core checkout, and setuptools-scm gives the build a dev version (X.Y.Z.devN+g<sha>).

Admin setup

• PyPI and TestPyPI trusted publishers: ci: PyPI trusted publisher for sdk-python #9.
• The pypi and testpypi GitHub environments: ci: pypi and testpypi environments #10.
• Core's publisher stays until core's final wheel is out, then it is removed (chore(python): migrate the Python SDK to ai-openshell/sdk-python NVIDIA/OpenShell#5287, phase 5).

Decisions

1. The first version from sdk-python is 0.1.8.

• Core's v0.1.7 wheel is the last one core builds. Core must remove publish-pypi before it tags v0.1.8; chore(python): migrate the Python SDK to ai-openshell/sdk-python NVIDIA/OpenShell#5287 tracks that. Otherwise core publishes 0.1.8 first, and PyPI rejects ours.
• If this repository's automation isn't ready when core tags v0.1.8, we publish 0.1.8 from here later, by hand. The version stays free as long as core doesn't publish it.

2. SDK releases follow core releases.

• Core releases every Tuesday, and the SDK releases the same day, from the bump PR for that core tag. There are no SDK-only releases between core releases. An SDK fix waits for the next Tuesday, which keeps the version scheme simple.

• On a release day, the steps run in this order:

  1. Core tags vX.Y.Z.
  2. core-watch opens the bump PR within the hour (ci: propose bindings updates for new core releases and track drift against core main #5).
  3. E2E runs against the published gateway (test: run an E2E suite against the published gateway of the pinned core version #6).
  4. A maintainer merges the bump PR, and release.yml publishes.

  The merge is the only manual step.

• If an urgent SDK-only fix is ever needed, we revisit this. A post-release (X.Y.Z.post1) is the likely option.

Acceptance criteria

• release.yml tags vX.Y.Z and creates a draft GitHub release when a stable bump merges, and does nothing on other pushes or for pre-release pins.
• It builds once, publishes with trusted publishing from the pypi environment, attaches the artifacts and checksums to the draft, and publishes the release last.
• A dry run against TestPyPI succeeds and installs with pip install -i https://test.pypi.org/simple/ openshell.
• ci: PyPI trusted publisher for sdk-python #9 and ci: pypi and testpypi environments #10 are done.
• The README describes the release procedure, the two decisions above, and installing from the repository to try unreleased changes.
• The first release installs as a drop-in replacement for 0.1.7: same exports and the same openshell._proto modules.