From bf928575ac75c160c1c56a1ea51df8adacaad78e Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 15:05:35 -0500 Subject: [PATCH 01/11] docs: Add the plan for the cortex-gateway release (#1723) Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- ...026-11-10-praxis-mode-7-release-gateway.md | 186 ++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 docs/superpowers/plans/2026-11-10-praxis-mode-7-release-gateway.md diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-7-release-gateway.md b/docs/superpowers/plans/2026-11-10-praxis-mode-7-release-gateway.md new file mode 100644 index 000000000..67b2330ac --- /dev/null +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-7-release-gateway.md @@ -0,0 +1,186 @@ +# Praxis mode 7/8: Release cortex-gateway as a standalone binary — implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Issue:** #1723 (part of #1714). **Depends on:** #1719 and #1724, both closed and merged. + +**Goal:** Ship `cortex-gateway` in each release as a standalone tarball for darwin/arm64, linux/amd64 and linux/arm64, with no system library to install. A gateway failure must not block the Go release. + +**Architecture:** One POSIX `sh` build script builds the release binary on the host it runs on (native macOS, or `rust:alpine` for a static musl Linux binary) and then runs one linkage check script. The release workflow gets a `build-gateway` matrix job that calls the build script, and `publish` treats the gateway tarballs as optional. A new weekly workflow builds against the newest Praxis release with the same script and opens an issue when it fails. + +**Tech Stack:** POSIX sh, GitHub Actions, Cargo (Rust 1.96.0, pinned in `praxis/rust-toolchain.toml`), `rust:1.96.0-alpine`, `readelf`, `otool`. + +**Spec:** `docs/superpowers/specs/2026-11-07-praxis-mode-design.md`, Section 4 "Release and install" and "A standalone binary", Section 6 step 7. + +## What already exists (do not redo) + +- `praxis/server/Cargo.toml` already turns on `openssl = { version = "0.10", features = ["vendored"] }`. No Cargo change is necessary. +- `praxis/server/src/main.rs` already prints `cortex-gateway ` for `--version`. `` is the build-time env `CORTEX_VERSION`, or `dev`. `cmd/cortex/supervise.go` refuses a gateway whose version differs from `cortex`. So the release build MUST set `CORTEX_VERSION=${VERSION}`. +- `scripts/install.sh --praxis` already downloads `cortex-gateway___.tar.gz`, expects one file `cortex-gateway` in it, and refuses a release whose `checksums.txt` does not list that file (`require_praxis_build`). The tarball name and its content are therefore fixed. +- `.github/dependabot.yml` already has a `cargo` entry for `/praxis`. It proposes `openssl-src` bumps. No change. +- `Cargo.lock` has no `aws-lc-sys` (TLS uses `ring` and vendored OpenSSL). The native code to link statically is `openssl-src`, `zstd-sys`, `tikv-jemalloc-sys` and `ring`. + +## Global Constraints + +- Targets: darwin/arm64, linux/amd64, linux/arm64. No darwin/amd64 build. +- Tarball name: `cortex-gateway_${TAG}_${os}_${arch}.tar.gz`. It holds one file: `cortex-gateway`. +- `cortex-gateway --version` must print exactly `cortex-gateway ${VERSION}`. +- Linux binaries: static musl, built in `rust:1.96.0-alpine` (pinned by index digest), with `-C target-feature=+crt-static`. The ELF must have no `NEEDED` entry. +- macOS binaries: only `/usr/lib/` and `/System/Library/` libraries in `otool -L`. `MACOSX_DEPLOYMENT_TARGET=12.0`. +- A gateway build failure does not block the Go release. `build` or `build-cpex` failure still blocks it, as today. +- The weekly Praxis job does not block anything, and it reports a failure. +- Pin every action and every container image by digest, as the rest of `.github/workflows/` does. +- Shell scripts pass `shellcheck --severity=error`. Workflows pass the relaxed `yamllint` config in `security-scans.yaml`. +- Prose follows the root `CLAUDE.md` naming rule (say "Cortex" for the product) and is in ASD-STE100 Simplified Technical English. + +## Decisions taken in this plan (and why) + +1. **One PR.** The estimate is about 600 counted lines, far below 2000. The tasks are small and share the build script. +2. **Build logic lives in `scripts/build-gateway.sh`, not inline YAML.** The release job, the weekly job and the local gate all run the same file. `build-cpex` keeps its inline `BUILD` because it is not reused. +3. **The script `cd`s into `praxis/`.** rustup reads `rust-toolchain.toml` only from the current directory and its parents, not from `--manifest-path` (see the note in `praxis.yaml`). +4. **The Linux build passes `--target `.** With `--target`, `RUSTFLAGS` does not reach build scripts and proc-macros. A proc-macro cannot be a `crt-static` musl artifact, so without `--target` the build fails. It also puts the output in `target//release`, so a macOS build and a container build in one checkout do not overwrite each other. +5. **`JEMALLOC_SYS_WITH_LG_PAGE=16` on linux/arm64.** jemalloc fixes the page size at build time. The arm64 runner has 4 KiB pages, but some arm64 Linux hosts use 64 KiB pages, and a 4 KiB jemalloc crashes there. 64 KiB (`2^16`) also works on 4 KiB hosts. +6. **Strip with `CARGO_PROFILE_RELEASE_STRIP=symbols` in the script.** The Go release strips with `-s -w`. The env keeps `praxis/Cargo.toml` unchanged for developers. +7. **`MACOSX_DEPLOYMENT_TARGET=12.0`.** Go 1.26 binaries need macOS 12, so the gateway claims the same floor. +8. **`build-gateway` uses `fail-fast: false`.** Each platform is optional on its own. `install.sh --praxis` checks per `os/arch` in `checksums.txt`, so two of three tarballs is a valid release. +9. **`publish` runs when `build` and `build-cpex` succeed, whatever `build-gateway` did:** `if: ${{ !cancelled() && needs.build.result == 'success' && needs.build-cpex.result == 'success' }}`. A missing gateway tarball gives a `::warning::`, not an error. +10. **On `main-latest`, the prune step deletes an old gateway asset when this build has none.** That is correct: `checksums.txt` would not list the old asset, so nothing could verify it. `install.sh --praxis --ref=main` then refuses with its existing message. +11. **The release notes name the gateway only when at least one gateway tarball is present.** +12. **No build cache in the release job.** A release builds clean, as `build-cpex` does. The weekly job also builds clean, so that it sees what a release sees. +13. **The clean-host check on Linux runs the binary in three places:** the glibc runner, plain `alpine` (musl), and `gcr.io/distroless/static-debian12` (no libc and no OpenSSL). With no `NEEDED` entry, that covers the issue's "glibc Linux and musl Linux with no OpenSSL package". Plain `alpine` is not the "no OpenSSL" host, because its base image ships `libssl3`. +14. **The weekly job reads the newest Praxis version from crates.io** (`https://crates.io/api/v1/crates/praxis-proxy`, field `.crate.max_stable_version`, with a `User-Agent` header, which crates.io requires). It rewrites the `=X.Y.Z` pins of `praxis-proxy` and `praxis-proxy-filter` in both Cargo manifests, runs `cargo update -p praxis-proxy -p praxis-proxy-filter`, `cargo test --locked`, then the static Linux build. It runs also when the newest version is the pinned one: it then checks the current pin against a fresh toolchain image. +15. **The weekly report is one GitHub issue.** On failure, a `report` job (`issues: write`) comments on the open issue titled `Weekly build against the latest Praxis failed`, or opens it. A human closes it. A red scheduled run alone reaches only the person who last edited the cron line. +16. **No change to `praxis.yaml`.** A static build on every `praxis/**` PR costs about 10 minutes. The release job and the weekly job catch a static-link break. Not in scope of #1723. +17. **No change to `release-smoke-linux.yaml`.** Its job count feeds the install statistics (see its header), so a new leg is not a local decision. + +## Review Focus + +1. A release where `build` passes, `build-cpex` passes and `build-gateway` fails on every leg: `publish` must run, publish no gateway asset, and warn. Check the `if:` of `publish` and the presence check (Task 2). +2. A release where `build-cpex` fails and `build-gateway` passes: `publish` must NOT run. The new `if:` must not open that path (Task 2). +3. `readelf` that fails or prints nothing (not an ELF file, or a missing tool): the check must fail, never read "no NEEDED lines" as static. Pinned by `test_readelf_error_fails` (Task 1). +4. A Homebrew OpenSSL path in `otool -L` (`/opt/homebrew/opt/openssl@3/lib/libssl.3.dylib`): the check must fail and name the library. Pinned by `test_darwin_homebrew_ssl_fails` (Task 1). +5. A binary that prints `cortex-gateway dev` in a release (`CORTEX_VERSION` not passed into the container): the check must fail, because the supervisor would then refuse it. Pinned by `test_version_mismatch_fails` (Task 1). + +## PRs + +| PR | Title | Parent | Tasks | estLines | +|---|---|---|---|---| +| P1 | `feat: Release cortex-gateway as a standalone binary` | (none) | T1, T2, T3, T4 | 600 | + +## Gate + +Run from the worktree root. It needs `pre-commit`, `shellcheck`, `yamllint`, a running `podman` machine, and Rust through rustup. It takes about 20 minutes on a cold cache. + +```sh +pre-commit run --all-files \ +&& shellcheck --severity=error scripts/build-gateway.sh scripts/check-gateway-linkage.sh scripts/check-gateway-linkage_test.sh \ +&& sh scripts/check-gateway-linkage_test.sh \ +&& yamllint -d '{extends: relaxed, rules: {line-length: {max: 150, level: warning}, truthy: {check-keys: false}, document-start: disable, comments: {min-spaces-from-content: 1}}}' .github/workflows \ +&& VERSION=gate-test sh scripts/build-gateway.sh \ +&& podman run --rm -v "$PWD:/src" -w /src -e VERSION=gate-test docker.io/library/rust:1.96.0-alpine sh scripts/build-gateway.sh \ +&& podman run --rm -v "$PWD/dist:/d:ro" gcr.io/distroless/static-debian12 /d/cortex-gateway --version +``` + +--- + +### Task T1: Build script and linkage check + +**Depends on:** none. + +**Files:** +- Create: `scripts/build-gateway.sh` +- Create: `scripts/check-gateway-linkage.sh` +- Create: `scripts/check-gateway-linkage_test.sh` +- Modify: `.github/workflows/ci.yaml` (add a job after `release-notes-script`, same shape) + +**Interfaces:** +- Produces: `sh scripts/build-gateway.sh`. Run from the repo root. Env `VERSION` is required (die with a message when empty). Writes `dist/cortex-gateway` and exits non-zero when the build or the check fails. Optional env `MACOSX_DEPLOYMENT_TARGET` (default `12.0`). +- Produces: `sh scripts/check-gateway-linkage.sh `. Exit 0 when the binary is standalone and prints `cortex-gateway `. Exit 1 otherwise, with the reason on stderr. Uses `uname -s` to pick `otool -L` (Darwin) or `readelf -d` (Linux). Any other OS: exit 1. + +- [ ] **Step 1: Write the failing test `scripts/check-gateway-linkage_test.sh`.** Follow the shape of `scripts/release_notes_test.sh` (POSIX sh, one function per case, a failure counter, exit 1 when any case fails). Each case makes a temp dir with stub `uname`, `otool` and `readelf` scripts first on `PATH`, and a stub binary that prints a given `--version` line. Cases and expected results: + - `test_linux_static_passes`: `uname` prints `Linux`, `readelf -d` prints `There is no dynamic section in this file.` and exits 0, binary prints `cortex-gateway v1.2.3`. Expect exit 0. + - `test_linux_needed_fails`: `readelf -d` prints ` 0x0000000000000001 (NEEDED) Shared library: [libssl.so.3]`. Expect exit 1, and stderr contains `libssl.so.3`. + - `test_readelf_error_fails`: `readelf` prints nothing and exits 1. Expect exit 1. + - `test_darwin_system_libs_pass`: `uname` prints `Darwin`, `otool -L` prints the binary path line, then `\t/usr/lib/libSystem.B.dylib (...)` and `\t/System/Library/Frameworks/Security.framework/Versions/A/Security (...)`. Expect exit 0. + - `test_darwin_homebrew_ssl_fails`: `otool -L` also lists `\t/opt/homebrew/opt/openssl@3/lib/libssl.3.dylib (...)`. Expect exit 1, and stderr contains `libssl.3.dylib`. + - `test_version_mismatch_fails`: static Linux stubs, binary prints `cortex-gateway dev`, version argument `v1.2.3`. Expect exit 1, and stderr contains `cortex-gateway dev`. +- [ ] **Step 2: Run** `sh scripts/check-gateway-linkage_test.sh`. Expected: FAIL (the script under test does not exist). +- [ ] **Step 3: Write `scripts/check-gateway-linkage.sh`.** Linux: run `readelf -d`, fail when it exits non-zero, fail and print each line that contains `(NEEDED)`. Darwin: run `otool -L`, skip its first line (the binary's own path), fail and print each dependency path that does not start with `/usr/lib/` or `/System/Library/`. Then run ` --version` and compare it with `cortex-gateway ` exactly. +- [ ] **Step 4: Run** `sh scripts/check-gateway-linkage_test.sh` and `shellcheck --severity=error scripts/check-gateway-linkage.sh scripts/check-gateway-linkage_test.sh`. Expected: all cases pass, no shellcheck error. +- [ ] **Step 5: Write `scripts/build-gateway.sh`** (POSIX sh, `set -eu`). Order: + 1. Die when `VERSION` is empty. + 2. When `/etc/alpine-release` exists: `apk add --no-cache musl-dev perl make linux-headers`. (perl and make build `openssl-src`. make builds jemalloc.) + 3. `cd praxis && rustup toolchain install` (decision 3). + 4. `triple=$(rustc -vV | sed -n 's/^host: //p')`. + 5. Linux: `export RUSTFLAGS="-C target-feature=+crt-static"`. On `aarch64-*` also `export JEMALLOC_SYS_WITH_LG_PAGE=16` (decision 5). Darwin: `export MACOSX_DEPLOYMENT_TARGET="${MACOSX_DEPLOYMENT_TARGET:-12.0}"`. + 6. `CORTEX_VERSION="${VERSION}" CARGO_PROFILE_RELEASE_STRIP=symbols cargo build --release --locked -p cortex-gateway --target "${triple}"`. + 7. Copy `target/${triple}/release/cortex-gateway` to `../dist/cortex-gateway` (`mkdir -p ../dist`). + 8. From the repo root: `sh scripts/check-gateway-linkage.sh dist/cortex-gateway "${VERSION}"`. + Each non-obvious line gets a one-line comment with its reason from the decisions above. +- [ ] **Step 6: Prove the static musl build first (the issue's first scope item).** Run `podman run --rm -v "$PWD:/src" -w /src -e VERSION=gate-test docker.io/library/rust:1.96.0-alpine sh scripts/build-gateway.sh`. Expected: exit 0 and the check passes. Then `podman run --rm -v "$PWD/dist:/d:ro" gcr.io/distroless/static-debian12 /d/cortex-gateway --version` prints `cortex-gateway gate-test`. If a native crate does not link statically, fix it in this script (an env var or an `apk` package) and record the fix in a comment. Do not change Praxis. +- [ ] **Step 7: Run the macOS build:** `VERSION=gate-test sh scripts/build-gateway.sh`. Expected: exit 0. `otool -L dist/cortex-gateway` lists only `/usr/lib/` and `/System/Library/` paths. +- [ ] **Step 8: Add the CI job** to `.github/workflows/ci.yaml`, named `gateway-linkage-script` ("check-gateway-linkage.sh tests"), `ubuntu-latest`, `timeout-minutes: 5`, checkout pinned as the neighbours are, then `run: sh scripts/check-gateway-linkage_test.sh`. One comment line: the check runs only at release time, so a regression would otherwise show only on a release. +- [ ] **Step 9: Commit** (`git commit -s`): `feat: Add the cortex-gateway release build and linkage check`. + +### Task T2: The `build-gateway` release job + +**Depends on:** T1. + +**Files:** +- Modify: `.github/workflows/release-binaries.yaml` (header comment, new job after `build-cpex`, `publish` job) + +**Interfaces:** +- Consumes: `scripts/build-gateway.sh` (T1). +- Produces: workflow artifacts `dist-gateway--`, each with `cortex-gateway_${TAG}_${os}_${arch}.tar.gz`. + +- [ ] **Step 1: Resolve the image digests.** For `docker.io/library/rust:1.96.0-alpine`, `docker.io/library/alpine:3.22` and `gcr.io/distroless/static-debian12`, get the multi-arch index digest: `podman manifest inspect ` must list both `amd64` and `arm64`, and the digest to pin is the one of that index (`skopeo inspect --raw docker:// | shasum -a 256`, or `crane digest `). Pin each as `@sha256:`. +- [ ] **Step 2: Add the `build-gateway` job.** Model it on `build-cpex`: `needs: version`, `contents: read`, matrix `{os: darwin, arch: arm64, runner: macos-15}`, `{os: linux, arch: amd64, runner: ubuntu-latest}`, `{os: linux, arch: arm64, runner: ubuntu-24.04-arm}`, `fail-fast: false` (decision 8), `timeout-minutes: 60`. Env `TAG`, `VERSION` from `needs.version.outputs`. Steps: + 1. Checkout (pinned, `persist-credentials: false`). + 2. darwin: `sh scripts/build-gateway.sh`. linux: `docker run --rm -v "${GITHUB_WORKSPACE}:/src" -w /src -e VERSION "${RUST_ALPINE}" sh scripts/build-gateway.sh` (`RUST_ALPINE` is a job env with the pinned image, as `ALPINE_GO` is). + 3. linux only, the clean-host runs (decision 13): `dist/cortex-gateway --version` on the runner, then the same in the pinned `alpine` image and in the pinned distroless image, each with `-v "${GITHUB_WORKSPACE}/dist:/d:ro"`. + 4. `tar -C dist -czf "dist/cortex-gateway_${TAG}_${os}_${arch}.tar.gz" cortex-gateway`, then `rm -f dist/cortex-gateway`. + 5. Upload with the pinned `actions/upload-artifact`, `name: dist-gateway-${{ matrix.os }}-${{ matrix.arch }}`, `retention-days: 1`, `if-no-files-found: error`. +- [ ] **Step 3: Change `publish`.** `needs: [version, build, build-cpex, build-gateway]` and the `if:` from decision 9. In "Check every tarball is present": after the required checks, loop over the three gateway targets and emit `::warning::No cortex-gateway tarball for /. This release ships without it.` for each missing one. Do not add them to `missing`. `checksums.txt` covers whatever is present, unchanged. +- [ ] **Step 4: Release notes.** In the publish step, when `ls dist/cortex-gateway_*.tar.gz` finds a file, add: ``'`cortex-gateway_*` is the praxis-mode gateway, for macOS arm64 and linux amd64/arm64'`` and ``'(static). Install it with `install.sh --praxis`.'`` (each line under 150 characters). +- [ ] **Step 5: Header comment.** Replace "Four jobs" with "Five jobs". Say that `build-gateway` is optional: a failed gateway leg does not block the release, and on `main-latest` the prune then removes the old gateway asset (decision 10). +- [ ] **Step 6: Verify.** Run the gate's `yamllint` line. Read the final `publish` `if:` and check Review Focus items 1 and 2 against it. +- [ ] **Step 7: Commit:** `feat: Build cortex-gateway in the release workflow`. + +### Task T3: Weekly build against the latest Praxis + +**Depends on:** T1. + +**Files:** +- Create: `.github/workflows/praxis-latest.yaml` + +**Interfaces:** +- Consumes: `scripts/build-gateway.sh` (T1). + +- [ ] **Step 1: Write the workflow.** `on: schedule: [{cron: "17 6 * * 1"}]` and `workflow_dispatch`. Top-level `permissions: contents: read`. Job `build` on `ubuntu-latest`, `timeout-minutes: 90`: + 1. Checkout (pinned, `persist-credentials: false`). + 2. Resolve the version (decision 14): `curl -fsSL -A "rossoctl-cortex-ci (github.com/rossoctl/cortex)" https://crates.io/api/v1/crates/praxis-proxy | jq -r .crate.max_stable_version`. Fail when the result does not match `^[0-9]+\.[0-9]+\.[0-9]+$`. Print the pinned and the newest version. + 3. Rewrite the pins: `sed -i -E 's/^(praxis-proxy(-filter)? = \{ version = )"=[0-9.]+"/\1"='"$v"'"/' praxis/server/Cargo.toml praxis/filters/Cargo.toml`, then `git diff --stat` for the log. + 4. In `praxis/`: `rustup toolchain install`, `cargo update -p praxis-proxy -p praxis-proxy-filter`, `cargo test --locked`. + 5. `docker run --rm -v "${GITHUB_WORKSPACE}:/src" -w /src -e VERSION=praxis-latest sh scripts/build-gateway.sh`. Use the same pinned image as T2 (copy the digest). +- [ ] **Step 2: Add the `report` job** (decision 15): `needs: build`, `if: failure()`, `permissions: issues: write`, env `GH_TOKEN: ${{ github.token }}`. Find the open issue with `gh issue list --repo "${GITHUB_REPOSITORY}" --state open --search 'in:title "Weekly build against the latest Praxis failed"' --json number --jq '.[0].number'`. When found, `gh issue comment` with the run URL (`${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}`). Otherwise `gh issue create` with that title and a body that has the run URL and one line: "The pinned Praxis release still builds. This job does not block anything." +- [ ] **Step 3: Header comment** (5 lines at most): what it builds, that it blocks nothing, where it reports. +- [ ] **Step 4: Verify.** Run the gate's `yamllint` line. Run the sed line of step 1.3 on a scratch copy of both manifests with `v=9.9.9` and check that `grep -n '"=9.9.9"'` finds exactly 3 lines (two in `server`, one in `filters`). +- [ ] **Step 5: Commit:** `ci: Build cortex-gateway weekly against the latest Praxis`. + +### Task T4: Documentation + +**Depends on:** T2, T3. + +**Files:** +- Modify: `.github/CLAUDE.md` (workflow table, "Release binaries" section) +- Modify: `praxis/README.md` (new section "Release build (#1723)" after "Run the tests") + +- [ ] **Step 1: `.github/CLAUDE.md`.** Add a table row for `praxis-latest.yaml` (weekly and manual, builds against the newest Praxis, opens or comments on one issue, blocks nothing). In "Release binaries", add one paragraph: `cortex-gateway___.tar.gz` from `build-gateway` (darwin/arm64 natively, linux static musl in `rust:alpine`), built by `scripts/build-gateway.sh` and checked by `scripts/check-gateway-linkage.sh`. It is optional: a failed leg does not block the release. Change "jobs are `version`, `build`, `build-cpex` ... and `publish`" to include `build-gateway`, and say that `publish` requires every tarball except the gateway ones. +- [ ] **Step 2: `praxis/README.md`.** The new section says, in a short list: how to run the release build locally (the two `build-gateway.sh` commands from the gate), why OpenSSL is vendored (upstream links the system OpenSSL, so a plain build is not portable), the duty "Cortex owns OpenSSL security updates for `cortex-gateway`: an OpenSSL CVE needs an `openssl-src` bump (Dependabot proposes it) and a release", and "no FIPS claim for this binary". +- [ ] **Step 3: Run** `pre-commit run --all-files`. Expected: pass. +- [ ] **Step 4: Commit:** `docs: Describe the cortex-gateway release build`. + +## Done when (from the issue), and who proves it + +- The three tarballs run on a clean host with no OpenSSL package: T1 step 6 and 7 locally, T2 step 2 in the release (`--version` in distroless and alpine, and the linkage check). +- A Go release succeeds when the gateway job fails: T2 step 3 (`publish`'s `if:` and the warning instead of the error). Proof after merge: a `workflow_dispatch` dry run. The PR body says that this proof is still to do. From b3a9fc0b068d03bb0554c6198ed7196e880b50cd Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 15:07:07 -0500 Subject: [PATCH 02/11] docs: Fix the plan dependencies and build details for #1723 T3 copies the rust:alpine digest that T2 resolves, so T3 now depends on T2. The linkage check calls its tools by bare name, so that the test stubs work. The Alpine build installs binutils for readelf. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .../2026-11-10-praxis-mode-7-release-gateway.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-11-10-praxis-mode-7-release-gateway.md b/docs/superpowers/plans/2026-11-10-praxis-mode-7-release-gateway.md index 67b2330ac..f35be5715 100644 --- a/docs/superpowers/plans/2026-11-10-praxis-mode-7-release-gateway.md +++ b/docs/superpowers/plans/2026-11-10-praxis-mode-7-release-gateway.md @@ -69,7 +69,7 @@ ## Gate -Run from the worktree root. It needs `pre-commit`, `shellcheck`, `yamllint`, a running `podman` machine, and Rust through rustup. It takes about 20 minutes on a cold cache. +Run from the worktree root. On the unchanged base, the `shellcheck` step fails with "does not exist" because T1 creates the three scripts; that is expected, and the gate passes only after T1. The `yamllint` step passes on the base (two line-length warnings, exit 0). It needs `pre-commit`, `shellcheck`, `yamllint`, a running `podman` machine, and Rust through rustup. It takes about 20 minutes on a cold cache. ```sh pre-commit run --all-files \ @@ -87,6 +87,8 @@ pre-commit run --all-files \ **Depends on:** none. +**Before you start:** read `.github/CLAUDE.md` (it covers CI workflows and release binaries). T2, T3 and T4 also change files below `.github/`. + **Files:** - Create: `scripts/build-gateway.sh` - Create: `scripts/check-gateway-linkage.sh` @@ -95,7 +97,7 @@ pre-commit run --all-files \ **Interfaces:** - Produces: `sh scripts/build-gateway.sh`. Run from the repo root. Env `VERSION` is required (die with a message when empty). Writes `dist/cortex-gateway` and exits non-zero when the build or the check fails. Optional env `MACOSX_DEPLOYMENT_TARGET` (default `12.0`). -- Produces: `sh scripts/check-gateway-linkage.sh `. Exit 0 when the binary is standalone and prints `cortex-gateway `. Exit 1 otherwise, with the reason on stderr. Uses `uname -s` to pick `otool -L` (Darwin) or `readelf -d` (Linux). Any other OS: exit 1. +- Produces: `sh scripts/check-gateway-linkage.sh `. Exit 0 when the binary is standalone and prints `cortex-gateway `. Exit 1 otherwise, with the reason on stderr. Uses `uname -s` to pick `otool -L` (Darwin) or `readelf -d` (Linux). Any other OS: exit 1. Call `uname`, `otool` and `readelf` by bare name (never an absolute path), so the test's `PATH` stubs replace them. - [ ] **Step 1: Write the failing test `scripts/check-gateway-linkage_test.sh`.** Follow the shape of `scripts/release_notes_test.sh` (POSIX sh, one function per case, a failure counter, exit 1 when any case fails). Each case makes a temp dir with stub `uname`, `otool` and `readelf` scripts first on `PATH`, and a stub binary that prints a given `--version` line. Cases and expected results: - `test_linux_static_passes`: `uname` prints `Linux`, `readelf -d` prints `There is no dynamic section in this file.` and exits 0, binary prints `cortex-gateway v1.2.3`. Expect exit 0. @@ -109,7 +111,7 @@ pre-commit run --all-files \ - [ ] **Step 4: Run** `sh scripts/check-gateway-linkage_test.sh` and `shellcheck --severity=error scripts/check-gateway-linkage.sh scripts/check-gateway-linkage_test.sh`. Expected: all cases pass, no shellcheck error. - [ ] **Step 5: Write `scripts/build-gateway.sh`** (POSIX sh, `set -eu`). Order: 1. Die when `VERSION` is empty. - 2. When `/etc/alpine-release` exists: `apk add --no-cache musl-dev perl make linux-headers`. (perl and make build `openssl-src`. make builds jemalloc.) + 2. When `/etc/alpine-release` exists: `apk add --no-cache musl-dev perl make linux-headers binutils`. (perl and make build `openssl-src`. make builds jemalloc. binutils gives `readelf` to the linkage check in step 8.) 3. `cd praxis && rustup toolchain install` (decision 3). 4. `triple=$(rustc -vV | sed -n 's/^host: //p')`. 5. Linux: `export RUSTFLAGS="-C target-feature=+crt-static"`. On `aarch64-*` also `export JEMALLOC_SYS_WITH_LG_PAGE=16` (decision 5). Darwin: `export MACOSX_DEPLOYMENT_TARGET="${MACOSX_DEPLOYMENT_TARGET:-12.0}"`. @@ -148,13 +150,13 @@ pre-commit run --all-files \ ### Task T3: Weekly build against the latest Praxis -**Depends on:** T1. +**Depends on:** T1, T2 (T2 step 1 resolves the pinned `rust:alpine` digest that step 1.5 copies). **Files:** - Create: `.github/workflows/praxis-latest.yaml` **Interfaces:** -- Consumes: `scripts/build-gateway.sh` (T1). +- Consumes: `scripts/build-gateway.sh` (T1), the `RUST_ALPINE` digest in `release-binaries.yaml` (T2). - [ ] **Step 1: Write the workflow.** `on: schedule: [{cron: "17 6 * * 1"}]` and `workflow_dispatch`. Top-level `permissions: contents: read`. Job `build` on `ubuntu-latest`, `timeout-minutes: 90`: 1. Checkout (pinned, `persist-credentials: false`). From 02bbf084ad306d097d6386c8014c7cbabd1a3b0a Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 15:16:54 -0500 Subject: [PATCH 03/11] feat: Add the cortex-gateway release build and linkage check scripts/build-gateway.sh builds the cortex-gateway release binary for the host it runs on: native on macOS, and static musl in rust:alpine on Linux. Then it runs scripts/check-gateway-linkage.sh. The check fails when the binary links a library that a clean host does not have, or when --version does not print the release version. Two changes from the plan: - The alpine packages include cmake, because libz-ng-sys needs it. - The script puts the pinned rustup toolchain first on PATH, so a Homebrew cargo does not build with an older rustc. It also skips the rustup self-update. A new ci.yaml job runs the check's tests on each PR. Part of #1723. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .github/workflows/ci.yaml | 12 +++ scripts/build-gateway.sh | 57 +++++++++++++ scripts/check-gateway-linkage.sh | 46 ++++++++++ scripts/check-gateway-linkage_test.sh | 116 ++++++++++++++++++++++++++ 4 files changed, 231 insertions(+) create mode 100755 scripts/build-gateway.sh create mode 100755 scripts/check-gateway-linkage.sh create mode 100644 scripts/check-gateway-linkage_test.sh diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index b650f05d4..2802d14ff 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -487,6 +487,18 @@ jobs: - name: Run release_notes.sh tests run: sh scripts/release_notes_test.sh + # check-gateway-linkage.sh runs only at release time, so without this job a + # regression in it would show only on a release. + gateway-linkage-script: + name: check-gateway-linkage.sh tests + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Run check-gateway-linkage.sh tests + run: sh scripts/check-gateway-linkage_test.sh + # `go mod tidy -diff` on every module in the repo. GOWORK=off so # replace directives resolve as they do in build.yaml and # release-binaries.yaml. diff --git a/scripts/build-gateway.sh b/scripts/build-gateway.sh new file mode 100755 index 000000000..7ccd054b7 --- /dev/null +++ b/scripts/build-gateway.sh @@ -0,0 +1,57 @@ +#!/bin/sh +# Build the cortex-gateway release binary for the host this runs on, then +# check that it is standalone (scripts/check-gateway-linkage.sh). +# +# macOS: a native build. Linux: run it in rust:alpine for a static musl binary. +# The release job, the weekly Praxis job and the local gate all run this file. +# +# Usage (from the repo root): VERSION=v1.2.3 sh scripts/build-gateway.sh +# Writes dist/cortex-gateway. Optional env: MACOSX_DEPLOYMENT_TARGET (12.0). +set -eu + +[ -n "${VERSION:-}" ] || { echo "build-gateway: VERSION is required" >&2; exit 1; } + +if [ -f /etc/alpine-release ]; then + # perl and make build openssl-src, make builds jemalloc, cmake builds + # libz-ng-sys, binutils gives readelf to the linkage check. + apk add --no-cache musl-dev perl make cmake linux-headers binutils +fi + +# rustup reads rust-toolchain.toml only from the current directory and its +# parents, not from --manifest-path. +cd praxis +# No self-update: the build needs only the pinned toolchain. +rustup toolchain install --no-self-update +# Put the pinned toolchain first, before any cargo or rustc not from rustup +# (for example Homebrew). +PATH="$(dirname "$(rustup which cargo)"):${PATH}" + +triple=$(rustc -vV | sed -n 's/^host: //p') +case $(uname -s) in +Linux) + export RUSTFLAGS="-C target-feature=+crt-static" + case ${triple} in + aarch64-*) + # jemalloc fixes the page size at build time. 64 KiB works on 4 KiB + # and 64 KiB page hosts, 4 KiB crashes on 64 KiB ones. + export JEMALLOC_SYS_WITH_LG_PAGE=16 + ;; + esac + ;; +Darwin) + # Go 1.26 binaries need macOS 12, so the gateway claims the same floor. + export MACOSX_DEPLOYMENT_TARGET="${MACOSX_DEPLOYMENT_TARGET:-12.0}" + ;; +esac + +# --target keeps RUSTFLAGS away from build scripts and proc-macros (which +# cannot be crt-static), and gives each host triple its own output dir. +# CORTEX_VERSION: cortex refuses a gateway whose version differs from its own. +# STRIP: the Go release strips too, and praxis/Cargo.toml stays unchanged. +CORTEX_VERSION="${VERSION}" CARGO_PROFILE_RELEASE_STRIP=symbols \ + cargo build --release --locked -p cortex-gateway --target "${triple}" + +mkdir -p ../dist +cp "target/${triple}/release/cortex-gateway" ../dist/cortex-gateway +cd .. +sh scripts/check-gateway-linkage.sh dist/cortex-gateway "${VERSION}" diff --git a/scripts/check-gateway-linkage.sh b/scripts/check-gateway-linkage.sh new file mode 100755 index 000000000..a229d05cc --- /dev/null +++ b/scripts/check-gateway-linkage.sh @@ -0,0 +1,46 @@ +#!/bin/sh +# Check that a cortex-gateway release binary is standalone and has the +# release version. +# +# Linux: the ELF must have no NEEDED entry (static musl). +# macOS: every library in `otool -L` must be below /usr/lib/ or /System/Library/. +# Both: ` --version` must print exactly `cortex-gateway `, +# because `cortex` refuses a gateway whose version differs from its own. +# +# Usage: sh scripts/check-gateway-linkage.sh +# Exit 0 when the binary passes. Exit 1 otherwise, with the reason on stderr. +set -eu + +die() { printf 'check-gateway-linkage: %s\n' "$*" >&2; exit 1; } + +[ $# -eq 2 ] || die "usage: $0 " +bin=$1 +version=$2 +[ -f "${bin}" ] || die "${bin}: no such file" + +case $(uname -s) in +Linux) + # A readelf failure must not read as "no NEEDED lines". + out=$(readelf -d "${bin}") || die "readelf -d ${bin} failed" + [ -n "${out}" ] || die "readelf -d ${bin} printed nothing" + needed=$(printf '%s\n' "${out}" | grep -F '(NEEDED)' || true) + [ -z "${needed}" ] || die "${bin} links shared libraries: +${needed}" + ;; +Darwin) + out=$(otool -L "${bin}") || die "otool -L ${bin} failed" + [ -n "${out}" ] || die "otool -L ${bin} printed nothing" + # The first line is the binary's own path. + bad=$(printf '%s\n' "${out}" | sed 1d | awk '{print $1}' | + grep -v -e '^/usr/lib/' -e '^/System/Library/' || true) + [ -z "${bad}" ] || die "${bin} links libraries outside /usr/lib/ and /System/Library/: +${bad}" + ;; +*) + die "unsupported OS: $(uname -s)" + ;; +esac + +got=$("${bin}" --version) || die "${bin} --version failed" +[ "${got}" = "cortex-gateway ${version}" ] || + die "${bin} --version printed '${got}', want 'cortex-gateway ${version}'" diff --git a/scripts/check-gateway-linkage_test.sh b/scripts/check-gateway-linkage_test.sh new file mode 100644 index 000000000..39381508f --- /dev/null +++ b/scripts/check-gateway-linkage_test.sh @@ -0,0 +1,116 @@ +#!/bin/sh +# Tests for check-gateway-linkage.sh. +# +# Black-box: each case puts stub `uname`, `otool` and `readelf` scripts first +# on PATH, and a stub binary that prints a given --version line. No build. +# +# Run: sh scripts/check-gateway-linkage_test.sh +set -eu + +# shellcheck disable=SC1007 # `CDPATH= cd` empties CDPATH for this one command. +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +CHECK_SH="${SCRIPT_DIR}/check-gateway-linkage.sh" +[ -f "${CHECK_SH}" ] || { printf 'cannot find %s\n' "${CHECK_SH}" >&2; exit 1; } + +TMP=$(mktemp -d) +trap 'rm -rf "${TMP}"' EXIT + +PASS=0 +FAIL=0 +N=0 + +ok() { PASS=$((PASS + 1)); printf ' ok %s\n' "$1"; } +bad() { FAIL=$((FAIL + 1)); printf ' FAIL %s: %s\n' "$1" "$2"; } + +# stub : a stub command that prints and exits. +stub() { + printf '%s\n' "$3" >"${CASE}/$1.out" + printf '#!/bin/sh\ncat "%s"\nexit %s\n' "${CASE}/$1.out" "$2" >"${CASE}/bin/$1" + chmod +x "${CASE}/bin/$1" +} + +# setup : a fresh case dir with a uname stub and a binary. +setup() { + N=$((N + 1)) + CASE="${TMP}/case${N}" + mkdir -p "${CASE}/bin" + stub uname 0 "$1" + printf '#!/bin/sh\necho "%s"\n' "$2" >"${CASE}/gateway" + chmod +x "${CASE}/gateway" +} + +# run_check : run the check. Exit status in $STATUS, stderr in $ERR. +run_check() { + STATUS=0 + PATH="${CASE}/bin:${PATH}" sh "${CHECK_SH}" "${CASE}/gateway" "$1" \ + >/dev/null 2>"${CASE}/err" || STATUS=$? + ERR=$(cat "${CASE}/err") +} + +expect() { # label want-status [stderr-substring] + if [ "${STATUS}" != "$2" ]; then + bad "$1" "exit ${STATUS}, want $2 (stderr: ${ERR})" + elif [ -n "${3:-}" ] && ! printf '%s' "${ERR}" | grep -qF -- "$3"; then + bad "$1" "stderr lacks '$3' (stderr: ${ERR})" + else + ok "$1" + fi +} + +TAB=$(printf '\t') + +test_linux_static_passes() { + setup Linux "cortex-gateway v1.2.3" + stub readelf 0 "There is no dynamic section in this file." + run_check v1.2.3 + expect test_linux_static_passes 0 +} + +test_linux_needed_fails() { + setup Linux "cortex-gateway v1.2.3" + stub readelf 0 " 0x0000000000000001 (NEEDED) Shared library: [libssl.so.3]" + run_check v1.2.3 + expect test_linux_needed_fails 1 libssl.so.3 +} + +test_readelf_error_fails() { + setup Linux "cortex-gateway v1.2.3" + stub readelf 1 "" + run_check v1.2.3 + expect test_readelf_error_fails 1 +} + +test_darwin_system_libs_pass() { + setup Darwin "cortex-gateway v1.2.3" + stub otool 0 "${CASE}/gateway: +${TAB}/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0) +${TAB}/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 61439.0.0)" + run_check v1.2.3 + expect test_darwin_system_libs_pass 0 +} + +test_darwin_homebrew_ssl_fails() { + setup Darwin "cortex-gateway v1.2.3" + stub otool 0 "${CASE}/gateway: +${TAB}/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0) +${TAB}/opt/homebrew/opt/openssl@3/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)" + run_check v1.2.3 + expect test_darwin_homebrew_ssl_fails 1 libssl.3.dylib +} + +test_version_mismatch_fails() { + setup Linux "cortex-gateway dev" + stub readelf 0 "There is no dynamic section in this file." + run_check v1.2.3 + expect test_version_mismatch_fails 1 "cortex-gateway dev" +} + +test_linux_static_passes +test_linux_needed_fails +test_readelf_error_fails +test_darwin_system_libs_pass +test_darwin_homebrew_ssl_fails +test_version_mismatch_fails + +printf '%s passed, %s failed\n' "${PASS}" "${FAIL}" +[ "${FAIL}" = 0 ] From 4ebe2a16140fb6d86138e697392f9f8cd5fc2d1e Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 15:17:31 -0500 Subject: [PATCH 04/11] fix: Remove the old gateway binary before the build copies a new one macOS kills a signed binary that cp overwrote in place, because the kernel keeps the code signature of the old file. A second local build then failed the --version check with SIGKILL. Part of #1723. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- scripts/build-gateway.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/build-gateway.sh b/scripts/build-gateway.sh index 7ccd054b7..c886b0667 100755 --- a/scripts/build-gateway.sh +++ b/scripts/build-gateway.sh @@ -52,6 +52,8 @@ CORTEX_VERSION="${VERSION}" CARGO_PROFILE_RELEASE_STRIP=symbols \ cargo build --release --locked -p cortex-gateway --target "${triple}" mkdir -p ../dist +# rm first: macOS kills a binary that cp overwrote in place (stale signature). +rm -f ../dist/cortex-gateway cp "target/${triple}/release/cortex-gateway" ../dist/cortex-gateway cd .. sh scripts/check-gateway-linkage.sh dist/cortex-gateway "${VERSION}" From 053bc2a708538f65b68b4abefbf2e7940da6a33d Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 15:21:21 -0500 Subject: [PATCH 05/11] feat: Build cortex-gateway in the release workflow Add a build-gateway matrix job for darwin/arm64, linux/amd64 and linux/arm64. Each leg runs scripts/build-gateway.sh, runs the binary on clean hosts (Linux), and uploads one tarball. The gateway is optional. Publish runs when build and build-cpex succeed, whatever build-gateway did. A missing gateway tarball gives a warning. The release notes name the gateway only when a gateway tarball is present. Part of #1723. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .github/workflows/release-binaries.yaml | 101 ++++++++++++++++++++++-- 1 file changed, 95 insertions(+), 6 deletions(-) diff --git a/.github/workflows/release-binaries.yaml b/.github/workflows/release-binaries.yaml index 8d7485421..64e16a141 100644 --- a/.github/workflows/release-binaries.yaml +++ b/.github/workflows/release-binaries.yaml @@ -12,11 +12,18 @@ name: Release binaries # platform builds on its own runner. Linux builds are static musl binaries, so # one tarball runs on glibc and musl hosts alike. # -# Four jobs: `version` derives the tag and the stamp once, `build` and -# `build-cpex` hold only contents: read, and `publish` alone can write. It -# publishes nothing unless every expected tarball arrived, so a failed cpex -# build blocks the release rather than shipping without the variant (and the -# main-latest prune below never deletes the variant's assets). +# `cortex-gateway` is the praxis-mode gateway, a Rust binary. scripts/build-gateway.sh +# builds it on each platform's runner (static musl in rust:alpine on Linux) and +# checks that it needs no library outside the OS. +# +# Five jobs: `version` derives the tag and the stamp once, `build`, +# `build-cpex` and `build-gateway` hold only contents: read, and `publish` alone +# can write. It publishes nothing unless every `build` and `build-cpex` tarball +# arrived, so a failed cpex build blocks the release rather than shipping without +# the variant (and the main-latest prune below never deletes the variant's assets). +# `build-gateway` is optional: a failed gateway leg does not block the release, and +# on main-latest the prune then removes the old gateway asset for that platform. +# That is correct: checksums.txt would not list it, so nothing could verify it. on: push: @@ -297,8 +304,80 @@ jobs: if-no-files-found: error retention-days: 1 + build-gateway: + name: build-gateway (${{ matrix.os }}/${{ matrix.arch }}) + needs: version + runs-on: ${{ matrix.runner }} + permissions: + contents: read + timeout-minutes: 60 + strategy: + # Each platform is optional on its own: install.sh --praxis checks per os/arch + # in checksums.txt, so two of three tarballs is a valid release. + fail-fast: false + matrix: + include: + - {os: darwin, arch: arm64, runner: macos-15} + - {os: linux, arch: amd64, runner: ubuntu-latest} + - {os: linux, arch: arm64, runner: ubuntu-24.04-arm} + env: + TAG: ${{ needs.version.outputs.tag }} + VERSION: ${{ needs.version.outputs.version }} + TARGET_OS: ${{ matrix.os }} + TARGET_ARCH: ${{ matrix.arch }} + # Run with `docker run`, not as a job container, for the reason ALPINE_GO gives. + RUST_ALPINE: docker.io/library/rust:1.96.0-alpine@sha256:f87aa870663e2b57ec8c69de82c7eedf7383bee987eef7612c0359635eaadb41 + # Clean hosts for the built binary: musl with no build tools, and no libc and + # no OpenSSL at all. Plain alpine ships libssl3, so it is not the second one. + ALPINE: docker.io/library/alpine:3.22@sha256:49340b346798875b5592e5d420241f2f9cfba5f47e62ebb1446a1b34734e898f + DISTROLESS: gcr.io/distroless/static-debian12@sha256:d75cdd72874d4790092fcb1b058493ecf6bb5bf2b2b897045b00ff01d91843f2 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Build cortex-gateway + run: | + set -euo pipefail + # The runner user makes dist/, so it can delete the binary root writes there. + mkdir -p dist + if [ "${TARGET_OS}" = linux ]; then + docker run --rm -v "${GITHUB_WORKSPACE}:/src" -w /src -e VERSION \ + "${RUST_ALPINE}" sh scripts/build-gateway.sh + else + sh scripts/build-gateway.sh + fi + + - name: Run cortex-gateway on clean hosts + if: matrix.os == 'linux' + run: | + set -euo pipefail + # glibc host, musl host, and a host with no libc and no OpenSSL. + dist/cortex-gateway --version + docker run --rm -v "${GITHUB_WORKSPACE}/dist:/d:ro" "${ALPINE}" /d/cortex-gateway --version + docker run --rm -v "${GITHUB_WORKSPACE}/dist:/d:ro" "${DISTROLESS}" /d/cortex-gateway --version + + - name: Package cortex-gateway + run: | + set -euo pipefail + tar -C dist -czf "dist/cortex-gateway_${TAG}_${TARGET_OS}_${TARGET_ARCH}.tar.gz" cortex-gateway + rm -f dist/cortex-gateway + ls -l dist + + - name: Upload the tarball for the publish job + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: dist-gateway-${{ matrix.os }}-${{ matrix.arch }} + path: dist/*.tar.gz + if-no-files-found: error + retention-days: 1 + publish: - needs: [version, build, build-cpex] + needs: [version, build, build-cpex, build-gateway] + # Run whatever build-gateway did, but only when build and build-cpex succeeded. + # !cancelled() is needed: without a status function, a failed need skips this job. + if: ${{ !cancelled() && needs.build.result == 'success' && needs.build-cpex.result == 'success' }} runs-on: ubuntu-latest permissions: # create/update the Release, upload assets, and move the rolling tag. @@ -353,6 +432,11 @@ jobs: echo "::error::Missing release tarballs:${missing}" exit 1 fi + # cortex-gateway is optional: warn, do not add it to missing. + for target in darwin/arm64 linux/amd64 linux/arm64; do + f="dist/cortex-gateway_${TAG}_${target%/*}_${target#*/}.tar.gz" + [ -s "${f}" ] || echo "::warning::No cortex-gateway tarball for ${target}. This release ships without it." + done ( cd dist && sha256sum ./*.tar.gz > checksums.txt ) ls -l dist cat dist/checksums.txt @@ -412,6 +496,11 @@ jobs: add 'Prebuilt `agentop` and `cortex` binaries for linux and macOS (amd64/arm64).' add '`cortex-cpex_*` is `cortex` built with the cpex policy plugin, for macOS arm64 and' add 'linux amd64/arm64 (static). Install it with `install.sh --cpex`.' + # Name the gateway only when this release has at least one gateway tarball. + if ls dist/cortex-gateway_*.tar.gz >/dev/null 2>&1; then + add '`cortex-gateway_*` is the praxis-mode gateway, for macOS arm64 and linux amd64/arm64' + add '(static). Install it with `install.sh --praxis`.' + fi add 'Every tarball carries build provenance:' add '`gh attestation verify --repo rossoctl/cortex`.' add '' From 0f133b03e3308d2af242f71662a7a88f5f513efc Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 15:28:09 -0500 Subject: [PATCH 06/11] ci: Build cortex-gateway weekly against the latest Praxis Add praxis-latest.yaml. Each Monday, and on demand, it moves the praxis-proxy pins to the newest crates.io version, runs the tests and the static musl release build. It blocks nothing. On failure, a report job comments on one open issue, or opens it. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .github/workflows/praxis-latest.yaml | 96 ++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 .github/workflows/praxis-latest.yaml diff --git a/.github/workflows/praxis-latest.yaml b/.github/workflows/praxis-latest.yaml new file mode 100644 index 000000000..d281e4e02 --- /dev/null +++ b/.github/workflows/praxis-latest.yaml @@ -0,0 +1,96 @@ +# Praxis latest - build cortex-gateway weekly against the newest Praxis release. +# +# Moves the praxis-proxy pins to the newest crates.io version, then runs the tests and +# the static release build. It blocks nothing. On failure, the report job comments on +# the open "Weekly build against the latest Praxis failed" issue, or opens it. +# +name: Praxis latest + +on: + schedule: + - cron: "17 6 * * 1" + workflow_dispatch: + +permissions: + contents: read + +jobs: + build: + name: Build against the latest Praxis + runs-on: ubuntu-latest + timeout-minutes: 90 + env: + # The same pinned image as build-gateway in release-binaries.yaml. + RUST_ALPINE: docker.io/library/rust:1.96.0-alpine@sha256:f87aa870663e2b57ec8c69de82c7eedf7383bee987eef7612c0359635eaadb41 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + # crates.io refuses a request with no User-Agent. + - name: Resolve the newest Praxis version + id: praxis + run: | + set -euo pipefail + v=$(curl -fsSL -A "rossoctl-cortex-ci (github.com/rossoctl/cortex)" \ + https://crates.io/api/v1/crates/praxis-proxy | jq -r .crate.max_stable_version) + if ! printf '%s\n' "$v" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then + echo "::error::crates.io gave no usable praxis-proxy version: '$v'" + exit 1 + fi + pinned=$(sed -n -E 's/^praxis-proxy = \{ version = "=([0-9.]+)".*/\1/p' praxis/server/Cargo.toml) + echo "Pinned Praxis: ${pinned}. Newest Praxis: ${v}." + echo "version=${v}" >> "$GITHUB_OUTPUT" + + - name: Move the Praxis pins + env: + PRAXIS_VERSION: ${{ steps.praxis.outputs.version }} + run: | + set -euo pipefail + v="${PRAXIS_VERSION}" + sed -i -E 's/^(praxis-proxy(-filter)? = \{ version = )"=[0-9.]+"/\1"='"$v"'"/' \ + praxis/server/Cargo.toml praxis/filters/Cargo.toml + git diff --stat + + # Installs the toolchain that praxis/rust-toolchain.toml names. + - name: Test + working-directory: praxis + run: | + set -euo pipefail + rustup toolchain install + cargo update -p praxis-proxy -p praxis-proxy-filter + cargo test --locked + + - name: Static release build + run: | + set -euo pipefail + docker run --rm -v "${GITHUB_WORKSPACE}:/src" -w /src -e VERSION=praxis-latest \ + "${RUST_ALPINE}" sh scripts/build-gateway.sh + + report: + name: Report a failure + needs: build + if: failure() + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + issues: write + env: + GH_TOKEN: ${{ github.token }} + TITLE: Weekly build against the latest Praxis failed + steps: + - name: Comment on or open the issue + run: | + set -euo pipefail + run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" + n=$(gh issue list --repo "${GITHUB_REPOSITORY}" --state open \ + --search "in:title \"${TITLE}\"" --json number --jq '.[0].number') + if [ -n "$n" ]; then + gh issue comment "$n" --repo "${GITHUB_REPOSITORY}" --body "Failed again: ${run_url}" + else + gh issue create --repo "${GITHUB_REPOSITORY}" --title "${TITLE}" \ + --body "Failed run: ${run_url} + + The pinned Praxis release still builds. This job does not block anything." + fi From aee5334aeb5e0b5d3cffb34c60f406f181b0b561 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 15:37:33 -0500 Subject: [PATCH 07/11] docs: Describe the cortex-gateway release build Add praxis-latest.yaml to the workflow table. Describe the build-gateway job and its optional tarballs in "Release binaries". Add a "Release build" section to praxis/README.md: the local build, the vendored OpenSSL, the OpenSSL update duty and no FIPS claim. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .github/CLAUDE.md | 14 +++++++++++--- praxis/README.md | 18 ++++++++++++++++++ 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/.github/CLAUDE.md b/.github/CLAUDE.md index 8c0a76384..b5429de51 100644 --- a/.github/CLAUDE.md +++ b/.github/CLAUDE.md @@ -13,6 +13,7 @@ including the PR title format and the Go checks that CI gates on, are in the roo | `praxis.yaml` | PR to main/release-*, push to main, on `praxis/**` or the Go parser paths it ports | `cargo fmt --check`, clippy and tests for `praxis/`. A failure blocks only when `praxis/` or the workflow changed | | `security-scans.yaml` | PR to main | Dependency review, shellcheck, YAML lint, Hadolint, Bandit, Trivy, CodeQL | | `scorecard.yaml` | Weekly / push to main | OpenSSF Scorecard security health metrics | +| `praxis-latest.yaml` | Weekly (Monday), manual | Builds and tests `cortex-gateway` against the newest Praxis on crates.io. On failure it opens one issue, or comments on the open one. It blocks nothing | | `release-binaries.yaml` | Tag push (`v*`), manual; push to main publishes the main-latest channel when `MAIN_CHANNEL_ENABLED` is set | Release tarballs — see [Release binaries](#release-binaries) | | `pr-verifier.yml` | PR | PR title check | | `spellcheck_action.yml` | PR | Spellcheck on markdown files — **disabled** (`if: false`) | @@ -63,7 +64,14 @@ musl, in `golang:alpine` via `docker run`). There is no darwin/amd64 FFI. The executable inside is still `cortex`, and `cortex --version` prints `cortex (cpex) `. `install.sh --cpex` installs it. -The workflow's jobs are `version`, `build`, `build-cpex` (both `contents: read`) and -`publish`. Only `publish` writes. It refuses to publish unless every expected tarball -arrived, writes one `checksums.txt` over all of them, and attests every tarball's +The praxis-mode gateway ships as `cortex-gateway___.tar.gz`, from the +`build-gateway` matrix job: darwin/arm64 natively, and linux/amd64 and linux/arm64 as +static musl binaries in `rust:alpine`. `scripts/build-gateway.sh` builds it and +`scripts/check-gateway-linkage.sh` checks that it is standalone. It is optional: a +failed `build-gateway` leg does not block the release. + +The workflow's jobs are `version`, `build`, `build-cpex`, `build-gateway` (all +`contents: read`) and `publish`. Only `publish` writes. It refuses to publish unless +every expected tarball arrived, except the gateway tarballs: a missing one gives a +warning. It writes one `checksums.txt` over all of them, and attests every tarball's build provenance. diff --git a/praxis/README.md b/praxis/README.md index 8c5e85794..33a916df5 100644 --- a/praxis/README.md +++ b/praxis/README.md @@ -24,6 +24,24 @@ cargo test --locked `rust-toolchain.toml` pins the toolchain (Rust 1.96.0, with `rustfmt` and `clippy`). `rustup toolchain install`, run in this directory, installs it, as CI does. +## Release build (#1723) + +- To build the release binary locally, from the repo root (the second command + gives the static Linux binary): + + ```sh + VERSION=dev sh scripts/build-gateway.sh + podman run --rm -v "$PWD:/src" -w /src -e VERSION=dev \ + docker.io/library/rust:1.96.0-alpine sh scripts/build-gateway.sh + ``` + + Each writes `dist/cortex-gateway` and runs `scripts/check-gateway-linkage.sh`. +- OpenSSL is vendored (`openssl-src`). Upstream Praxis links the system + OpenSSL, so a plain build does not run on a host without that library. +- Cortex owns OpenSSL security updates for `cortex-gateway`: an OpenSSL CVE + needs an `openssl-src` bump (Dependabot proposes it) and a release. +- There is no FIPS claim for this binary. + ## Run the gateway ```sh From 4d1a7f9fb55fe82d0e89a80db7594a2219d26a07 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 15:57:18 -0500 Subject: [PATCH 08/11] fix: Keep the smoke test when a gateway leg fails A failed build-gateway leg made the run fail. release-smoke-linux.yaml runs only after a successful run, so the release shipped with no smoke test. The job now has continue-on-error. publish still warns for each missing gateway tarball. The first weekly Praxis issue said that the pinned release still builds. Nothing checks this, so the body now points to the run log. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .github/workflows/praxis-latest.yaml | 2 +- .github/workflows/release-binaries.yaml | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/praxis-latest.yaml b/.github/workflows/praxis-latest.yaml index d281e4e02..cd0635c12 100644 --- a/.github/workflows/praxis-latest.yaml +++ b/.github/workflows/praxis-latest.yaml @@ -92,5 +92,5 @@ jobs: gh issue create --repo "${GITHUB_REPOSITORY}" --title "${TITLE}" \ --body "Failed run: ${run_url} - The pinned Praxis release still builds. This job does not block anything." + This job does not block anything. See the run log for the cause." fi diff --git a/.github/workflows/release-binaries.yaml b/.github/workflows/release-binaries.yaml index 64e16a141..8aba396f6 100644 --- a/.github/workflows/release-binaries.yaml +++ b/.github/workflows/release-binaries.yaml @@ -311,6 +311,9 @@ jobs: permissions: contents: read timeout-minutes: 60 + # A failed leg must not fail the run: release-smoke-linux.yaml runs only after a + # successful run, so the release would ship with no smoke test. + continue-on-error: true strategy: # Each platform is optional on its own: install.sh --praxis checks per os/arch # in checksums.txt, so two of three tarballs is a valid release. From d79ab5e2dd1c87c3aeeb3b13c2d19123f46f0f13 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 16:05:13 -0500 Subject: [PATCH 09/11] fix: Fail the weekly Praxis job when its pins do not move The job now stops with an error when a Praxis pin line in praxis/server/Cargo.toml or praxis/filters/Cargo.toml is not at the newest version after the sed step. It also updates the dependencies of praxis-proxy and praxis-proxy-filter with cargo update --recursive. Fixes review: sed could match nothing, and the job then tested the old pin and passed Fixes review: cargo update -p kept shared dependencies locked, so a new Praxis that needs newer ones failed resolution Files: - .github/workflows/praxis-latest.yaml Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .github/workflows/praxis-latest.yaml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/workflows/praxis-latest.yaml b/.github/workflows/praxis-latest.yaml index cd0635c12..15c482056 100644 --- a/.github/workflows/praxis-latest.yaml +++ b/.github/workflows/praxis-latest.yaml @@ -52,6 +52,14 @@ jobs: sed -i -E 's/^(praxis-proxy(-filter)? = \{ version = )"=[0-9.]+"/\1"='"$v"'"/' \ praxis/server/Cargo.toml praxis/filters/Cargo.toml git diff --stat + # If sed matched nothing, the job would test the old pin and pass. + for f in praxis/server/Cargo.toml praxis/filters/Cargo.toml; do + pins=$(grep -E '^praxis-proxy(-filter)? *=' "$f" || true) + if [ -z "$pins" ] || printf '%s\n' "$pins" | grep -vqF "\"=${v}\""; then + echo "::error::${f}: the Praxis pins did not move to =${v}" + exit 1 + fi + done # Installs the toolchain that praxis/rust-toolchain.toml names. - name: Test @@ -59,7 +67,8 @@ jobs: run: | set -euo pipefail rustup toolchain install - cargo update -p praxis-proxy -p praxis-proxy-filter + # --recursive: a new Praxis can need newer shared dependencies too. + cargo update --recursive -p praxis-proxy -p praxis-proxy-filter cargo test --locked - name: Static release build From aa7c5788aed3ba1d1535b408dbe4a9780ce6dfd3 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 16:14:43 -0500 Subject: [PATCH 10/11] fix: Do not list gateway platforms in the release notes The notes said that cortex-gateway is for macOS arm64 and linux amd64/arm64. A failed build-gateway leg does not block the release, so a release can have fewer platforms than the notes said. Now the notes do not list the platforms. The release assets show which platforms the release has. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .github/workflows/release-binaries.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release-binaries.yaml b/.github/workflows/release-binaries.yaml index 8aba396f6..dce770fdf 100644 --- a/.github/workflows/release-binaries.yaml +++ b/.github/workflows/release-binaries.yaml @@ -500,9 +500,10 @@ jobs: add '`cortex-cpex_*` is `cortex` built with the cpex policy plugin, for macOS arm64 and' add 'linux amd64/arm64 (static). Install it with `install.sh --cpex`.' # Name the gateway only when this release has at least one gateway tarball. + # No platform list: a failed leg leaves a platform out, and the assets show which. if ls dist/cortex-gateway_*.tar.gz >/dev/null 2>&1; then - add '`cortex-gateway_*` is the praxis-mode gateway, for macOS arm64 and linux amd64/arm64' - add '(static). Install it with `install.sh --praxis`.' + add '`cortex-gateway_*` is the praxis-mode gateway (static on linux).' + add 'Install it with `install.sh --praxis`.' fi add 'Every tarball carries build provenance:' add '`gh attestation verify --repo rossoctl/cortex`.' From cf14f588dd758295067ee681907d4605b37f38b7 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Tue, 10 Nov 2026 16:38:04 -0500 Subject: [PATCH 11/11] fix: Let Dependabot bump openssl-src for praxis praxis/README.md says that Dependabot proposes the openssl-src bump for an OpenSSL CVE. openssl-src is a transitive dependency, through openssl with the vendored feature. Dependabot version updates skip transitive dependencies by default, and security updates are off for this repo. So no bump came. The /praxis cargo entry now has an allow list. It keeps the direct dependencies and adds openssl-src as an indirect dependency. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- .github/dependabot.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index ba61d4afd..1306592d1 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -94,6 +94,13 @@ updates: directory: /praxis schedule: interval: weekly + # openssl-src is transitive (openssl, vendored), so the default skips it. Its bump + # is how cortex-gateway gets OpenSSL CVE fixes (praxis/README.md). With an allow + # list, only listed dependencies update, so direct must stay. + allow: + - dependency-type: direct + - dependency-name: openssl-src + dependency-type: indirect # Python - root (tests) - package-ecosystem: pip