From abf624083e1873bb8aa310796a5f596d42934945 Mon Sep 17 00:00:00 2001 From: swapnil <78632212+swapnilpaliwal-sd@users.noreply.github.com> Date: Sat, 12 Sep 2026 00:32:41 -0700 Subject: [PATCH 1/7] ci: gate main on the three regression suites, and make a skipped suite a failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit main had no gate. The suites only ran when somebody remembered to run them, and running them is not cheap enough to be reliable as a habit. Three required tiers: build and typecheck; the parser-free repo invariants (no-ignored-fixtures, the staging guard for all three front ends, shell syntax); and the java / python / typescript suites in parallel. The part that needed care is that these suites are written for a laptop, where a missing dependency is a good reason to step aside: they print SKIP and exit 77. On CI that is the one outcome that must never be tolerated, because a gate that opens when its input is missing is worse than no gate — the green gets read as evidence. Two instances that would otherwise have gone unnoticed: - the parser is an external repository, and without it all three suites exit 77; - test/java/torture reads class files with java.lang.classfile, which is not final before JDK 24, so on the runner's default JDK the ten-family oracle exits 77 while the suite still reports success. So .github/scripts/run-suite.sh converts every shape of "did not actually run" into a failure: exit 77, any sub-harness that swallowed its own 77, and a suite that reports "passed 0" and therefore asserted nothing. CI installs JDK 24 and pinned Python 3.10 rather than relaxing that check. The parser is pinned by SHA in .github/parser-ref. The goldens are a function of both these rules and the IR, so tracking the parser's main branch would let a change in the other repository turn this one red with no commit here to point at, and would let a parser regression quietly become the new expectation. A nightly workflow runs against the parser's main and opens an issue when the pin falls behind, so the pin cannot rot unnoticed. Soufflé is pinned to 2.5 for the same reason: its evaluation changes between minor versions, and an unpinned solver makes every golden a moving target. Branch protection is not applied here because it cannot be: rulesets are refused with 403 on a private repository in a Free organisation. It is written as .github/scripts/protect-main.sh, to run on the day this repository goes public, when rulesets become free. Until then main-guard.yml records any commit that reaches main without a pull request — detection rather than prevention, and the file says so and says to delete it once the ruleset exists. Also adds the contribution path for people who are not maintainers: issue templates, a pull request template, CONTRIBUTING.md, CODEOWNERS, and build badges on the README. Fixes #417 Co-Authored-By: Claude Opus 5 (1M context) --- .github/CODEOWNERS | 16 ++ .github/ISSUE_TEMPLATE/config.yml | 1 + .github/ISSUE_TEMPLATE/engine-defect.yml | 52 +++++ .github/ISSUE_TEMPLATE/enhancement.yml | 25 +++ .github/ISSUE_TEMPLATE/harness.yml | 26 +++ .github/ISSUE_TEMPLATE/parser-blocked.yml | 42 ++++ .github/dependabot.yml | 23 +++ .github/parser-ref | 16 ++ .github/pull_request_template.md | 40 ++++ .github/scripts/protect-main.sh | 115 +++++++++++ .github/scripts/run-suite.sh | 76 ++++++++ .github/workflows/ci.yml | 223 ++++++++++++++++++++++ .github/workflows/main-guard.yml | 92 +++++++++ .github/workflows/parser-drift.yml | 115 +++++++++++ CONTRIBUTING.md | 99 ++++++++++ README.md | 6 + 16 files changed, 967 insertions(+) create mode 100644 .github/CODEOWNERS create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/engine-defect.yml create mode 100644 .github/ISSUE_TEMPLATE/enhancement.yml create mode 100644 .github/ISSUE_TEMPLATE/harness.yml create mode 100644 .github/ISSUE_TEMPLATE/parser-blocked.yml create mode 100644 .github/dependabot.yml create mode 100644 .github/parser-ref create mode 100644 .github/pull_request_template.md create mode 100755 .github/scripts/protect-main.sh create mode 100755 .github/scripts/run-suite.sh create mode 100644 .github/workflows/ci.yml create mode 100644 .github/workflows/main-guard.yml create mode 100644 .github/workflows/parser-drift.yml create mode 100644 CONTRIBUTING.md diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 000000000..e7212d9c6 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,16 @@ +# Every pull request needs a review from an owner listed here. +# +# Until the project opens up, that is one person; the file exists now so the +# ruleset can require owner review from the day it is turned on, and so adding a +# maintainer later is a one-line change rather than a policy discussion. + +* @swapnilpaliwal-sd + +# The engine rules and the goldens that pin their behaviour. A change to either +# without a change to the other is the thing review is for. +/src/ @swapnilpaliwal-sd +/test/ @swapnilpaliwal-sd + +# CI, protection, and the parser pin decide what "passing" means. They get the +# same scrutiny as the engine. +/.github/ @swapnilpaliwal-sd diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 000000000..3ba13e0ce --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1 @@ +blank_issues_enabled: false diff --git a/.github/ISSUE_TEMPLATE/engine-defect.yml b/.github/ISSUE_TEMPLATE/engine-defect.yml new file mode 100644 index 000000000..f09f7b89f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/engine-defect.yml @@ -0,0 +1,52 @@ +name: Engine defect +description: A call edge the engine resolves wrongly, or does not resolve at all. +labels: [bug, engine] +body: + - type: markdown + attributes: + value: | + **Describe the defect with a minimal synthetic example.** Do not paste code from + a real project, name the project, or quote its identifiers or file names — this + tracker is a durable public record, and naming someone's codebase reads as a + benchmark claim about it. Reduce the finding to `class Widget`, `pkg/_helpers.py` + and the like, and describe the mechanism. + + - type: dropdown + id: language + attributes: + label: Front end + options: [java, python, typescript] + validations: { required: true } + + - type: textarea + id: repro + attributes: + label: Minimal example + description: The smallest synthetic source that shows it. Include the call site. + render: text + validations: { required: true } + + - type: textarea + id: expected + attributes: + label: What the engine should resolve, and what it resolves instead + description: | + Name the edge both ways — `A.foo -> B.bar`, versus the `ambiguous_unknown` or + wrong target you actually get. + validations: { required: true } + + - type: textarea + id: mechanism + attributes: + label: Mechanism + description: | + What does the rule join on, and why does the join fail? If you already know which + relation derives zero rows, say which. + validations: { required: false } + + - type: input + id: parser + attributes: + label: Parser commit + description: The SHA in `.github/parser-ref` you reproduced against. + validations: { required: false } diff --git a/.github/ISSUE_TEMPLATE/enhancement.yml b/.github/ISSUE_TEMPLATE/enhancement.yml new file mode 100644 index 000000000..f437ade66 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/enhancement.yml @@ -0,0 +1,25 @@ +name: Enhancement +description: Resolution power the engine does not have yet. +labels: [enhancement] +body: + - type: dropdown + id: language + attributes: + label: Front end + options: [java, python, typescript, all] + validations: { required: true } + + - type: textarea + id: construct + attributes: + label: The construct + description: A synthetic example of the code shape that does not resolve today. + render: text + validations: { required: true } + + - type: textarea + id: why + attributes: + label: Why it matters + description: What becomes answerable once this resolves. Mechanism, not corpus. + validations: { required: true } diff --git a/.github/ISSUE_TEMPLATE/harness.yml b/.github/ISSUE_TEMPLATE/harness.yml new file mode 100644 index 000000000..4fd1424c3 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/harness.yml @@ -0,0 +1,26 @@ +name: Test harness or CI +description: A suite, golden, oracle or workflow that measures the wrong thing. +labels: [test] +body: + - type: markdown + attributes: + value: | + A measurement fault outranks an engine fault: while it stands, every number the + suite reports is unsafe to act on. Say what the harness claims and why that claim + is not true. + + - type: textarea + id: claim + attributes: + label: What the harness reports, and why it is wrong + validations: { required: true } + + - type: textarea + id: isolate + attributes: + label: How you isolated it + description: | + The rows you read, the A/B you ran, the commit on both arms. A regression that + was really the harness looks identical to one that was really the engine until + this part exists. + validations: { required: true } diff --git a/.github/ISSUE_TEMPLATE/parser-blocked.yml b/.github/ISSUE_TEMPLATE/parser-blocked.yml new file mode 100644 index 000000000..a2cf118e7 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/parser-blocked.yml @@ -0,0 +1,42 @@ +name: Parser-blocked +description: The engine cannot resolve something because the IR does not carry it. +labels: [parser-blocked] +body: + - type: markdown + attributes: + value: | + Use this when the fix does not belong in this repository. The engine's rules are + read-only with respect to the parser: a missing fact is filed here, proven, and + fixed in `AxiomCodeAI/parser`. + + **A parser defect needs three things before it is one:** the parser source that + drops the fact, a synthetic input that reproduces it, and the real construct it + came from. Without all three this is a hypothesis. + + - type: dropdown + id: language + attributes: + label: Front end + options: [java, python, typescript] + validations: { required: true } + + - type: textarea + id: missing + attributes: + label: The fact that is missing + description: Which relation, which column, and what it should contain. + validations: { required: true } + + - type: textarea + id: evidence + attributes: + label: Evidence in the parser source + description: The file and function that drops it. Synthetic example only. + validations: { required: true } + + - type: textarea + id: blocked + attributes: + label: What this blocks here + description: The rule that cannot be written, or the edge that cannot resolve. + validations: { required: true } diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 000000000..27b47936c --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,23 @@ +version: 2 +updates: + # Actions are pinned by major tag; this keeps them current and, more to the + # point, surfaces the deprecations that otherwise fail CI without warning. + - package-ecosystem: github-actions + directory: / + schedule: + interval: weekly + labels: [build, dependencies] + open-pull-requests-limit: 3 + + - package-ecosystem: npm + directory: / + schedule: + interval: weekly + labels: [build, dependencies] + open-pull-requests-limit: 3 + # The engine's behaviour is pinned to the toolchain that produced the + # goldens. A TypeScript major arriving on its own schedule is a change to + # what this repo measures, not a routine bump. + ignore: + - dependency-name: typescript + update-types: [version-update:semver-major] diff --git a/.github/parser-ref b/.github/parser-ref new file mode 100644 index 000000000..4ee45fbfc --- /dev/null +++ b/.github/parser-ref @@ -0,0 +1,16 @@ +# The commit of AxiomCodeAI/parser that CI builds and scores this repo's goldens +# against. +# +# PINNED ON PURPOSE. The goldens in test/*/expected are a function of two things: +# these rules, and the IR the parser produces. Tracking the parser's main branch +# would let a change in the other repository turn this one red with no commit here +# to point at, and — worse — would let a parser regression quietly re-bless itself +# as the new expectation. +# +# To move the pin: change the SHA, run the three suites locally against that exact +# parser build, and open a pull request. A golden that moves belongs in that diff, +# where it can be reviewed as a change in resolution power rather than noticed later. +# +# The nightly parser-drift workflow runs against the parser's main branch and opens +# an issue when this pin has fallen behind, so it cannot rot unnoticed. +b25fe8aeaf6ba81cf968e2a8d5a90fb7887739d4 diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 000000000..ba5e3b9f5 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,40 @@ + + +Fixes # + +## What changed + + + +## Why the goldens moved, or why they did not + + + +## Evidence + + + +## Checklist + +- [ ] The three suites pass locally against the parser commit in `.github/parser-ref` +- [ ] Any golden that moved is explained above +- [ ] A fix validated on more than one shape, so this is not overfitting to one case +- [ ] Labels set, including the front end (`java` / `python` / `typescript`) diff --git a/.github/scripts/protect-main.sh b/.github/scripts/protect-main.sh new file mode 100755 index 000000000..c291c9966 --- /dev/null +++ b/.github/scripts/protect-main.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# Apply the branch ruleset that makes `main` unpushable. +# +# WHY THIS IS A SCRIPT AND NOT ALREADY APPLIED +# Rulesets and protected branches are refused with HTTP 403 on a PRIVATE +# repository owned by a FREE organisation. AxiomCodeAI is on Free, so the rule +# below cannot exist yet. It becomes available the moment either is true: +# +# * this repository is made public — rulesets are free on public repos, or +# * the organisation moves to Team — ~$4 per seat per month. +# +# Run this script on that day. It is idempotent: it updates the existing ruleset +# if one is already there, and creates it otherwise. +# +# bash .github/scripts/protect-main.sh # apply +# bash .github/scripts/protect-main.sh --dry-run # print the payload only +# +# WHAT IT ENFORCES +# - no direct push to main, and no force-push, by anyone including admins +# - main cannot be deleted +# - every change arrives by pull request, with 1 approving review +# - a review is dismissed when new commits are pushed +# - CODEOWNERS review required +# - the `CI` check must pass, against the merge commit, not a stale run +# - linear history — squash or rebase, no merge bubbles +# +# `bypass_actors` is deliberately EMPTY. A ruleset an admin can walk around is a +# convention, not a control, and the admin is the only account here that can push +# to main in the first place. +# ───────────────────────────────────────────────────────────────────────────── +set -euo pipefail + +REPO="${REPO:-AxiomCodeAI/axiom-code-graph}" +DRY=0 +[ "${1:-}" = "--dry-run" ] && DRY=1 + +payload="$(cat <<'JSON' +{ + "name": "protect-main", + "target": "branch", + "enforcement": "active", + "conditions": { + "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] } + }, + "bypass_actors": [], + "rules": [ + { "type": "deletion" }, + { "type": "non_fast_forward" }, + { "type": "required_linear_history" }, + { + "type": "pull_request", + "parameters": { + "required_approving_review_count": 1, + "dismiss_stale_reviews_on_push": true, + "require_code_owner_review": true, + "require_last_push_approval": false, + "required_review_thread_resolution": true, + "allowed_merge_methods": ["squash", "rebase"] + } + }, + { + "type": "required_status_checks", + "parameters": { + "strict_required_status_checks_policy": true, + "do_not_enforce_on_create": false, + "required_status_checks": [ + { "context": "CI" } + ] + } + } + ] +} +JSON +)" + +if [ "$DRY" = "1" ]; then + echo "$payload" + exit 0 +fi + +# Refuse early with a readable reason rather than a raw 403. +if ! gh api "repos/$REPO/rulesets" >/dev/null 2>&1; then + cat >&2 <<'MSG' +Rulesets are not available on this repository yet. + + A private repository in a Free organisation cannot have protected branches. + Make the repository public, or move AxiomCodeAI to the Team plan, then run + this script again. Nothing else about it needs to change. +MSG + exit 1 +fi + +existing="$(gh api "repos/$REPO/rulesets" --jq '.[] | select(.name=="protect-main") | .id' || true)" + +if [ -n "$existing" ]; then + echo "updating ruleset $existing on $REPO" + printf '%s' "$payload" | gh api -X PUT "repos/$REPO/rulesets/$existing" --input - >/dev/null +else + echo "creating ruleset on $REPO" + printf '%s' "$payload" | gh api -X POST "repos/$REPO/rulesets" --input - >/dev/null +fi + +# Merge-method hygiene lives on the repository, not the ruleset: squash-only, and +# delete the branch once it has landed so the branch list stops accumulating the +# stale aliases this repo has collected before. +gh api -X PATCH "repos/$REPO" \ + -F allow_squash_merge=true \ + -F allow_merge_commit=false \ + -F allow_rebase_merge=false \ + -F delete_branch_on_merge=true \ + -F allow_auto_merge=true >/dev/null + +echo "done. main now requires a pull request and a green CI check." +gh api "repos/$REPO/rulesets" --jq '.[] | " ruleset: \(.name) enforcement=\(.enforcement)"' diff --git a/.github/scripts/run-suite.sh b/.github/scripts/run-suite.sh new file mode 100755 index 000000000..4c2abc552 --- /dev/null +++ b/.github/scripts/run-suite.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# Run one engine regression suite under CI semantics. +# +# The suites are written for a developer laptop, where "the parser isn't built +# yet" is a good reason to step aside: they print SKIP and exit 77. On CI that is +# the one outcome that must never be tolerated. A gate that opens when its input +# is missing is worse than no gate at all — the pull request goes green having +# tested nothing, and the next person reads that green as evidence. +# +# So this wrapper turns every shape of "did not actually run" into a failure: +# +# 1. exit 77 from the suite itself; +# 2. a `SKIP: parser not found` line from a sub-harness (torture, fixtures) +# that caught its own 77 and carried on; +# 3. a suite that reported zero passing cases, which means the case loop found +# nothing to do and the exit status is meaningless. +# +# Usage: run-suite.sh [extra args passed to the suite] +# ───────────────────────────────────────────────────────────────────────────── +set -uo pipefail + +lang="${1:?usage: run-suite.sh [args...]}"; shift +root="$(cd "$(dirname "$0")/../.." && pwd)" +suite="$root/test/$lang/run-tests.sh" + +[ -f "$suite" ] || { echo "::error::no suite at $suite"; exit 1; } + +# The suites default AXIOM_PARSER to $ROOT/../Parser/dist/index.js, which is only +# correct for a side-by-side developer layout. CI must be told explicitly, and must +# refuse to start rather than discover the absence halfway through as a SKIP. +if [ -z "${AXIOM_PARSER:-}" ]; then + echo "::error::AXIOM_PARSER is unset — refusing to run a suite that would skip itself" + exit 1 +fi +if [ ! -f "$AXIOM_PARSER" ]; then + echo "::error::AXIOM_PARSER=$AXIOM_PARSER does not exist. The parser checkout did not build." + exit 1 +fi + +log="$(mktemp)" +echo "── $lang suite ── parser: $AXIOM_PARSER" +bash "$suite" "$@" 2>&1 | tee "$log" +rc="${PIPESTATUS[0]}" + +if [ "$rc" -eq 77 ]; then + echo "::error::the $lang suite skipped itself (exit 77). A skipped suite is a failed gate." + exit 1 +fi + +# A sub-harness that swallowed its own 77. The suite's exit status cannot see this: +# it counted the sub-harness as neither a pass nor a failure, so the run is green +# with a whole family unexecuted. +# +# EVERY skip is a failure here, not just the parser ones. The sub-harnesses decline +# for several different reasons — no parser, no javac, a JDK without +# java.lang.classfile (the torture oracle needs 24+), no oracle checkout — and each +# one is a dependency CI is supposed to provide. If a skip is ever legitimate it +# should be an explicit exclusion in this file, visible in a diff, rather than a +# green run that quietly tested less than the last one. +if grep -qE '(^|[[:space:]])SKIP([: (]|PED)' "$log"; then + echo "::error::a sub-harness in the $lang suite skipped instead of running:" + grep -nE '(^|[[:space:]])SKIP([: (]|PED)' "$log" | sed 's/^/ /' + echo "::error::CI must supply what it wanted. Do not relax this check to go green." + exit 1 +fi + +# "passed 0" means the case loop matched nothing — a rename or a bad filter, not a +# clean run. Guard it, because exit 0 with zero assertions is the quietest failure +# this harness can produce. +if grep -qE '^passed 0([^0-9]|$)|^passed 0,' "$log"; then + echo "::error::the $lang suite passed 0 cases — it asserted nothing." + exit 1 +fi + +exit "$rc" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 000000000..73ac85ea3 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,223 @@ +# ───────────────────────────────────────────────────────────────────────────── +# CI — the gate every change to main passes through. +# +# Three tiers, cheapest first, all required: +# build the TypeScript driver compiles and typechecks +# hygiene repo invariants that need no parser and catch silent breakage +# engine the java / python / typescript regression suites, in parallel +# +# NO PATH FILTERS, deliberately. A required check that is skipped by a path +# filter never reports, and a pull request waiting on a check that will never +# report can never merge. Correctness beats the saved minutes. +# ───────────────────────────────────────────────────────────────────────────── +name: CI + +on: + push: + branches: [main] + pull_request: + merge_group: + workflow_dispatch: + +# One run per ref. A new push to a pull request cancels the previous run, but a +# run on main is always allowed to finish — main's history is the record. +concurrency: + group: ci-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +permissions: + contents: read + +env: + NODE_VERSION: '20' + SOUFFLE_VERSION: '2.5' + +jobs: + build: + name: build & typecheck + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: npm + # `npm ci` runs the `prepare` script, which builds. Keeping the explicit + # build step anyway means a prepare-script change cannot silently stop + # compiling this repo. + - run: npm ci + - run: npm run typecheck + - run: npm run build + + hygiene: + name: repo invariants + runs-on: ubuntu-24.04 + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + + # A fixture input that .gitignore matches passes on the machine that wrote + # it and fails on every clone. The suites run this too; running it here as + # well means the answer arrives in seconds rather than after the engine. + - name: every fixture input is tracked by git + run: bash test/tools/no-ignored-fixtures.sh + + # A relation staged for the client but not for libraries is EMPTY on every + # run and nothing errors — no golden can see it. This is the only check + # that can. + - name: IR staging maps are consistent + run: | + fail=0 + for lang in java python typescript; do + echo "── $lang" + python3 test/tools/check_staging.py --lang "$lang" || fail=1 + done + exit $fail + + - name: shell scripts parse + run: | + fail=0 + while IFS= read -r f; do + bash -n "$f" || { echo "::error file=$f::does not parse"; fail=1; } + done < <(git ls-files '*.sh') + exit $fail + + engine: + name: engine (${{ matrix.lang }}) + runs-on: ubuntu-24.04 + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + lang: [java, python, typescript] + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + + # The python suite's tier-1 attribution preflight reads CPython opcodes, + # which are not stable across minor versions. 3.10 is the pinned one. + - uses: actions/setup-python@v5 + with: + python-version: '3.10' + + # JDK 24, not the runner's default. The java torture harness reads class files + # with java.lang.classfile, which is not final before 24 — on an older JDK it + # exits 77 and the whole ten-family oracle silently does not run. + - uses: actions/setup-java@v4 + if: matrix.lang == 'java' + with: + distribution: temurin + java-version: '24' + + # ── Soufflé ──────────────────────────────────────────────────────────── + # Installed from the upstream release .deb for this exact runner image. + # Pinned: Soufflé's evaluation changes between minor versions, so an + # unpinned solver makes every golden in this repo a moving target. + - name: install Soufflé ${{ env.SOUFFLE_VERSION }} + run: | + set -euo pipefail + deb="x86_64-ubuntu-2404-souffle-${SOUFFLE_VERSION}-Linux.deb" + url="https://github.com/souffle-lang/souffle/releases/download/${SOUFFLE_VERSION}/${deb}" + curl -fsSL --retry 3 -o "/tmp/${deb}" "$url" + sudo apt-get update -qq + sudo apt-get install -y --no-install-recommends "/tmp/${deb}" + souffle --version | head -2 + + # ── the parser ───────────────────────────────────────────────────────── + # An external, private repository. Pinned by SHA in .github/parser-ref so + # that a parser change cannot turn this repo red without a reviewable + # commit here — see the nightly parser-drift workflow, which is what + # notices the pin has gone stale. + - name: read the pinned parser ref + id: parser + run: | + set -euo pipefail + ref="$(grep -vE '^\s*(#|$)' .github/parser-ref | head -1 | tr -d '[:space:]')" + [ -n "$ref" ] || { echo "::error::.github/parser-ref is empty"; exit 1; } + echo "ref=$ref" >> "$GITHUB_OUTPUT" + echo "parser pinned at $ref" + + - name: fail early if the parser key is missing + env: + KEY: ${{ secrets.PARSER_DEPLOY_KEY }} + run: | + if [ -z "${KEY:-}" ]; then + echo "::error::secret PARSER_DEPLOY_KEY is not set, so the parser cannot be" \ + "fetched and every suite would skip itself. Fork pull requests do not" \ + "receive secrets by design — such a pull request must be run from a" \ + "branch in this repository." + exit 1 + fi + + - uses: actions/checkout@v4 + with: + repository: AxiomCodeAI/parser + ref: ${{ steps.parser.outputs.ref }} + ssh-key: ${{ secrets.PARSER_DEPLOY_KEY }} + path: .parser + + # tree-sitter builds native addons, which is the slow part. Keyed on the + # pinned parser SHA, so a bump rebuilds and nothing else does. + - name: cache the built parser + id: parser-cache + uses: actions/cache@v4 + with: + path: | + .parser/node_modules + .parser/dist + key: parser-${{ runner.os }}-node${{ env.NODE_VERSION }}-${{ steps.parser.outputs.ref }} + + # ALWAYS build when the cache missed. A stale dist/ scores stale IR and the + # goldens move for a reason that is not in any diff. + - name: build the parser + if: steps.parser-cache.outputs.cache-hit != 'true' + working-directory: .parser + run: | + npm ci + npm run build + + - name: the parser actually built + run: | + test -f .parser/dist/index.js \ + || { echo "::error::.parser/dist/index.js is missing after build"; exit 1; } + echo "parser $(cd .parser && git rev-parse --short HEAD) built" + + # ── the compiled engine ──────────────────────────────────────────────── + # run-souffle.sh caches the compiled solver under a content hash of the + # .dl program text. Mirroring that key here skips a multi-minute C++ build + # on every run whose rules did not change. + - name: cache the compiled Soufflé engine + uses: actions/cache@v4 + with: + path: .souffle-cache + key: souffle-${{ env.SOUFFLE_VERSION }}-${{ matrix.lang }}-${{ hashFiles('src/**/*.dl') }} + restore-keys: | + souffle-${{ env.SOUFFLE_VERSION }}-${{ matrix.lang }}- + + - name: ${{ matrix.lang }} regression suite + env: + AXIOM_PARSER: ${{ github.workspace }}/.parser/dist/index.js + run: bash .github/scripts/run-suite.sh ${{ matrix.lang }} + + # A single job the branch ruleset can require. Without it, every new matrix + # entry has to be added to the protection rules by hand, and a matrix job that + # fails to start reports nothing at all — which a ruleset reads as "not + # failing" rather than as "did not run". + ci: + name: CI + runs-on: ubuntu-24.04 + needs: [build, hygiene, engine] + if: always() + steps: + - name: every required job succeeded + run: | + results='${{ join(needs.*.result, " ") }}' + echo "upstream results: $results" + for r in $results; do + [ "$r" = "success" ] || { echo "::error::a required job reported '$r'"; exit 1; } + done + echo "all required jobs passed" diff --git a/.github/workflows/main-guard.yml b/.github/workflows/main-guard.yml new file mode 100644 index 000000000..463af830c --- /dev/null +++ b/.github/workflows/main-guard.yml @@ -0,0 +1,92 @@ +# ───────────────────────────────────────────────────────────────────────────── +# Direct-push guard for `main`. +# +# This is a DETECTIVE control, not a preventive one, and it exists only because +# a preventive one is currently unavailable: protected branches are refused on a +# private repository in a Free organisation. Actions cannot reject a push — by +# the time a workflow runs, the commit is already on the branch. +# +# So this notices instead: it fails, and opens an issue, when a commit arrives on +# main that no pull request carried. When .github/scripts/protect-main.sh can +# finally run, this workflow becomes redundant and should be deleted — leaving it +# in place afterwards is how a repository ends up with a guard nobody reads +# standing next to a rule that already made it impossible. +# ───────────────────────────────────────────────────────────────────────────── +name: main-guard + +on: + push: + branches: [main] + +permissions: + contents: read + issues: write + +jobs: + direct-push: + name: every commit on main came from a pull request + runs-on: ubuntu-24.04 + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: check the pushed commits + id: check + env: + GH_TOKEN: ${{ github.token }} + BEFORE: ${{ github.event.before }} + AFTER: ${{ github.event.after }} + run: | + set -uo pipefail + + # A branch created or force-updated from nothing has no usable range. + if [ "$BEFORE" = "0000000000000000000000000000000000000000" ]; then + echo "branch created — nothing to compare"; exit 0 + fi + + orphans=() + while IFS= read -r sha; do + [ -n "$sha" ] || continue + n="$(gh api "repos/${GITHUB_REPOSITORY}/commits/${sha}/pulls" --jq 'length' 2>/dev/null || echo 0)" + if [ "$n" = "0" ]; then + orphans+=("$sha $(git log -1 --format=%s "$sha")") + fi + done < <(git rev-list "${BEFORE}..${AFTER}" 2>/dev/null) + + if [ ${#orphans[@]} -eq 0 ]; then + echo "all pushed commits arrived through a pull request" + exit 0 + fi + + { + echo "### Direct push to \`main\` detected" + echo + echo "These commits are on \`main\` without a pull request:" + echo + printf ' - `%s`\n' "${orphans[@]}" + } >> "$GITHUB_STEP_SUMMARY" + + printf '%s\n' "${orphans[@]}" > /tmp/orphans.txt + echo "found=1" >> "$GITHUB_OUTPUT" + echo "::error::${#orphans[@]} commit(s) reached main without a pull request" + exit 1 + + - name: record it as an issue + if: failure() && steps.check.outputs.found == '1' + env: + GH_TOKEN: ${{ github.token }} + run: | + set -uo pipefail + title="Direct push to main on $(date -u +%Y-%m-%d)" + # One issue per day, not one per push. + existing="$(gh issue list --state open --search "\"$title\" in:title" --json number --jq '.[0].number' || true)" + body="$(printf 'Commits reached \`main\` without a pull request:\n\n```\n%s\n```\n\nRun: %s/%s/actions/runs/%s\n\nThis is caught after the fact because branch protection is unavailable on a private repository in a Free organisation. `.github/scripts/protect-main.sh` makes it impossible instead.\n' \ + "$(cat /tmp/orphans.txt)" "$GITHUB_SERVER_URL" "$GITHUB_REPOSITORY" "$GITHUB_RUN_ID")" + if [ -n "${existing:-}" ]; then + gh issue comment "$existing" --body "$body" + else + gh issue create --title "$title" --body "$body" --label platform || \ + gh issue create --title "$title" --body "$body" + fi diff --git a/.github/workflows/parser-drift.yml b/.github/workflows/parser-drift.yml new file mode 100644 index 000000000..7a3885d5e --- /dev/null +++ b/.github/workflows/parser-drift.yml @@ -0,0 +1,115 @@ +# ───────────────────────────────────────────────────────────────────────────── +# Nightly: does this repo still pass against the parser's CURRENT main? +# +# CI pins the parser by SHA so that a change in the other repository cannot turn +# this one red with no commit here to point at. That pin is correct and it also +# rots: months later nobody knows whether the engine still works against the +# parser everyone else is using, and finding out at bump time means debugging a +# hundred commits of drift at once. +# +# So this runs the same suites against the parser's main branch every night. It +# does not gate anything — it opens an issue when the answer changes. +# ───────────────────────────────────────────────────────────────────────────── +name: parser-drift + +on: + schedule: + - cron: '17 7 * * *' # 07:17 UTC daily, off the hour to dodge the rush + workflow_dispatch: + +permissions: + contents: read + issues: write + +env: + NODE_VERSION: '20' + SOUFFLE_VERSION: '2.5' + +jobs: + against-parser-main: + name: suites vs parser@main + runs-on: ubuntu-24.04 + timeout-minutes: 60 + outputs: + pinned: ${{ steps.refs.outputs.pinned }} + head: ${{ steps.refs.outputs.head }} + behind: ${{ steps.refs.outputs.behind }} + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: '20' } + - uses: actions/setup-python@v5 + with: { python-version: '3.10' } + + - name: install Soufflé + run: | + set -euo pipefail + deb="x86_64-ubuntu-2404-souffle-${SOUFFLE_VERSION}-Linux.deb" + curl -fsSL --retry 3 -o "/tmp/${deb}" \ + "https://github.com/souffle-lang/souffle/releases/download/${SOUFFLE_VERSION}/${deb}" + sudo apt-get update -qq + sudo apt-get install -y --no-install-recommends "/tmp/${deb}" + + - uses: actions/checkout@v4 + with: + repository: AxiomCodeAI/parser + ref: main + ssh-key: ${{ secrets.PARSER_DEPLOY_KEY }} + path: .parser + + - name: compare the pin against parser main + id: refs + run: | + set -euo pipefail + pinned="$(grep -vE '^\s*(#|$)' .github/parser-ref | head -1 | tr -d '[:space:]')" + head="$(cd .parser && git rev-parse HEAD)" + behind="$(cd .parser && git rev-list --count "${pinned}..HEAD" 2>/dev/null || echo unknown)" + echo "pinned=$pinned" >> "$GITHUB_OUTPUT" + echo "head=$head" >> "$GITHUB_OUTPUT" + echo "behind=$behind" >> "$GITHUB_OUTPUT" + echo "pin is $behind commit(s) behind parser main ($head)" + + - name: build the parser + working-directory: .parser + run: npm ci && npm run build + + - name: java suite + env: { AXIOM_PARSER: '${{ github.workspace }}/.parser/dist/index.js' } + run: bash .github/scripts/run-suite.sh java + + - name: python suite + env: { AXIOM_PARSER: '${{ github.workspace }}/.parser/dist/index.js' } + run: bash .github/scripts/run-suite.sh python + + - name: typescript suite + env: { AXIOM_PARSER: '${{ github.workspace }}/.parser/dist/index.js' } + run: bash .github/scripts/run-suite.sh typescript + + report: + name: report drift + needs: against-parser-main + if: always() && needs.against-parser-main.result == 'failure' + runs-on: ubuntu-24.04 + permissions: + contents: read + issues: write + steps: + - name: open or update the drift issue + env: + GH_TOKEN: ${{ github.token }} + PINNED: ${{ needs.against-parser-main.outputs.pinned }} + HEAD: ${{ needs.against-parser-main.outputs.head }} + BEHIND: ${{ needs.against-parser-main.outputs.behind }} + run: | + set -uo pipefail + title="parser drift: the suites do not pass against parser main" + body="$(printf 'The nightly run against \`AxiomCodeAI/parser@main\` failed.\n\n- pinned in \`.github/parser-ref\`: \`%s\`\n- parser main: \`%s\`\n- the pin is **%s** commit(s) behind\n\nRun: %s/%s/actions/runs/%s\n\nThis does not gate anything — CI still builds the pinned parser. It means the next pin bump will carry a behaviour change, and the diff is cheapest to read now rather than after another month of drift.\n' \ + "$PINNED" "$HEAD" "$BEHIND" "$GITHUB_SERVER_URL" "$GITHUB_REPOSITORY" "$GITHUB_RUN_ID")" + n="$(gh issue list --state open --search "\"$title\" in:title" --json number --jq '.[0].number' || true)" + if [ -n "${n:-}" ]; then + echo "updating #$n" + gh issue comment "$n" --body "$body" + else + gh issue create --title "$title" --body "$body" --label build,test || \ + gh issue create --title "$title" --body "$body" + fi diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..6f0c8ee52 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,99 @@ +# Contributing + +## The short version + +`main` is not writable. Every change arrives as a pull request from a branch, is +reviewed, and merges only when CI is green. + +## Before you write code: open an issue + +Not only for large changes — for every change. The issue is where the measurement +that found the problem lives; the pull request is only the fix. This repository runs +several people and agents against different front ends at once, and issue-first is +what keeps that legible. + +Pick the template that matches: engine defect, parser-blocked, harness, enhancement. + +## Describing a defect + +**Use a synthetic example. Never name the project you found it in.** No identifiers, +no file names, no pasted code from a real codebase — in issues, pull requests, or +commit messages. Reduce it to `class Widget`, `pkg/_helpers.py`, `options: "Options"` +and describe the mechanism: what the rule joins on, why the join fails, what the fix +is. + +The tracker is a durable record. Naming someone's codebase turns a bug report into a +benchmark claim about their code, which is not what this is. + +Keep the corpus measurement out of the prose too. Describe the defect by mechanism, +not by what it cost on some project. + +## Branches + +Short-lived, cut from `origin/main`, named for the front end they touch: + + java/... python/... typescript/... + test/... parser-blocked/... + +Squash merge, and the branch is deleted on merge. There is no long-lived `dev` or +`staging` branch: this is a library, nothing deploys from it, and release stages are +git tags. A forward-merge tax buys nothing here. + +**Work in a `git worktree`, not the shared checkout.** More than one agent works this +tree at a time and it switches branches underneath you. + + git fetch origin + git worktree add ../wt/my-change -b python/my-change origin/main + +Cut from `origin/main`, never from the local `main` ref — it is not fast-forwarded by +the squash merges and drifts arbitrarily far behind. A baseline cut from a stale +`main` has produced a "collapse" that was entirely the baseline. + +## Running the tests + +The suites need two things that are not in this repository: Soufflé, and a built +parser. + + brew install souffle # or: apt-get install souffle + + git clone git@github.com:AxiomCodeAI/parser.git ../Parser + cd ../Parser && git checkout "$(grep -v '^#' ../axiom-code-graph/.github/parser-ref | head -1)" + npm ci && npm run build + +Then, from this repository: + + export AXIOM_PARSER="$PWD/../Parser/dist/index.js" + bash test/java/run-tests.sh + bash test/python/run-tests.sh + bash test/typescript/run-tests.sh + +**A suite that prints `SKIP` has not passed.** It exits 77 when it cannot find the +parser. CI treats that as a failure, and so should you — a green run that tested +nothing is the most expensive kind. + +### The parser is pinned + +`.github/parser-ref` names the exact parser commit CI builds and scores the goldens +against. The goldens are a function of both the rules and the IR, so tracking the +parser's `main` would let a change in the other repository turn this one red with no +commit here to point at. + +To move the pin: change the SHA, run all three suites against that exact build, and +put both in one pull request. A nightly workflow runs against the parser's `main` and +opens an issue when the pin has fallen behind, so it cannot rot unnoticed. + +### Re-blessing a golden + +`--bless` regenerates goldens from the current engine. It is not a way past a red +check. A golden that moves is a change in resolution power and belongs in the pull +request diff with an explanation of which edges changed and why each is correct now. + +The CPython ground truth cannot be re-blessed from here at all — it lives outside the +repository, on purpose. + +## Never modify parser code from here + +The engine is read-only with respect to the parser. A missing fact is filed as +`parser-blocked`, with three things: the parser source that drops it, a synthetic +input that reproduces it, and the real construct it came from. Without all three it +is a hypothesis, not a defect. diff --git a/README.md b/README.md index efb5babeb..a84fe4c51 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,11 @@ # AxiomCode code graph +[![CI](https://github.com/AxiomCodeAI/axiom-code-graph/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/AxiomCodeAI/axiom-code-graph/actions/workflows/ci.yml) +[![parser pin](https://github.com/AxiomCodeAI/axiom-code-graph/actions/workflows/parser-drift.yml/badge.svg?branch=main)](https://github.com/AxiomCodeAI/axiom-code-graph/actions/workflows/parser-drift.yml) +[![Soufflé 2.5](https://img.shields.io/badge/Souffl%C3%A9-2.5-blue)](https://souffle-lang.github.io/) +[![Node 18+](https://img.shields.io/badge/Node-18%2B-brightgreen)](https://nodejs.org/) + + **A knowledge graph of what code actually does, derived formally rather than guessed.** `axiom-code-graph` builds a *type-directed call graph*: for every call site in a codebase it resolves From e0995b9f860e6c7e6d2d1fbacc0991a0f7d9e871 Mon Sep 17 00:00:00 2001 From: swapnil <78632212+swapnilpaliwal-sd@users.noreply.github.com> Date: Sat, 12 Sep 2026 00:49:16 -0700 Subject: [PATCH 2/7] ci: score against ground truth, verify the solver, and drop CONTRIBUTING MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-ups on the review of the previous commit. The workflow was invalid and no job had ever started: a GitHub expression may only delimit strings with single quotes, and join(needs.*.result, " ") used double ones, which invalidates the whole file. actionlint catches this class of fault in a second and would have caught it before the push. Ground truth, not only goldens. A golden says "the same as last time", which a wrong answer satisfies perfectly well as long as it was wrong last time too, so java and typescript now run with --oracle: scored against javac and javap, and against the TypeScript compiler. That needs `npm ci` in the engine job — the compiler comes from this repo's own devDependencies, and without it every case reports "oracle refused" while the run still looks like it did something. No third-party library is downloaded to build ground truth. A case whose oracle would need an external classpath reports itself unscored instead. Soufflé is the solver the engine is compiled and linked against, so it is pinned and its package verified against the checksum upstream published for that release: what CI links is decided in the workflow file rather than by whatever the archive serves that day. The package is cached, so a run does not depend on that download succeeding twice. The parser is fetched with a token rather than a deploy key. CODEOWNERS now lists everyone on the project. CONTRIBUTING.md is removed. Co-Authored-By: Claude Opus 5 (1M context) --- .github/CODEOWNERS | 17 +---- .github/scripts/protect-main.sh | 42 ++++--------- .github/workflows/ci.yml | 89 ++++++++++++++++++++------- .github/workflows/main-guard.yml | 23 +++---- .github/workflows/parser-drift.yml | 41 ++++++++++--- CONTRIBUTING.md | 99 ------------------------------ 6 files changed, 124 insertions(+), 187 deletions(-) delete mode 100644 CONTRIBUTING.md diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e7212d9c6..484fec503 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,16 +1,3 @@ -# Every pull request needs a review from an owner listed here. -# -# Until the project opens up, that is one person; the file exists now so the -# ruleset can require owner review from the day it is turned on, and so adding a -# maintainer later is a one-line change rather than a policy discussion. +# Reviewers requested automatically on every pull request. -* @swapnilpaliwal-sd - -# The engine rules and the goldens that pin their behaviour. A change to either -# without a change to the other is the thing review is for. -/src/ @swapnilpaliwal-sd -/test/ @swapnilpaliwal-sd - -# CI, protection, and the parser pin decide what "passing" means. They get the -# same scrutiny as the engine. -/.github/ @swapnilpaliwal-sd +* @swapnilpaliwal-sd @JaredHLZhang @suyashpaliwal26 @Whua689 diff --git a/.github/scripts/protect-main.sh b/.github/scripts/protect-main.sh index c291c9966..058aedb81 100755 --- a/.github/scripts/protect-main.sh +++ b/.github/scripts/protect-main.sh @@ -1,33 +1,23 @@ #!/usr/bin/env bash # ───────────────────────────────────────────────────────────────────────────── -# Apply the branch ruleset that makes `main` unpushable. -# -# WHY THIS IS A SCRIPT AND NOT ALREADY APPLIED -# Rulesets and protected branches are refused with HTTP 403 on a PRIVATE -# repository owned by a FREE organisation. AxiomCodeAI is on Free, so the rule -# below cannot exist yet. It becomes available the moment either is true: -# -# * this repository is made public — rulesets are free on public repos, or -# * the organisation moves to Team — ~$4 per seat per month. -# -# Run this script on that day. It is idempotent: it updates the existing ruleset -# if one is already there, and creates it otherwise. +# Apply the branch ruleset for `main`. # # bash .github/scripts/protect-main.sh # apply # bash .github/scripts/protect-main.sh --dry-run # print the payload only # -# WHAT IT ENFORCES -# - no direct push to main, and no force-push, by anyone including admins +# Idempotent: updates the ruleset if it exists, creates it otherwise. +# +# What it enforces +# - no direct push to main, and no force-push # - main cannot be deleted -# - every change arrives by pull request, with 1 approving review +# - every change arrives by pull request, with an approving review # - a review is dismissed when new commits are pushed -# - CODEOWNERS review required -# - the `CI` check must pass, against the merge commit, not a stale run -# - linear history — squash or rebase, no merge bubbles +# - review from a code owner +# - the `CI` check must pass, evaluated against an up-to-date branch +# - linear history: squash or rebase, no merge bubbles # -# `bypass_actors` is deliberately EMPTY. A ruleset an admin can walk around is a -# convention, not a control, and the admin is the only account here that can push -# to main in the first place. +# `bypass_actors` is empty on purpose. A rule that some accounts can step around +# is a convention rather than a control, and the point of this file is the control. # ───────────────────────────────────────────────────────────────────────────── set -euo pipefail @@ -79,15 +69,9 @@ if [ "$DRY" = "1" ]; then exit 0 fi -# Refuse early with a readable reason rather than a raw 403. +# Fail with something readable rather than a raw API error. if ! gh api "repos/$REPO/rulesets" >/dev/null 2>&1; then - cat >&2 <<'MSG' -Rulesets are not available on this repository yet. - - A private repository in a Free organisation cannot have protected branches. - Make the repository public, or move AxiomCodeAI to the Team plan, then run - this script again. Nothing else about it needs to change. -MSG + echo "cannot read rulesets on $REPO — check that this account has admin rights there" >&2 exit 1 fi diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 73ac85ea3..e03ce7bbf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -31,6 +31,7 @@ permissions: env: NODE_VERSION: '20' SOUFFLE_VERSION: '2.5' + SOUFFLE_SHA512: '6b86e554f6aa5abf8a8b55d8312ae37c0957c5bd6c9edeea89246db9406f645ec5e600b84fe6636b1c163da556f0da6c3d2dad46c1083413f2fcf4f95b9ac62c' jobs: build: @@ -90,13 +91,39 @@ jobs: strategy: fail-fast: false matrix: - lang: [java, python, typescript] + include: + # --oracle scores the engine against GROUND TRUTH, not only against the + # goldens. A golden says "the same as last time", which a wrong answer + # satisfies perfectly well as long as it was wrong last time too. + # + # The ground truth is built with the toolchain that defines the language: + # javac and javap for Java, the TypeScript compiler for TypeScript. No + # third-party analyzer, and no third-party library is downloaded to do it. + # A case whose ground truth would need an external classpath reports + # itself unscored rather than pulling one in. + - lang: java + oracle: '--oracle' + - lang: typescript + oracle: '--oracle' + # Python's ground truth is frozen CPython output, authored by a separate + # harness that CI cannot reach yet, so this leg is goldens-only for now. + # That separation is deliberate — test/python/run-tests.sh explains why + # the ability to re-bless ground truth must not sit beside the code under + # test — but it does mean the python leg is a weaker check than the other + # two until the harness is reachable from here. + - lang: python + oracle: '' steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} + cache: npm + + # The TypeScript ground-truth oracle runs the compiler from this repo's own + # devDependencies. Without this, every case reports "oracle refused". + - run: npm ci # The python suite's tier-1 attribution preflight reads CPython opcodes, # which are not stable across minor versions. 3.10 is the pinned one. @@ -113,25 +140,39 @@ jobs: distribution: temurin java-version: '24' - # ── Soufflé ──────────────────────────────────────────────────────────── - # Installed from the upstream release .deb for this exact runner image. - # Pinned: Soufflé's evaluation changes between minor versions, so an - # unpinned solver makes every golden in this repo a moving target. + - name: cache the Soufflé package + uses: actions/cache@v4 + with: + path: ~/souffle-pkg + key: souffle-deb-${{ env.SOUFFLE_VERSION }}-ubuntu-2404 + + # Soufflé is the solver, not a library under test: the engine is compiled + # from .dl to C++ and linked against Soufflé's headers, so a build of it has + # to be present the way a compiler has to be present. + # + # It is pinned to an exact version AND verified against the checksum upstream + # published for that release, so what CI links against is decided in this + # file rather than by whatever the archive happens to serve today. Building + # it from source instead would take roughly twenty minutes a run and buy + # nothing unless this project ever needs a patched solver. - name: install Soufflé ${{ env.SOUFFLE_VERSION }} run: | set -euo pipefail deb="x86_64-ubuntu-2404-souffle-${SOUFFLE_VERSION}-Linux.deb" - url="https://github.com/souffle-lang/souffle/releases/download/${SOUFFLE_VERSION}/${deb}" - curl -fsSL --retry 3 -o "/tmp/${deb}" "$url" + dir="$HOME/souffle-pkg"; mkdir -p "$dir" + if [ ! -f "$dir/$deb" ]; then + curl -fsSL --retry 3 -o "$dir/$deb" \ + "https://github.com/souffle-lang/souffle/releases/download/${SOUFFLE_VERSION}/${deb}" + fi + echo "${SOUFFLE_SHA512} ${dir}/${deb}" | sha512sum -c - sudo apt-get update -qq - sudo apt-get install -y --no-install-recommends "/tmp/${deb}" + sudo apt-get install -y --no-install-recommends "$dir/$deb" souffle --version | head -2 # ── the parser ───────────────────────────────────────────────────────── - # An external, private repository. Pinned by SHA in .github/parser-ref so - # that a parser change cannot turn this repo red without a reviewable - # commit here — see the nightly parser-drift workflow, which is what - # notices the pin has gone stale. + # A separate repository, pinned by commit in .github/parser-ref, so that a + # change there cannot turn this repository red with no commit here to point + # at. The nightly parser-drift workflow is what notices the pin has aged. - name: read the pinned parser ref id: parser run: | @@ -141,14 +182,16 @@ jobs: echo "ref=$ref" >> "$GITHUB_OUTPUT" echo "parser pinned at $ref" - - name: fail early if the parser key is missing + # Fail here, with a sentence that says what is wrong, rather than thirty + # lines later as an unexplained checkout error. + - name: the parser credential is present env: - KEY: ${{ secrets.PARSER_DEPLOY_KEY }} + TOKEN: ${{ secrets.PARSER_TOKEN }} run: | - if [ -z "${KEY:-}" ]; then - echo "::error::secret PARSER_DEPLOY_KEY is not set, so the parser cannot be" \ - "fetched and every suite would skip itself. Fork pull requests do not" \ - "receive secrets by design — such a pull request must be run from a" \ + if [ -z "${TOKEN:-}" ]; then + echo "::error::secret PARSER_TOKEN is not set, so the parser cannot be" \ + "fetched and every suite would skip itself. Pull requests opened from" \ + "a fork do not receive secrets, by design — run such a change from a" \ "branch in this repository." exit 1 fi @@ -157,7 +200,7 @@ jobs: with: repository: AxiomCodeAI/parser ref: ${{ steps.parser.outputs.ref }} - ssh-key: ${{ secrets.PARSER_DEPLOY_KEY }} + token: ${{ secrets.PARSER_TOKEN }} path: .parser # tree-sitter builds native addons, which is the slow part. Keyed on the @@ -201,7 +244,7 @@ jobs: - name: ${{ matrix.lang }} regression suite env: AXIOM_PARSER: ${{ github.workspace }}/.parser/dist/index.js - run: bash .github/scripts/run-suite.sh ${{ matrix.lang }} + run: bash .github/scripts/run-suite.sh ${{ matrix.lang }} ${{ matrix.oracle }} # A single job the branch ruleset can require. Without it, every new matrix # entry has to be added to the protection rules by hand, and a matrix job that @@ -215,7 +258,11 @@ jobs: steps: - name: every required job succeeded run: | - results='${{ join(needs.*.result, " ") }}' + # The expression quotes its separator with SINGLE quotes because that is + # the only string delimiter a GitHub expression has. A double quote there + # is a lex error that invalidates the entire workflow file, and the run + # then fails in zero seconds with no job having started. + results="${{ join(needs.*.result, ' ') }}" echo "upstream results: $results" for r in $results; do [ "$r" = "success" ] || { echo "::error::a required job reported '$r'"; exit 1; } diff --git a/.github/workflows/main-guard.yml b/.github/workflows/main-guard.yml index 463af830c..040023dff 100644 --- a/.github/workflows/main-guard.yml +++ b/.github/workflows/main-guard.yml @@ -1,16 +1,10 @@ # ───────────────────────────────────────────────────────────────────────────── -# Direct-push guard for `main`. +# Every commit on `main` should have arrived through a pull request. # -# This is a DETECTIVE control, not a preventive one, and it exists only because -# a preventive one is currently unavailable: protected branches are refused on a -# private repository in a Free organisation. Actions cannot reject a push — by -# the time a workflow runs, the commit is already on the branch. -# -# So this notices instead: it fails, and opens an issue, when a commit arrives on -# main that no pull request carried. When .github/scripts/protect-main.sh can -# finally run, this workflow becomes redundant and should be deleted — leaving it -# in place afterwards is how a repository ends up with a guard nobody reads -# standing next to a rule that already made it impossible. +# This reports when one did not. It is a backstop and not the rule itself: a +# workflow runs after the push has already been accepted, so it can record a +# direct push but never refuse one. The rule that refuses lives in +# .github/scripts/protect-main.sh. # ───────────────────────────────────────────────────────────────────────────── name: main-guard @@ -65,7 +59,7 @@ jobs: echo echo "These commits are on \`main\` without a pull request:" echo - printf ' - `%s`\n' "${orphans[@]}" + printf -- '- %s\n' "${orphans[@]}" } >> "$GITHUB_STEP_SUMMARY" printf '%s\n' "${orphans[@]}" > /tmp/orphans.txt @@ -82,8 +76,9 @@ jobs: title="Direct push to main on $(date -u +%Y-%m-%d)" # One issue per day, not one per push. existing="$(gh issue list --state open --search "\"$title\" in:title" --json number --jq '.[0].number' || true)" - body="$(printf 'Commits reached \`main\` without a pull request:\n\n```\n%s\n```\n\nRun: %s/%s/actions/runs/%s\n\nThis is caught after the fact because branch protection is unavailable on a private repository in a Free organisation. `.github/scripts/protect-main.sh` makes it impossible instead.\n' \ - "$(cat /tmp/orphans.txt)" "$GITHUB_SERVER_URL" "$GITHUB_REPOSITORY" "$GITHUB_RUN_ID")" + run_url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" + body="$(printf "Commits reached \`main\` without a pull request:\n\n\`\`\`\n%s\n\`\`\`\n\nRun: %s\n\nReported after the fact: a workflow runs once the push has been accepted, so it can record a direct push but not refuse one. See .github/scripts/protect-main.sh\n" \ + "$(cat /tmp/orphans.txt)" "$run_url")" if [ -n "${existing:-}" ]; then gh issue comment "$existing" --body "$body" else diff --git a/.github/workflows/parser-drift.yml b/.github/workflows/parser-drift.yml index 7a3885d5e..076a61539 100644 --- a/.github/workflows/parser-drift.yml +++ b/.github/workflows/parser-drift.yml @@ -24,6 +24,7 @@ permissions: env: NODE_VERSION: '20' SOUFFLE_VERSION: '2.5' + SOUFFLE_SHA512: '6b86e554f6aa5abf8a8b55d8312ae37c0957c5bd6c9edeea89246db9406f645ec5e600b84fe6636b1c163da556f0da6c3d2dad46c1083413f2fcf4f95b9ac62c' jobs: against-parser-main: @@ -41,20 +42,40 @@ jobs: - uses: actions/setup-python@v5 with: { python-version: '3.10' } - - name: install Soufflé + - name: cache the Soufflé package + uses: actions/cache@v4 + with: + path: ~/souffle-pkg + key: souffle-deb-${{ env.SOUFFLE_VERSION }}-ubuntu-2404 + + # Soufflé is the solver, not a library under test: the engine is compiled + # from .dl to C++ and linked against Soufflé's headers, so a build of it has + # to be present the way a compiler has to be present. + # + # It is pinned to an exact version AND verified against the checksum upstream + # published for that release, so what CI links against is decided in this + # file rather than by whatever the archive happens to serve today. Building + # it from source instead would take roughly twenty minutes a run and buy + # nothing unless this project ever needs a patched solver. + - name: install Soufflé ${{ env.SOUFFLE_VERSION }} run: | set -euo pipefail deb="x86_64-ubuntu-2404-souffle-${SOUFFLE_VERSION}-Linux.deb" - curl -fsSL --retry 3 -o "/tmp/${deb}" \ - "https://github.com/souffle-lang/souffle/releases/download/${SOUFFLE_VERSION}/${deb}" + dir="$HOME/souffle-pkg"; mkdir -p "$dir" + if [ ! -f "$dir/$deb" ]; then + curl -fsSL --retry 3 -o "$dir/$deb" \ + "https://github.com/souffle-lang/souffle/releases/download/${SOUFFLE_VERSION}/${deb}" + fi + echo "${SOUFFLE_SHA512} ${dir}/${deb}" | sha512sum -c - sudo apt-get update -qq - sudo apt-get install -y --no-install-recommends "/tmp/${deb}" + sudo apt-get install -y --no-install-recommends "$dir/$deb" + souffle --version | head -2 - uses: actions/checkout@v4 with: repository: AxiomCodeAI/parser ref: main - ssh-key: ${{ secrets.PARSER_DEPLOY_KEY }} + token: ${{ secrets.PARSER_TOKEN }} path: .parser - name: compare the pin against parser main @@ -64,9 +85,11 @@ jobs: pinned="$(grep -vE '^\s*(#|$)' .github/parser-ref | head -1 | tr -d '[:space:]')" head="$(cd .parser && git rev-parse HEAD)" behind="$(cd .parser && git rev-list --count "${pinned}..HEAD" 2>/dev/null || echo unknown)" - echo "pinned=$pinned" >> "$GITHUB_OUTPUT" - echo "head=$head" >> "$GITHUB_OUTPUT" - echo "behind=$behind" >> "$GITHUB_OUTPUT" + { + echo "pinned=$pinned" + echo "head=$head" + echo "behind=$behind" + } >> "$GITHUB_OUTPUT" echo "pin is $behind commit(s) behind parser main ($head)" - name: build the parser @@ -103,7 +126,7 @@ jobs: run: | set -uo pipefail title="parser drift: the suites do not pass against parser main" - body="$(printf 'The nightly run against \`AxiomCodeAI/parser@main\` failed.\n\n- pinned in \`.github/parser-ref\`: \`%s\`\n- parser main: \`%s\`\n- the pin is **%s** commit(s) behind\n\nRun: %s/%s/actions/runs/%s\n\nThis does not gate anything — CI still builds the pinned parser. It means the next pin bump will carry a behaviour change, and the diff is cheapest to read now rather than after another month of drift.\n' \ + body="$(printf "The nightly run against AxiomCodeAI/parser@main failed.\n\n- pinned in .github/parser-ref: %s\n- parser main: %s\n- the pin is **%s** commit(s) behind\n\nRun: %s/%s/actions/runs/%s\n\nThis does not gate anything: CI still builds the pinned parser. It means the next pin bump will carry a behaviour change, and that diff is cheapest to read now rather than after another month of drift.\n" \ "$PINNED" "$HEAD" "$BEHIND" "$GITHUB_SERVER_URL" "$GITHUB_REPOSITORY" "$GITHUB_RUN_ID")" n="$(gh issue list --state open --search "\"$title\" in:title" --json number --jq '.[0].number' || true)" if [ -n "${n:-}" ]; then diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index 6f0c8ee52..000000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,99 +0,0 @@ -# Contributing - -## The short version - -`main` is not writable. Every change arrives as a pull request from a branch, is -reviewed, and merges only when CI is green. - -## Before you write code: open an issue - -Not only for large changes — for every change. The issue is where the measurement -that found the problem lives; the pull request is only the fix. This repository runs -several people and agents against different front ends at once, and issue-first is -what keeps that legible. - -Pick the template that matches: engine defect, parser-blocked, harness, enhancement. - -## Describing a defect - -**Use a synthetic example. Never name the project you found it in.** No identifiers, -no file names, no pasted code from a real codebase — in issues, pull requests, or -commit messages. Reduce it to `class Widget`, `pkg/_helpers.py`, `options: "Options"` -and describe the mechanism: what the rule joins on, why the join fails, what the fix -is. - -The tracker is a durable record. Naming someone's codebase turns a bug report into a -benchmark claim about their code, which is not what this is. - -Keep the corpus measurement out of the prose too. Describe the defect by mechanism, -not by what it cost on some project. - -## Branches - -Short-lived, cut from `origin/main`, named for the front end they touch: - - java/... python/... typescript/... - test/... parser-blocked/... - -Squash merge, and the branch is deleted on merge. There is no long-lived `dev` or -`staging` branch: this is a library, nothing deploys from it, and release stages are -git tags. A forward-merge tax buys nothing here. - -**Work in a `git worktree`, not the shared checkout.** More than one agent works this -tree at a time and it switches branches underneath you. - - git fetch origin - git worktree add ../wt/my-change -b python/my-change origin/main - -Cut from `origin/main`, never from the local `main` ref — it is not fast-forwarded by -the squash merges and drifts arbitrarily far behind. A baseline cut from a stale -`main` has produced a "collapse" that was entirely the baseline. - -## Running the tests - -The suites need two things that are not in this repository: Soufflé, and a built -parser. - - brew install souffle # or: apt-get install souffle - - git clone git@github.com:AxiomCodeAI/parser.git ../Parser - cd ../Parser && git checkout "$(grep -v '^#' ../axiom-code-graph/.github/parser-ref | head -1)" - npm ci && npm run build - -Then, from this repository: - - export AXIOM_PARSER="$PWD/../Parser/dist/index.js" - bash test/java/run-tests.sh - bash test/python/run-tests.sh - bash test/typescript/run-tests.sh - -**A suite that prints `SKIP` has not passed.** It exits 77 when it cannot find the -parser. CI treats that as a failure, and so should you — a green run that tested -nothing is the most expensive kind. - -### The parser is pinned - -`.github/parser-ref` names the exact parser commit CI builds and scores the goldens -against. The goldens are a function of both the rules and the IR, so tracking the -parser's `main` would let a change in the other repository turn this one red with no -commit here to point at. - -To move the pin: change the SHA, run all three suites against that exact build, and -put both in one pull request. A nightly workflow runs against the parser's `main` and -opens an issue when the pin has fallen behind, so it cannot rot unnoticed. - -### Re-blessing a golden - -`--bless` regenerates goldens from the current engine. It is not a way past a red -check. A golden that moves is a change in resolution power and belongs in the pull -request diff with an explanation of which edges changed and why each is correct now. - -The CPython ground truth cannot be re-blessed from here at all — it lives outside the -repository, on purpose. - -## Never modify parser code from here - -The engine is read-only with respect to the parser. A missing fact is filed as -`parser-blocked`, with three things: the parser source that drops it, a synthetic -input that reproduces it, and the real construct it came from. Without all three it -is a hypothesis, not a defect. From 3d90f95f6bb88ac58ac50797c64c07b56ea87a68 Mon Sep 17 00:00:00 2001 From: swapnil <78632212+swapnilpaliwal-sd@users.noreply.github.com> Date: Sat, 12 Sep 2026 01:10:28 -0700 Subject: [PATCH 3/7] test: exclude the java torture families from CI, and pin client->library coverage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two faults found while getting the suites to run on a clean machine. THE TORTURE FAMILIES CANNOT BE SCORED WITHOUT THE PLATFORM IR. The harness stages the JVM platform IR as a library because half the families call java.util.List, Map and the functional interfaces; with it absent those receivers cannot be typed by any rule. Measured: the missing-edge census goes from 10 to 23 and recall vs possible falls to 0.847. The harness already says this in its own warning — the score then measures the staging, not the rules. That IR is 1.8 GB and is built from a JDK source checkout, so it cannot live in a repository, a cache, or a runner. Running the families anyway would paint a red that reads as a regression in whatever change happened to meet it, so CI passes --no-torture and the suite PRINTS that the families were excluded. An excluded family must never be mistakeable for one that passed. Java client->library resolution is still covered: six cases ship their own stub library in lib-src/ and are solved with it as --library. CLIENT->LIBRARY COVERAGE CAN VANISH WITHOUT ANYTHING NOTICING. Most assertions here are client->client. The client->library half rests on fewer fixtures, and deleting one of its goldens does not break anything: the case still runs, still passes, and quietly stops making the claim. A suite can only check the assertions it still has. test/tools/lib-coverage.sh pins the shape of that coverage — every TypeScript case with a lib/ has BOTH goldens, since the delta between them IS the client->library mapping; every Java case with a lib-src/ has its golden; and the counts cannot fall without lowering a floor in the same commit. It needs no parser and no solver, so it runs in seconds alongside the other invariants. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 22 +++++++++++- test/java/run-tests.sh | 18 ++++++++-- test/tools/lib-coverage.sh | 69 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 105 insertions(+), 4 deletions(-) create mode 100755 test/tools/lib-coverage.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e03ce7bbf..bf1efea54 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,6 +76,13 @@ jobs: done exit $fail + # The client->library half is carried by a smaller set of fixtures than the + # client->client half, and it is the half that disappears silently: delete a + # golden and the case still runs, still passes, and simply stops claiming + # anything. A suite can only check the assertions it still has. + - name: client->library coverage has not shrunk + run: bash test/tools/lib-coverage.sh + - name: shell scripts parse run: | fail=0 @@ -101,8 +108,21 @@ jobs: # third-party analyzer, and no third-party library is downloaded to do it. # A case whose ground truth would need an external classpath reports # itself unscored rather than pulling one in. + # --no-torture for java ONLY. Those families call java.util.List, Map and + # the functional interfaces, so they need the JVM platform IR staged as a + # library. That IR is 1.8 GB and is built from a JDK source checkout, so it + # cannot live in a repository or a cache. Without it those receivers are + # unresolvable BY CONSTRUCTION — the census goes from 10 missing edges to + # 23 and recall to 0.847 — which measures the staging, not the rules, and + # the resulting red would read as a regression in whatever PR met it. + # + # Java client->library resolution is still covered here: six cases ship + # their own stub library in lib-src/ and are solved with it as --library. + # The torture families remain a local gate until the platform IR can be + # produced reproducibly; the suite prints EXCLUDED so it is never mistaken + # for a family that passed. - lang: java - oracle: '--oracle' + oracle: '--oracle --no-torture' - lang: typescript oracle: '--oracle' # Python's ground truth is frozen CPython output, authored by a separate diff --git a/test/java/run-tests.sh b/test/java/run-tests.sh index cd63166f1..4a02b9f51 100755 --- a/test/java/run-tests.sh +++ b/test/java/run-tests.sh @@ -21,6 +21,15 @@ # (javac + javap invoke instructions — no third-party analyzer): # every bytecode-declared client->client edge must be present. # ./run-tests.sh --keep keep the per-case work dirs for debugging +# ./run-tests.sh --no-torture skip the torture families. +# ONLY for an environment that cannot hold the JVM platform IR. +# The families call java.util.List, Map and the functional +# interfaces, so with the platform absent those receivers are +# unresolvable BY CONSTRUCTION: the census goes from 10 missing +# edges to 23 and recall to 0.847, which measures the staging +# rather than the rules. Running them without it is worse than +# not running them, because the resulting red looks like a +# regression. Excluding them is stated in the output, never silent. # # With --oracle, a case carrying a spring-oracle.conf ALSO boots its sources in a real # AnnotationConfigApplicationContext and scores bean_def / di_edge against what Spring @@ -98,9 +107,9 @@ if ! bash "$ROOT/test/tools/souffle-include-test.sh"; then fi PARSER="${AXIOM_PARSER:-$ROOT/../Parser/dist/index.js}" WORK="$HERE/.work" -BLESS=0; KEEP=0; ORACLE=0; FILTERS=() +BLESS=0; KEEP=0; ORACLE=0; NO_TORTURE=0; FILTERS=() for a in "$@"; do case "$a" in - --bless) BLESS=1;; --keep) KEEP=1;; --oracle) ORACLE=1;; + --bless) BLESS=1;; --keep) KEEP=1;; --oracle) ORACLE=1;; --no-torture) NO_TORTURE=1;; -h|--help) sed -n '2,34p' "$0"; exit 0;; *) FILTERS+=("$a");; esac; done # ── PREFLIGHT: every relation the parser emits must actually reach the solver ────────── @@ -339,7 +348,7 @@ done # The cases above each pin ONE rule. This asks what happens when a project uses everything at # once, and reports WHICH construct is the gap rather than one number. It stages the platform IR, # because half the families call java.util types and scoring them without it measures the staging. -if [ -d "$HERE/torture" ] && [ ${#FILTERS[@]} -eq 0 ]; then +if [ -d "$HERE/torture" ] && [ ${#FILTERS[@]} -eq 0 ] && [ "$NO_TORTURE" = 0 ]; then printf '%-34s ' "torture (10 families)" # --bless has to reach the torture harness too, or a run that regenerates every other golden # leaves this one stale and the very next run fails on a diff the operator just approved. @@ -353,6 +362,9 @@ if [ -d "$HERE/torture" ] && [ ${#FILTERS[@]} -eq 0 ]; then fi fi +if [ "$NO_TORTURE" = 1 ] && [ ${#FILTERS[@]} -eq 0 ]; then + echo "torture (10 families) EXCLUDED (--no-torture)" +fi echo "─────────────────────────────────────────────" echo "passed $pass failed $fail" [ $fail -eq 0 ] || { printf 'failing: %s\n' "${failed[*]}"; exit 1; } diff --git a/test/tools/lib-coverage.sh b/test/tools/lib-coverage.sh new file mode 100755 index 000000000..c1b119d18 --- /dev/null +++ b/test/tools/lib-coverage.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# ── The client->library half must not quietly stop being tested ────────────── +# Most of what these suites assert is client->client. The client->library hand-off +# is carried by a smaller set of fixtures, and it is the half that disappears +# silently: delete a golden and the case still runs, still passes, and simply +# stops making the claim. Nothing in the suites notices, because a suite can only +# check the assertions it still has. +# +# So this pins the SHAPE of that coverage, and it needs no parser and no solver: +# +# 1. every TypeScript case carrying a lib/ is solved twice, and both goldens +# exist — the delta between them IS the client->library mapping; +# 2. every Java case carrying a lib-src/ stub library still has its golden; +# 3. the counts never fall. A number here going DOWN is either a deletion that +# wanted review, or coverage lost by accident. +set -uo pipefail +cd "$(dirname "$0")/../.." || exit 2 + +fail=0 + +# Floors, not targets. Raise one when real coverage is added; lowering one is a +# deliberate reduction in what this repository proves, and belongs in a diff. +TS_LIB_FLOOR=23 +JAVA_LIB_FLOOR=6 +PY_LIB_FLOOR=1 + +ts_libs=0; ts_missing=() +for d in test/typescript/cases/*/lib; do + [ -d "$d" ] || continue + c="$(basename "$(dirname "$d")")"; ts_libs=$((ts_libs+1)) + [ -f "test/typescript/expected/$c.edges" ] || ts_missing+=("$c.edges") + [ -f "test/typescript/expected/$c.lib.edges" ] || ts_missing+=("$c.lib.edges") +done + +java_libs=0; java_missing=() +for d in test/java/cases/*/lib-src; do + [ -d "$d" ] || continue + c="$(basename "$(dirname "$d")")"; java_libs=$((java_libs+1)) + [ -f "test/java/expected/$c.edges" ] || java_missing+=("$c.edges") +done + +py_libs=0 +[ -d test/python/torture/lib ] && py_libs=1 + +printf 'client->library coverage: typescript %d (both goldens each), java %d stub libs, python %d torture lib\n' \ + "$ts_libs" "$java_libs" "$py_libs" + +if [ ${#ts_missing[@]} -gt 0 ]; then + echo " a TypeScript case ships a lib/ but is missing a golden, so its client->library half asserts nothing:" + printf ' %s\n' "${ts_missing[@]}"; fail=1 +fi +if [ ${#java_missing[@]} -gt 0 ]; then + echo " a Java case ships a lib-src/ but is missing its golden:" + printf ' %s\n' "${java_missing[@]}"; fail=1 +fi + +check_floor() { # name actual floor + if [ "$2" -lt "$3" ]; then + echo " $1 client->library coverage fell: $2, was $3." + echo " If that removal was intended, lower the floor in this file in the same commit." + fail=1 + fi +} +check_floor typescript "$ts_libs" "$TS_LIB_FLOOR" +check_floor java "$java_libs" "$JAVA_LIB_FLOOR" +check_floor python "$py_libs" "$PY_LIB_FLOOR" + +[ "$fail" = 0 ] && echo " 0 violation(s)" +exit $fail From a39ed5404c0996852d2762da49a2a9d2a2e741a6 Mon Sep 17 00:00:00 2001 From: swapnil <78632212+swapnilpaliwal-sd@users.noreply.github.com> Date: Sat, 12 Sep 2026 09:02:10 -0700 Subject: [PATCH 4/7] ci: name the parser credential PARSER_ACCESS_TOKEN The secret says what it is rather than leaving "token" to carry the meaning. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 6 +++--- .github/workflows/parser-drift.yml | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bf1efea54..3a21dc5d4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -206,10 +206,10 @@ jobs: # lines later as an unexplained checkout error. - name: the parser credential is present env: - TOKEN: ${{ secrets.PARSER_TOKEN }} + TOKEN: ${{ secrets.PARSER_ACCESS_TOKEN }} run: | if [ -z "${TOKEN:-}" ]; then - echo "::error::secret PARSER_TOKEN is not set, so the parser cannot be" \ + echo "::error::secret PARSER_ACCESS_TOKEN is not set, so the parser cannot be" \ "fetched and every suite would skip itself. Pull requests opened from" \ "a fork do not receive secrets, by design — run such a change from a" \ "branch in this repository." @@ -220,7 +220,7 @@ jobs: with: repository: AxiomCodeAI/parser ref: ${{ steps.parser.outputs.ref }} - token: ${{ secrets.PARSER_TOKEN }} + token: ${{ secrets.PARSER_ACCESS_TOKEN }} path: .parser # tree-sitter builds native addons, which is the slow part. Keyed on the diff --git a/.github/workflows/parser-drift.yml b/.github/workflows/parser-drift.yml index 076a61539..f1b269fd6 100644 --- a/.github/workflows/parser-drift.yml +++ b/.github/workflows/parser-drift.yml @@ -75,7 +75,7 @@ jobs: with: repository: AxiomCodeAI/parser ref: main - token: ${{ secrets.PARSER_TOKEN }} + token: ${{ secrets.PARSER_ACCESS_TOKEN }} path: .parser - name: compare the pin against parser main From 30dd3ddcfc20588abf73e419667339665a4386a2 Mon Sep 17 00:00:00 2001 From: swapnil <78632212+swapnilpaliwal-sd@users.noreply.github.com> Date: Sat, 12 Sep 2026 09:37:54 -0700 Subject: [PATCH 5/7] ci: say what is actually wrong with the parser credential MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit git reports every credential problem against a private repository as "remote: Write access to repository not granted ... 403", whatever the real cause was. It is not about write access, and the token is often not the thing that is wrong, so the message sends you looking in the wrong place. Ask the API first and name the fault: 401 invalid, 403 policy, 404 invisible — and 404 is what GitHub returns rather than confirming a private repository exists, so it is equally what a token with the wrong RESOURCE OWNER looks like. That last one is the trap, so the step also reports how many repositories the credential can reach and how many are the organisation's. None means the token belongs to a personal account. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3a21dc5d4..1bae3e94c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -216,6 +216,41 @@ jobs: exit 1 fi + # git reports EVERY credential problem against a private repository as + # "remote: Write access to repository not granted ... error: 403", whatever + # the real cause was. That message is actively misleading — it is not about + # write access, and the token is usually not even the thing that is wrong — + # so this asks the API first and says what actually happened. The secret is + # masked in logs by the runner; only status codes are printed. + - name: the credential can actually read the parser repository + env: + TOKEN: ${{ secrets.PARSER_ACCESS_TOKEN }} + run: | + set -uo pipefail + code=$(curl -s -o /dev/null -w '%{http_code}' \ + -H "Authorization: Bearer $TOKEN" \ + https://api.github.com/repos/AxiomCodeAI/parser) + echo "GET /repos/AxiomCodeAI/parser -> $code" + case "$code" in + 200) echo "credential can read the parser repository"; exit 0;; + 401) echo "::error::the credential is not valid — expired, revoked, or truncated when it was stored.";; + 403) echo "::error::the credential is recognised but blocked by organisation policy.";; + 404) echo "::error::the credential cannot see AxiomCodeAI/parser at all. GitHub returns 404 rather than 403 to avoid confirming that a private repository exists, so this is what a token with the WRONG RESOURCE OWNER looks like, and also what an unapproved one looks like.";; + *) echo "::error::unexpected status $code";; + esac + echo "--- what this credential can reach, to locate the fault ---" + n=$(curl -s -H "Authorization: Bearer $TOKEN" \ + "https://api.github.com/user/repos?per_page=100" \ + | grep -c '"full_name"' || true) + org=$(curl -s -H "Authorization: Bearer $TOKEN" \ + "https://api.github.com/user/repos?per_page=100" \ + | grep -c '"full_name": "AxiomCodeAI/' || true) + echo " repositories visible to it: $n, of which under AxiomCodeAI: $org" + if [ "$org" = "0" ]; then + echo "::error::it can see no AxiomCodeAI repository, so its resource owner is a personal account rather than the organisation. Recreate the token with Resource owner = AxiomCodeAI." + fi + exit 1 + - uses: actions/checkout@v4 with: repository: AxiomCodeAI/parser From 97e89719396ef336e2f84ece2f89055d66eaa67b Mon Sep 17 00:00:00 2001 From: swapnil <78632212+swapnilpaliwal-sd@users.noreply.github.com> Date: Sat, 12 Sep 2026 09:49:06 -0700 Subject: [PATCH 6/7] ci: the python suite needs two interpreters, not one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The torture family failed in CI with nothing but "trace failed", because the harness sends the tracer's output to /dev/null. Run directly, the cause is exact: client/f27_with_target.py does `from typing import Self`, and typing.Self is 3.11+ (PEP 673), so on 3.10 the tracer dies at import and the whole client+lib family — the only python coverage of a library boundary — scores nothing. 3.10 cannot simply be dropped: the tier-1 attribution preflight reads CPython opcodes, whose shapes are not stable across minor versions, and it resolves python3.10 by name. So both are installed, 3.12 second because the later setup-python wins for plain python3, and a step prints both so a future divergence is visible in the log rather than inferred from a failure. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1bae3e94c..5d2290d0e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -145,12 +145,31 @@ jobs: # devDependencies. Without this, every case reports "oracle refused". - run: npm ci - # The python suite's tier-1 attribution preflight reads CPython opcodes, - # which are not stable across minor versions. 3.10 is the pinned one. + # TWO interpreters, because the python suite needs two different things and + # they cannot be the same version. + # + # 3.10 — the tier-1 attribution preflight reads CPython OPCODES, whose + # shapes are not stable across minor versions. It resolves + # `python3.10` by name, so this only has to exist on PATH. + # 3.12 — the torture fixtures are SOURCE that has to import: one of them + # uses `typing.Self`, which is 3.11+ (PEP 673), so on 3.10 the + # tracer dies at import and the family scores nothing. + # + # The later setup-python wins for plain `python3`, so 3.12 must come second. - uses: actions/setup-python@v5 with: python-version: '3.10' + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: both interpreters are on PATH + run: | + set -euo pipefail + echo "python3 -> $(python3 --version)" + echo "python3.10 -> $(python3.10 --version)" + # JDK 24, not the runner's default. The java torture harness reads class files # with java.lang.classfile, which is not final before 24 — on an older JDK it # exits 77 and the whole ten-family oracle silently does not run. From 8df2c1c003a9eb5fb334883eb7cde91127c01cc5 Mon Sep 17 00:00:00 2001 From: swapnil <78632212+swapnilpaliwal-sd@users.noreply.github.com> Date: Mon, 14 Sep 2026 02:42:19 -0700 Subject: [PATCH 7/7] ci: no lock file is committed, so the gate installs with npm install and uses no npm cache Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d41499dbc..dc667397f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,11 +47,12 @@ jobs: - uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} - cache: npm - # `npm ci` runs the `prepare` script, which builds the parser workspace and - # the driver. Keeping the explicit build step anyway means a prepare-script + # No lock file is committed (#476), so this is `npm install`, not `npm ci`, + # and setup-node's npm cache (keyed on a lock file) is not used. The install + # runs the `prepare` script, which builds the parser workspace and the + # driver. Keeping the explicit build step anyway means a prepare-script # change cannot silently stop compiling this repo. - - run: npm ci + - run: npm install - run: npm run typecheck - run: npm run build - name: the parser actually built @@ -164,7 +165,7 @@ jobs: # JavaScript: 19 cases; the library case ships its dependency under # src/node_modules and is solved twice. The execution oracles (torture/, # realapp/) are separate harnesses and stay a local gate — realapp needs - # network for its own npm ci. + # network for its own npm install. - lang: javascript oracle: '--oracle' steps: @@ -173,16 +174,16 @@ jobs: - uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} - cache: npm # Builds the in-repo parser (parser/dist) through the prepare script, and # supplies the TypeScript compiler the typescript and javascript oracles run. - - run: npm ci + # `npm install`, not `npm ci`: no lock file is committed (#476). + - run: npm install - name: the parser actually built run: | test -f parser/dist/index.js \ - || { echo "::error::parser/dist/index.js is missing after npm ci"; exit 1; } + || { echo "::error::parser/dist/index.js is missing after npm install"; exit 1; } # TWO interpreters, because the python suite needs two different things and # they cannot be the same version.