Skip to content

ci(test-impact): give the engine full history; publish maps with an incomplete suite - #2567

Merged
DeusData merged 1 commit into
mainfrom
ci/test-impact-full-history
Oct 9, 2026
Merged

DeusData merged 1 commit into
mainfrom
ci/test-impact-full-history

Conversation

@DeusData

@DeusData DeusData commented Oct 8, 2026

Copy link
Copy Markdown
Owner

What

The smart-CI engine never ran for real in CI. This PR fixes the two causes:

  1. Shallow checkouts. test-impact select and publish certify the history they diff and refuse a shallow clone: Graft or shallow state prevents history certification. Three jobs checked out with fetch-depth: 2, and all three now use fetch-depth: 0:

    • test-impact-shadow
    • select-tests (gate mode)
    • the producer in test-impact-artifact.yml

    Full history is about 400 MiB, in these jobs only.

  2. The producer never published. test_impact_git failed under the coverage runner, and test-impact-publish.sh ran coverage-map.py under set -e, so one failing suite aborted every publish.

    • The test: the hostile-diff probe re-enters the runner as a child. That child inherited CBM_TEST_COVERAGE_DIR, started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE it inherits. The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears CBM_TEST_ONLY_FILE. The child's coverage then goes to the spawning test, like any other child process.
    • The script: coverage-map.py exit 1 means the map was written and that suite's tests are incomplete. Incomplete tests only ever select more tests, so the script now publishes with a warning that names the suite. It fails only on exit 2 (usage or toolchain).

Why

Measured on the 24 successful PR runs since #2458 merged:

  • Lane selection already saves 54.6% of runner-minutes against a full run of 753 runner-minutes.
  • The engine shadow answered run_all in 24 of 24 runs, with GIT_UNAVAILABLE and POLICY_UNAVAILABLE (the policy is read through the same git facts).
  • On every push to main, the producer failed or was cancelled, so there was never a bundle and never coverage.

Test-level narrowing needs both fixes:

  • Without coverage, the engine's static reach selects every suite. On fix(extract): preserve Go router group prefixes #2497's change, with full history, it answered selected with all 170 suites (99.8% of x86 test time).
  • With coverage, the engine selected 55.6% of tests in narrowed jobs with 0 misses (oracle, 191 replayed jobs).

Proof

Step Result
Depth-2 clone at f9ce386, test-impact select run_all [GIT_UNAVAILABLE, POLICY_UNAVAILABLE], "Graft or shallow state ...", as in CI
Same commit, full history graph certified, run_all [RUNNER_REACHED]: a genuine decision, since the change reaches the runner's main
#2497's change, full history selected with no run_all reasons, 170 of 170 suites whole (no coverage)
Coverage runner (Homebrew LLVM, macOS), test_impact_git before the fix 43 passed, 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null")
Same, after the fix 45 passed
coverage-map.py --suite test_impact_git after the fix 45 tests, 0 incomplete, exit 0

Scope (CI change)

  • .github/workflows/pr.yml: the fetch-depth of two jobs. Both are report-only today: the shadow job is continue-on-error and not in ci-ok's needs, and select-tests runs only when TEST_IMPACT_MODE is gate.
  • .github/workflows/test-impact-artifact.yml: the fetch-depth of the producer.
  • scripts/ci/test-impact-publish.sh: exit-1 tolerance with a warning.
  • tests/test_test_impact_git.c: the probe clears one more inherited variable.

Nothing gates on any of this yet. ci-ok's inputs are unchanged.

…ncomplete suite

The test-impact engine certifies the history it diffs and refuses a shallow
clone ("Graft or shallow state prevents history certification"). The shadow
job, the gate-mode select-tests job and the main-branch producer all checked
out with fetch-depth 2. So every shadow answer since #2458 was run_all:
GIT_UNAVAILABLE, then POLICY_UNAVAILABLE, because the merge base's
.codebase-memory.json is read through the same git facts. That was 24 of 24
measured PR runs. All three jobs now check out full history (about 400 MiB,
these jobs only).

The producer never published a bundle either. coverage-map.py exits 1 when a
suite fails under coverage: the map is still written and that suite's tests
are incomplete. test-impact-publish.sh ran it under set -e, so the publish
aborted. The failing suite was test_impact_git. Its hostile-diff probe
re-enters the runner as a child, which inherited CBM_TEST_COVERAGE_DIR,
started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE
it inherits.

Two changes:
- The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears
  CBM_TEST_ONLY_FILE. The child's profile then goes to the spawning test, like
  any other child process.
- publish treats exit 1 as a written, conservative map (incomplete tests run
  whole), with a warning that names the suite. It fails only on exit 2.

Proof:
- Depth-2 clone at f9ce386: test-impact select answers run_all
  [GIT_UNAVAILABLE, POLICY_UNAVAILABLE] with "Graft or shallow state ...", as
  in CI. Same commit with full history: graph certified, run_all
  [RUNNER_REACHED] (the change reaches the runner's main).
- Coverage runner (Homebrew LLVM, macOS), test_impact_git: before, 43 passed
  and 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null");
  after, 45 passed. coverage-map.py --suite test_impact_git: 45 tests,
  0 incomplete, exit 0.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit a96064c into main Oct 9, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant