From 674a731fcc9c8ffd0e2679389ead87ce5836142f Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Fri, 9 Oct 2026 01:59:34 +0200 Subject: [PATCH] ci(test-impact): give the engine full history; publish maps with an incomplete suite The test-impact engine certifies the history it diffs and refuses a shallow clone ("Graft or shallow state prevents history certification"). The shadow job, the gate-mode select-tests job and the main-branch producer all checked out with fetch-depth 2. So every shadow answer since #2458 was run_all: GIT_UNAVAILABLE, then POLICY_UNAVAILABLE, because the merge base's .codebase-memory.json is read through the same git facts. That was 24 of 24 measured PR runs. All three jobs now check out full history (about 400 MiB, these jobs only). The producer never published a bundle either. coverage-map.py exits 1 when a suite fails under coverage: the map is still written and that suite's tests are incomplete. test-impact-publish.sh ran it under set -e, so the publish aborted. The failing suite was test_impact_git. Its hostile-diff probe re-enters the runner as a child, which inherited CBM_TEST_COVERAGE_DIR, started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE it inherits. Two changes: - The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears CBM_TEST_ONLY_FILE. The child's profile then goes to the spawning test, like any other child process. - publish treats exit 1 as a written, conservative map (incomplete tests run whole), with a warning that names the suite. It fails only on exit 2. Proof: - Depth-2 clone at f9ce3865: test-impact select answers run_all [GIT_UNAVAILABLE, POLICY_UNAVAILABLE] with "Graft or shallow state ...", as in CI. Same commit with full history: graph certified, run_all [RUNNER_REACHED] (the change reaches the runner's main). - Coverage runner (Homebrew LLVM, macOS), test_impact_git: before, 43 passed and 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null"); after, 45 passed. coverage-map.py --suite test_impact_git: 45 tests, 0 incomplete, exit 0. Signed-off-by: Martin Vogel --- .github/workflows/pr.yml | 12 ++++++++---- .github/workflows/test-impact-artifact.yml | 4 +++- scripts/ci/test-impact-publish.sh | 12 +++++++++++- tests/test_test_impact_git.c | 7 ++++++- 4 files changed, 28 insertions(+), 7 deletions(-) diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 7af2efa5e..33cd4a5af 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -65,8 +65,9 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: - # The PR merge commit and its base parent: the change to select for. - fetch-depth: 2 + # Full history: the engine certifies the history it diffs and refuses + # a shallow clone, so a depth-2 checkout always answered `all`. + fetch-depth: 0 persist-credentials: false - name: Build this checkout's engine run: | @@ -281,8 +282,11 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: - # The PR merge commit and its base parent: the diff to predict for. - fetch-depth: 2 + # Full history: the engine (test-impact select) certifies the history + # it diffs and refuses a shallow clone ("Graft or shallow state + # prevents history certification"), which made every shadow answer + # run_all (GIT_UNAVAILABLE). About 400 MiB, this job only. + fetch-depth: 0 persist-credentials: false - name: Fetch the latest released binary id: release diff --git a/.github/workflows/test-impact-artifact.yml b/.github/workflows/test-impact-artifact.yml index 9a13d9ad0..4db216ecd 100644 --- a/.github/workflows/test-impact-artifact.yml +++ b/.github/workflows/test-impact-artifact.yml @@ -38,7 +38,9 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: - fetch-depth: 2 + # Full history: `test-impact select/publish` certify the history they + # read and refuse a shallow clone. + fetch-depth: 0 persist-credentials: false - name: Install build deps and LLVM 21 (coverage toolchain) diff --git a/scripts/ci/test-impact-publish.sh b/scripts/ci/test-impact-publish.sh index 8e9dbeeb6..fe8f1b053 100755 --- a/scripts/ci/test-impact-publish.sh +++ b/scripts/ci/test-impact-publish.sh @@ -56,9 +56,19 @@ fi map() { # out [suite...] local out="$1"; shift local args=() + local rc=0 for suite in "$@"; do args+=(--suite "$suite"); done python3 "$ROOT/scripts/test-impact/coverage-map.py" --runner "$RUNNER" --out "$out" \ - ${LLVM_BIN:+--llvm-bin "$LLVM_BIN"} ${args[@]+"${args[@]}"} + ${LLVM_BIN:+--llvm-bin "$LLVM_BIN"} ${args[@]+"${args[@]}"} || rc=$? + # 1 = a suite failed or timed out under coverage: the map is still written + # and that suite's tests are `incomplete`, which only ever selects MORE + # tests (an incomplete test runs whole). Publish it, but say which suite. + # 2 = usage or toolchain error: no trustworthy map, fail. + if [ "$rc" -eq 1 ]; then + echo "::warning::coverage map: a suite failed under coverage; its tests stay incomplete ($(python3 -c 'import json,sys; print(", ".join(s["suite"] for s in json.load(open(sys.argv[1]))["suites"] if s["exit"] != 0))' "$out/meta.json"))" + return 0 + fi + return "$rc" } OBSERVED="" diff --git a/tests/test_test_impact_git.c b/tests/test_test_impact_git.c index 85b26951d..0964429b6 100644 --- a/tests/test_test_impact_git.c +++ b/tests/test_test_impact_git.c @@ -64,9 +64,14 @@ int tf_maybe_run_git_facts_diff_probe(int argc, char **argv) { * Re-enter the existing exact-diff test; the parent process remains untouched. * An inherited CBM_TEST_ONLY_FILE (a narrowed CI run) is cleared: the runner * unions it with CBM_TEST_ONLY, and one naming this suite would re-run the - * spawning test, which spawns again. */ + * spawning test, which spawns again. An inherited CBM_TEST_COVERAGE_DIR (a + * coverage-map run) is cleared too: with it the child would start a coverage + * run of its own, which refuses the per-test LLVM_PROFILE_FILE it inherits. + * Without it the child's profile goes to that file, credited to the spawning + * test like any other child process. */ if (cbm_setenv("GIT_DIFF_OPTS", "--unified=999", 1) != 0 || cbm_setenv("CBM_TEST_ONLY_FILE", "", 1) != 0 || + cbm_unsetenv("CBM_TEST_COVERAGE_DIR") != 0 || cbm_setenv( "CBM_TEST_ONLY", "test_impact_git:test_git_facts_diff_uses_merge_base_zero_context_and_nul_metadata",