Repository navigation
ci(test-impact): give the engine full history; publish maps with an incomplete suite - #2567
Merged
Merged
Conversation
…ncomplete suite
The test-impact engine certifies the history it diffs and refuses a shallow
clone ("Graft or shallow state prevents history certification"). The shadow
job, the gate-mode select-tests job and the main-branch producer all checked
out with fetch-depth 2. So every shadow answer since #2458 was run_all:
GIT_UNAVAILABLE, then POLICY_UNAVAILABLE, because the merge base's
.codebase-memory.json is read through the same git facts. That was 24 of 24
measured PR runs. All three jobs now check out full history (about 400 MiB,
these jobs only).
The producer never published a bundle either. coverage-map.py exits 1 when a
suite fails under coverage: the map is still written and that suite's tests
are incomplete. test-impact-publish.sh ran it under set -e, so the publish
aborted. The failing suite was test_impact_git. Its hostile-diff probe
re-enters the runner as a child, which inherited CBM_TEST_COVERAGE_DIR,
started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE
it inherits.
Two changes:
- The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears
CBM_TEST_ONLY_FILE. The child's profile then goes to the spawning test, like
any other child process.
- publish treats exit 1 as a written, conservative map (incomplete tests run
whole), with a warning that names the suite. It fails only on exit 2.
Proof:
- Depth-2 clone at f9ce386: test-impact select answers run_all
[GIT_UNAVAILABLE, POLICY_UNAVAILABLE] with "Graft or shallow state ...", as
in CI. Same commit with full history: graph certified, run_all
[RUNNER_REACHED] (the change reaches the runner's main).
- Coverage runner (Homebrew LLVM, macOS), test_impact_git: before, 43 passed
and 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null");
after, 45 passed. coverage-map.py --suite test_impact_git: 45 tests,
0 incomplete, exit 0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The smart-CI engine never ran for real in CI. This PR fixes the two causes:
Shallow checkouts.
test-impact selectandpublishcertify the history they diff and refuse a shallow clone:Graft or shallow state prevents history certification. Three jobs checked out withfetch-depth: 2, and all three now usefetch-depth: 0:test-impact-shadowselect-tests(gate mode)test-impact-artifact.ymlFull history is about 400 MiB, in these jobs only.
The producer never published.
test_impact_gitfailed under the coverage runner, andtest-impact-publish.shrancoverage-map.pyunderset -e, so one failing suite aborted every publish.CBM_TEST_COVERAGE_DIR, started a coverage run of its own, and refused the per-testLLVM_PROFILE_FILEit inherits. The probe now clearsCBM_TEST_COVERAGE_DIR, as it already clearsCBM_TEST_ONLY_FILE. The child's coverage then goes to the spawning test, like any other child process.coverage-map.pyexit 1 means the map was written and that suite's tests are incomplete. Incomplete tests only ever select more tests, so the script now publishes with a warning that names the suite. It fails only on exit 2 (usage or toolchain).Why
Measured on the 24 successful PR runs since #2458 merged:
run_allin 24 of 24 runs, withGIT_UNAVAILABLEandPOLICY_UNAVAILABLE(the policy is read through the same git facts).Test-level narrowing needs both fixes:
selectedwith all 170 suites (99.8% of x86 test time).Proof
test-impact selectrun_all[GIT_UNAVAILABLE, POLICY_UNAVAILABLE], "Graft or shallow state ...", as in CIrun_all[RUNNER_REACHED]: a genuine decision, since the change reaches the runner's mainselectedwith no run_all reasons, 170 of 170 suites whole (no coverage)test_impact_gitbefore the fixcoverage-map.py --suite test_impact_gitafter the fixScope (CI change)
.github/workflows/pr.yml: thefetch-depthof two jobs. Both are report-only today: the shadow job iscontinue-on-errorand not inci-ok's needs, andselect-testsruns only whenTEST_IMPACT_MODEisgate..github/workflows/test-impact-artifact.yml: thefetch-depthof the producer.scripts/ci/test-impact-publish.sh: exit-1 tolerance with a warning.tests/test_test_impact_git.c: the probe clears one more inherited variable.Nothing gates on any of this yet.
ci-ok's inputs are unchanged.