Skip to content

Commit 4e34fb3

Browse files
committed
Shard coverage across two runners
coverage is the merge queue's critical path since #1697 split the Gradle e2e leg: it ends at 6.05 min in merge_group run 40032760038, while the next-slowest leg ends at 4.82. Its test step is ~330 s, of which the instrumented build is ~110 s (the push run's warm step), so ~220 s is test execution that a second runner can halve. Run it as two shards with the partitioner `test` and `test-release` already use. ci-test-shard.py gains a CI_TEST_COVERAGE mode that runs `cargo llvm-cov --no-report` and skips doctests, as the unsharded `cargo llvm-cov --workspace` did. Each shard uploads coverage-host-<n>, which coverage-merge's `coverage-{host*,docker-*}` pattern already picks up. Shard 1 alone warms and saves the cache. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174QivDXDqjSDhKth66oLF4
1 parent 4b05ab2 commit 4e34fb3

3 files changed

Lines changed: 83 additions & 16 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 28 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -588,7 +588,17 @@ jobs:
588588
# numbers are report-only so contributors get visibility without flaky
589589
# CI when coverage shifts naturally with test edits. A failing TEST
590590
# fails the job: this is the Linux leg of `test`.
591+
#
592+
# Two shards (scripts/ci-test-shard.py, as `test` and `test-release`):
593+
# the instrumented build takes ~110 s and running the tests ~220 s, so
594+
# one leg was the queue's critical path (~6 min). Each shard compiles
595+
# and runs only its targets and uploads its own LCOV; coverage-merge
596+
# unions them. `ci-ok` waits for every shard.
591597
runs-on: ${{ vars.DISABLE_DEPOT_RUNNERS == 'true' && 'ubuntu-latest' || 'depot-ubuntu-24.04-16' }}
598+
strategy:
599+
fail-fast: false
600+
matrix:
601+
shard: [1, 2]
592602
timeout-minutes: 35
593603
permissions:
594604
contents: read
@@ -616,12 +626,14 @@ jobs:
616626
- name: Cache cargo
617627
# Swatinem/rust-cache, main-only saves: see the first `Cache cargo`
618628
# step in this file.
629+
# The shards share one cache key; shard 1 alone writes it, from the
630+
# warm step that builds every target.
619631
uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
620632
with:
621-
save-if: ${{ github.ref == 'refs/heads/main' }}
633+
save-if: ${{ github.ref == 'refs/heads/main' && matrix.shard == 1 }}
622634

623635
- name: Warm the coverage cache after merge-queue validation
624-
if: needs.release-readiness.outputs.reuse == 'true'
636+
if: needs.release-readiness.outputs.reuse == 'true' && matrix.shard == 1
625637
run: |
626638
cargo llvm-cov show-env --sh > "$RUNNER_TEMP/coverage-env.sh"
627639
source "$RUNNER_TEMP/coverage-env.sh"
@@ -672,6 +684,9 @@ jobs:
672684
env:
673685
SOCKET_PATCH_GO_E2E_REQUIRED: '1'
674686
SOCKET_PATCH_GO_E2E_VERSION: '1.24'
687+
CI_TEST_COVERAGE: '1'
688+
TEST_SHARD: ${{ matrix.shard }}
689+
TEST_SHARD_COUNT: ${{ strategy.job-total }}
675690
#
676691
# The test output is also teed to a file so that a failing test's
677692
# panic becomes an annotation (scripts/ci-test-failures.py): in a
@@ -680,27 +695,29 @@ jobs:
680695
# can't be diagnosed once the re-run has passed.
681696
run: |
682697
set -o pipefail
683-
cargo llvm-cov --workspace --no-fail-fast \
684-
--no-report 2>&1 | tee "$RUNNER_TEMP/coverage-tests.log" || {
698+
python3 scripts/ci-test-shard.py "$TEST_SHARD" "$TEST_SHARD_COUNT" \
699+
2>&1 | tee "$RUNNER_TEMP/coverage-tests.log" || {
685700
status=$?
686701
python3 scripts/ci-test-failures.py "$RUNNER_TEMP/coverage-tests.log" || true
687702
exit "$status"
688703
}
689704
set +o pipefail
690-
cargo llvm-cov report --lcov --output-path coverage-host.lcov
691-
python3 scripts/ci-lcov-summary.py coverage-host.lcov | tee coverage-summary.txt
705+
cargo llvm-cov report --lcov --output-path "coverage-host-$TEST_SHARD.lcov"
706+
python3 scripts/ci-lcov-summary.py "coverage-host-$TEST_SHARD.lcov" | tee coverage-summary.txt
692707
693708
- name: Publish coverage summary to job summary
694709
if: needs.release-readiness.outputs.reuse != 'true'
695710
# Render the per-file LCOV line/function/branch totals as a fenced
696711
# block in the GitHub Actions job summary so reviewers don't
697712
# need to crack open the artifact for a quick look.
713+
env:
714+
TEST_SHARD: ${{ matrix.shard }}
698715
run: |
699716
{
700-
echo "## Host coverage summary"
717+
echo "## Host coverage summary (shard $TEST_SHARD)"
701718
echo ""
702-
echo "(In-process tests only. See coverage-merge for the"
703-
echo "full picture including docker-e2e binary coverage.)"
719+
echo "(This shard's in-process tests only. See coverage-merge"
720+
echo "for the full picture including docker-e2e binary coverage.)"
704721
echo ""
705722
echo '```'
706723
cat coverage-summary.txt
@@ -711,8 +728,8 @@ jobs:
711728
if: needs.release-readiness.outputs.reuse != 'true'
712729
uses: ./.github/actions/upload-artifact
713730
with:
714-
name: coverage-host
715-
path: coverage-host.lcov
731+
name: coverage-host-${{ matrix.shard }}
732+
path: coverage-host-${{ matrix.shard }}.lcov
716733
if-no-files-found: error
717734
retention-days: 30
718735

‎scripts/ci-test-shard.py‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env python3
2-
"""Run one shard of ci.yml's `test` or `test-release` job:
2+
"""Run one shard of ci.yml's `test`, `test-release` or `coverage` job:
33
`cargo test --workspace` split over `COUNT` runners. Each runner compiles
44
and executes only its assigned integration targets, shortening both the
55
debug and release-mode jobs without changing their compilation profiles.
@@ -14,6 +14,11 @@
1414
Those shards balance measured test time plus linking work, with shard 1's
1515
unit tests reserved up front. Other runners retain the count-based split.
1616
17+
CI_TEST_COVERAGE=1 runs the shard under cargo-llvm-cov for ci.yml's
18+
`coverage` job: `cargo llvm-cov --no-report` in place of `cargo test`, and
19+
no doctests, which cargo-llvm-cov leaves out without `--doctests`. A later
20+
`cargo llvm-cov report` exports the shard's profiles.
21+
1722
Extra arguments after `SHARD COUNT` go to every `cargo test` invocation.
1823
Each invocation runs with `--no-fail-fast`; the exit status is non-zero if
1924
any of them failed.
@@ -68,16 +73,18 @@ def cost(name):
6873
return shards
6974

7075

71-
def invocations(shard, count, names, extra=(), timings=None):
76+
def invocations(shard, count, names, extra=(), timings=None, coverage=False):
7277
"""The `cargo test` argument lists shard `shard` (1-based) runs."""
7378
if not 1 <= shard <= count:
7479
raise ValueError(f"shard {shard} not in 1..{count}")
75-
base = ["cargo", "test", "--workspace", "--no-fail-fast", *extra]
80+
runner = ["llvm-cov", "--no-report"] if coverage else ["test"]
81+
base = ["cargo", *runner, "--workspace", "--no-fail-fast", *extra]
7682
mine = partition(names, count, timings)[shard - 1]
7783
runs = []
7884
if shard == 1:
7985
runs.append(base + ["--lib", "--bins"] + [a for n in mine for a in ("--test", n)])
80-
runs.append(base + ["--doc"])
86+
if not coverage:
87+
runs.append(base + ["--doc"])
8188
elif mine:
8289
runs.append(base + [a for n in mine for a in ("--test", n)])
8390
return runs
@@ -94,10 +101,11 @@ def main(argv):
94101
names = integration_targets(metadata)
95102
timing_path = os.environ.get("CI_TEST_TIMINGS")
96103
timings = json.loads(Path(timing_path).read_text(encoding="utf-8")) if timing_path else None
104+
coverage = os.environ.get("CI_TEST_COVERAGE") == "1"
97105
print(f"ci-test-shard: shard {shard}/{count}: "
98106
f"{len(partition(names, count, timings)[shard - 1])} of {len(names)} integration targets", flush=True)
99107
status = 0
100-
for args in invocations(shard, count, names, argv[2:], timings):
108+
for args in invocations(shard, count, names, argv[2:], timings, coverage):
101109
print("+ " + " ".join(args), flush=True)
102110
if subprocess.run(args).returncode != 0:
103111
status = 1

‎scripts/tests/test_ci_test_shard.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,30 @@ def test_failed_unit_or_integration_run_still_runs_docs_and_fails_the_shard(self
7171
self.assertEqual(run.call_count, 3)
7272
self.assertIn("--doc", run.call_args_list[-1].args[0])
7373

74+
def test_coverage_shards_run_llvm_cov_without_doctests(self):
75+
# The unsharded `cargo llvm-cov --workspace` ran no doctests; the
76+
# shards must neither add them nor drop a unit or integration target.
77+
for count in (1, 2):
78+
runs = [args for k in range(1, count + 1)
79+
for args in shard.invocations(k, count, self.NAMES, coverage=True)]
80+
self.assertTrue(all(a[:5] == ["cargo", "llvm-cov", "--no-report", "--workspace", "--no-fail-fast"]
81+
for a in runs))
82+
self.assertFalse(any("--doc" in a for a in runs))
83+
self.assertEqual(sum("--lib" in a and "--bins" in a for a in runs), 1)
84+
self.assertEqual(sorted(selected(runs)), self.NAMES)
85+
86+
def test_coverage_mode_comes_from_the_environment(self):
87+
metadata = {"workspace_members": ["a"], "packages": [
88+
{"id": "a", "targets": [{"name": "integration", "kind": ["test"]}]}]}
89+
with patch.dict(shard.os.environ, {"CI_TEST_COVERAGE": "1"}), \
90+
patch.object(shard.subprocess, "run", side_effect=[
91+
subprocess.CompletedProcess([], 0, stdout=json.dumps(metadata)),
92+
subprocess.CompletedProcess([], 0),
93+
]) as run:
94+
self.assertEqual(shard.main(["1", "2"]), 0)
95+
self.assertEqual(run.call_count, 2)
96+
self.assertEqual(run.call_args_list[-1].args[0][:3], ["cargo", "llvm-cov", "--no-report"])
97+
7498
def test_negative_bad_shard(self):
7599
with self.assertRaises(ValueError):
76100
shard.invocations(3, 2, self.NAMES)
@@ -134,6 +158,24 @@ def test_all_release_shards_are_required_and_use_the_matrix_size(self):
134158
self.assertIn("if: always()", verdict)
135159
self.assertIn('if v["result"] not in ("success", "skipped")', verdict)
136160

161+
def test_all_coverage_shards_are_required_and_upload_distinct_lcov(self):
162+
# coverage is the gating Linux test run: every shard must run under
163+
# cargo-llvm-cov, gate ci-ok, and upload an LCOV coverage-merge picks
164+
# up (its `coverage-{host*,docker-*}` pattern) without name clashes.
165+
workflow = (ROOT / ".github/workflows/ci.yml").read_text(encoding="utf-8")
166+
cov = workflow.split("\n coverage:\n")[1].split("\n docker-base:\n")[0]
167+
count = json.loads(re.search(r"^ shard: (\[.*\])$", cov, re.M)[1])
168+
self.assertEqual(count, list(range(1, len(count) + 1)))
169+
self.assertIn("fail-fast: false", cov)
170+
self.assertIn("CI_TEST_COVERAGE: '1'", cov)
171+
self.assertIn('python3 scripts/ci-test-shard.py "$TEST_SHARD" "$TEST_SHARD_COUNT"', cov)
172+
self.assertIn("TEST_SHARD_COUNT: ${{ strategy.job-total }}", cov)
173+
self.assertIn('--output-path "coverage-host-$TEST_SHARD.lcov"', cov)
174+
self.assertIn("name: coverage-host-${{ matrix.shard }}", cov)
175+
self.assertIn("pattern: coverage-{host*,docker-*}", workflow)
176+
verdict = workflow.split("\n ci-ok:\n")[1]
177+
self.assertIn("coverage", re.search(r"^ needs: \[(.*)\]$", verdict, re.M)[1].split(", "))
178+
137179

138180
@unittest.skipUnless(shutil.which("cargo"), "cargo not available")
139181
class CargoSelection(unittest.TestCase):

0 commit comments

Comments
 (0)