Skip to content

ci: parallelize the fast integration tier and refresh the shard durations - #570

Merged
andrewpark-scaleai merged 2 commits into
mainfrom
andrew-park/ci-rebalance-shards
Nov 9, 2026
Merged

andrewpark-scaleai merged 2 commits into
mainfrom
andrew-park/ci-rebalance-shards

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What and why

The goal is shorter Local backends runs. That workflow takes 16 to 18 minutes, while every other workflow takes under 5. This PR makes two small changes.

  • The fast integration tier runs on every core. integration-local ran its 181 tests serially, which took 585s in run 42675210519. It now uses -n auto --dist loadscope, the same flags as make int-test-fast. loadscope keeps a module's tests on one worker, so module fixtures still run once. The serial setting came with the initial import, and no commit gives a reason for it.
  • The durations table is refreshed. Two slow files were missing from tst/integration/slow_durations.json. They were ca_trust_test.py (270s on main) and playwright_mcp_local_test.py (89s). The balancer gave each the median weight, so they landed on shards at random. I rebuilt the table with .github/scripts/slow_test_durations.py from main's run 42667684447. I added the installer reports from run 42675261871, because main skipped that tier and a rewrite without them would drop its 59 entries.

What this does not fix

The refresh does not reliably shorten the slow tier. I assigned the shards with each table and timed the result against run 42675210519, which the new table was not built from. The slowest shard came out at 875s with the old table and 908s with the new one. Between that run and main, computer tests changed by 60 to 114s each. Four of that run's slow shards missed the computer-image cache, because the repo's Actions cache is at 10.7GB, which is over the 10GB limit, and its 12 per-shard entries evict each other. The fix for that is a shared image cache, which is a separate change.

On the same check, the unit tier's heavier shard drops from 1395s to 1293s of test time, with the two shards within 14s of each other.

How it was tested

  • I simulated shard assignment by calling tst/util/shards.py select() directly with the old and new tables over the junit times from runs 42667684447 and 42675210519.
  • This PR's own Local backends run is the test of the fast tier under xdist.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

The PR runs fast integration tests with -n auto --dist loadscope and refreshes the times used to balance CI shards.

  • The fast integration job runs tests across available runner cores.

andrewpark-scaleai explicitly acknowledges that the refresh does not reliably shorten the slow tier and defers the shared image-cache fix to a separate change.

Reviews (1) · Last reviewed commit: "ci: run the fast integration tier on eve..." · Reviewed by Greptile

andrewpark-scaleai and others added 2 commits November 9, 2026 11:03
Two slow files, ca_trust_test.py (270s) and playwright_mcp_local_test.py (89s), were missing from the table, so each weighed the median and landed on a shard at random. The installer tier's times come from run 42675261871, since main skipped that tier.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
It ran serially, at 585s for 181 tests. make int-test-fast already runs it with -n auto --dist loadscope.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@andrewpark-scaleai
andrewpark-scaleai requested a review from a team as a code owner November 9, 2026 19:03
@andrewpark-scaleai
andrewpark-scaleai enabled auto-merge (squash) November 9, 2026 20:50
@andrewpark-scaleai
andrewpark-scaleai merged commit 36151aa into main Nov 9, 2026
35 checks passed
@andrewpark-scaleai
andrewpark-scaleai deleted the andrew-park/ci-rebalance-shards branch November 9, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants