Skip to content

test: cut the unit tier's slowest waits - #576

Merged
andrewpark-scaleai merged 1 commit into
mainfrom
andrew-park/ci-faster-unit-tests
Nov 9, 2026
Merged

andrewpark-scaleai merged 1 commit into
mainfrom
andrew-park/ci-faster-unit-tests

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Why

The goal is a faster unit tier that still fails on the changes it exists to catch. The slowest unit tests in run 42679706897 spent their time on three waits that test nothing.

Test Time What it waited on
test_changelog.py::test_enable_installs_the_capture_after_both_... and two turn-ordering tests 57s, 20s, 18s a GNU tar baseline of the runner's $HOME, /usr/local and /etc
test_disk_recipe.py::test_the_launcher_refuses_a_host_port_..., both cases 30.0s each a socat the refused launch left running
test_optional_extras_stay_optional.py, about 40 cases 12 to 28s each the same dependency closures, recomputed in every test

What changes

  • The launcher validates every port before it starts a proxy. Before this change, launch started each guest port's socat and then checked VM_HOST_PORTS. A refused launch exited 2 with those proxies still running, because the EXIT trap that stops them is set after the loops. Now both checks run first, then the proxies start. The two refusal tests also assert that no socat started. The 30s was the test's stand-in socat holding stderr open.
  • _closure is cached. Each _only_in call parsed every distribution's metadata 39 times. Installed metadata can't change during a run, so _closure now returns a frozenset under functools.cache.
  • The changelog client's default roots are the test's own folders. _changelog_client sets the agent's roots to the test's CLI state and workspace. The real defaults are covered by test_the_default_roots_hold_the_workspace_and_the_clis_state, which this change leaves alone. macOS tar has no --listed-incremental, so the baseline failed at once there, and the tests were only slow on Linux.

Verification

Each changed test still fails on the change it guards:

  • The new refusal assertion fails against the old launch (2 failed, after 61s of waiting).
  • Dropping PostToolUseFailure from _CAPTURE_EVENTS fails test_enable_installs_the_capture_after_both_....
  • A top-level import boto3 in agent_env.config.errors fails 7 of the optional-extras tests.

Timings, on a 14-core Mac with -n 8:

  • test_optional_extras_stay_optional.py takes 18 to 28s on this branch and 38s on main. That's 94s of CPU against 167s.
  • The launcher tests pass in 14s. The two refusal cases no longer appear among the slowest.
  • tst/unit/agents/claude_code/ passes: 244 passed, and 30 skipped as Linux-only.

The changelog speedup only shows on Linux, so this PR's unit shards are the measurement.

Editing launch changes the agentenv-qemu-launcher image's source tag. The disk images build from the rootfs and disk tools alone, so they keep their tags.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

This PR removes repeated work and unnecessary waits from unit tests.

  • The launcher checks port conflicts before it starts proxies.
  • Optional-extra checks reuse each dependency closure.
  • Changelog tests capture only the CLI state and workspace.

No actionable issues found. Tests were not run during this review.

Reviews (1) · Last reviewed commit: "test: cut the unit tier's slowest waits" · Reviewed by Greptile

- The launcher validated the host ports only after it had started a guest port's proxy, so a refused
  launch exited 2 and left that socat running. It now checks every port before it starts any proxy, and
  the refusal tests assert nothing started. They waited 30s each on the stand-in socat holding stderr.
- The optional-extras checks recomputed each extra's dependency closure from installed metadata 39 times
  a test. The closure is cached, as the installed metadata can't change during a run.
- The changelog client's enable baselined the runner's home, /usr/local and /etc with GNU tar. Its roots
  are now the test's own workspace and CLI state.

Co-Authored-By: Claude <noreply@anthropic.com>
@andrewpark-scaleai
andrewpark-scaleai requested a review from a team as a code owner November 9, 2026 21:05
@andrewpark-scaleai
andrewpark-scaleai merged commit 55bfc1a into main Nov 9, 2026
34 checks passed
@andrewpark-scaleai
andrewpark-scaleai deleted the andrew-park/ci-faster-unit-tests branch November 9, 2026 22:51
andrewpark-scaleai added a commit that referenced this pull request Nov 10, 2026
…image in the mirror

The slow tier's times are from PR #580's run 42716509774, attempt 2, which built no image; the installer's from
the same run; the gateway's and unit tier's from main's run 42715765775, after #575 and #576.

Co-Authored-By: Claude <noreply@anthropic.com>
andrewpark-scaleai added a commit that referenced this pull request Nov 10, 2026
…image in the mirror

The slow tier's times are from PR #580's run 42716509774, attempt 2, which built no image; the installer's from
the same run; the gateway's and unit tier's from main's run 42715765775, after #575 and #576.

Co-Authored-By: Claude <noreply@anthropic.com>
kaihackney-scale added a commit that referenced this pull request Nov 10, 2026
Keeps main's cached _closure (#576) alongside this branch's pruned argument.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andrewpark-scaleai added a commit that referenced this pull request Nov 10, 2026
…image in the mirror

The slow tier's times are from PR #580's run 42716509774, attempt 2, which built no image; the installer's from
the same run; the gateway's and unit tier's from main's run 42715765775, after #575 and #576.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants