Skip to content

ci: share the slow tier's computer images through a mirror registry rather than per-shard caches - #580

Merged
andrewpark-scaleai merged 6 commits into
mainfrom
andrew-park/ci-shared-computer-images
Nov 13, 2026
Merged

andrewpark-scaleai merged 6 commits into
mainfrom
andrew-park/ci-shared-computer-images

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Why

The goal is a shorter Local backends run on the same 12 slow-tier runners. The run finishes when the slowest slow shard does, and that shard is usually rebuilding computer images another shard already built.

  • Each shard saved its whole local registry to the Actions cache, keyed computers-<shard>-<week>-<tags>.
  • In main run 42701339058 the tests used 15 distinct computer images, and those images were stored 25 times across 10 shards. One image sat on shards 3, 6 and 7. Four sat on both 8 and 12.
  • Entries were 1.6 to 6.3GB each, against the repository's 10GB cache. They evicted each other, so only two shards' entries survived at a time, and every Monday the week in the key reset them all.
  • In PR runs 42699224484 and 42699237630, shard 8 found no cache both times. It ran 1065s and 1126s of pytest, against the 596s the shard balancer planned. Every other shard ran within 0.94x to 1.22x of its plan.

What changes

The composer copies an image it would build from a mirror.

  • _stored looks for a key's image in the env's image store, as before.
  • When AGENT_ENV_COMPUTER_IMAGE_MIRROR is set and the store lacks the image, the composer reads the image anonymously from the mirror and copies it into the store. It then reuses the image as if it had been built earlier.
  • The mirror tag is <composer digest>-<key>. The composer digest is composition(), which already covers the composer, the builder, the computers' build files and the protocol's computer package. So a change to the code that builds images misses the mirror and builds them again, and the build path stays tested.
  • A mirror that lacks the key, refuses, or fails mid-copy leaves the image to be built, with a warning. The setting is empty by default, so nothing changes outside CI.
  • The key is a digest of everything the image is built from, so a mirror image under a key is the image a build of that key makes.

registry_api.copy_image copies an image between registries by its digest.

  • It sends only the blobs the target lacks, then each manifest byte for byte, so the copy keeps the source's digest. An index's platform manifests go first.
  • A held token that the registry refuses is asked for once more. GHCR scopes its token to the first challenge, which is a pull's, so a push needs a second one.
  • Blobs stream through a temporary file with a Content-Length. They are never held in memory.

The slow tier shares images through ghcr.io/scaleapi/agentenv-ci-computers.

  • Every slow shard reads the mirror.
  • Once a shard's tests pass on main, it shares each image it built or copied, under its mirror tag. share_computer_images.py does this as GITHUB_TOKEN, and sends no blob the mirror already holds. If anonymous reads fail, it warns that the package needs to be public.
  • The job's checkout no longer persists its token, so only the share step, which runs on main, gets one. A failed share doesn't fail the job.
  • The actions/cache restore, the "keep only the computer images" step, and the save of the registry are gone.
  • The test packages folder from ci(slow tier): keep the computer tests' resolved packages between modules and runs #591 keeps its own per-shard weekly cache, saved only on main and named by the locks it holds. prune_registry.py and its test, which only trimmed the registry cache, are removed.

No prune. The package is public, so GHCR stores it for free. If storage ever needs trimming, a prune can delete every version whose tag doesn't start with the current composition() digest, which needs no extra tags.

Why copy_image and not skopeo copy --all. The composer copies an image in-process, while it composes an env. That runs on developers' machines as well as in CI, and skopeo isn't a dependency of the library. The share script reuses the same function, so the copy has one implementation.

The durations table is in #622. The refresh of slow_durations.json moved there, so this diff holds only the mirror.

Before merge

  • The package must stay public. Shards read the mirror anonymously. The first run created agentenv-ci-computers, and it's already readable without credentials.
  • Main-only sharing. This PR's runs pushed to the mirror once so a rerun could show hits. Sharing is now main-only again.

Measured on this PR

Run 42716509774 was measured twice. Attempt 1 started with an empty mirror and built everything. Attempt 2 found every image in the mirror.

Attempt 1, empty mirror Attempt 2, every image in the mirror
Image builds the composer logged, across the 12 shards 72 0
Copies from the mirror it logged 0 50
Slowest shard, job time 1238s (shard 8) 941s (shard 8)
Sum of the 12 shards' pytest time 8882s 7773s

Main's slowest slow shard has a median of 940s over its last 46 runs (800s to 1198s). Those runs mixed cache hits and misses from shard to shard. With the mirror, every shard reuses every image whose sources are unchanged, so a run stops depending on which cache entries survived. Shard 8 is still the slowest at 865s of pytest, against the 596s the durations table plans for it. Refreshing the table from a run with every image in the mirror is #622's job.

Verification

  • Unit tests.
    • The composer copies a missing key from the mirror and reuses it rather than building it.
    • A mirror that refuses leaves the image to be built.
    • copy_image sends only missing blobs and keeps each manifest's bytes.
    • A change to the composer's code misses the mirror and builds.
    • A copy to GHCR asks again for a push token after its pull token is refused.
    • The share step shares each key, skips index tags, shares nothing when the shard composed no image, and warns when the mirror can't be read anonymously.
  • Mutation checks. Each of these fails its test:
    • having the composer ignore the mirror;
    • the old retry rule, which kept a pull-only token for a push to GHCR;
    • having copy_image resend blobs the target has.
  • Real registries.
    • copy_image copied the published 2-platform agentenv-db-web index from GHCR into a local registry:2 with the same digest, in 10.6s. A second copy, with the blobs already there, took 0.6s.
    • share_computer_images.py between two local registries pushed each key and skipped index-* tags.
  • This PR's CI can't show mirror hits. Sharing runs only on main, and the mirror holds images under an older composer digest, so this branch's runs build every image. The first main runs after merge share the images, and the runs after those are the measurement.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge, with no new findings or remaining blockers from the earlier threads.

Summary

The PR replaces per-shard image caches with a shared GHCR mirror. The latest change names the existing tag-prefix length MIRROR_PREFIX_LENGTH without changing its value.

  • The composer reuses matching computer images from a configured mirror.
  • Registry images can move between stores without changing their digest.
  • Slow-tier shards reuse and share computer images through one GHCR mirror.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Composer needs an image] --> B{Image store has it?}
  B -->|Yes| E[Reuse image]
  B -->|No| C{Mirror serves it?}
  C -->|Yes| D[Copy into image store]
  D --> E
  C -->|Missing or copy fails| F[Build image]
  E --> G[Run tests]
  F --> G
  G --> H[Passing main shards share images]
Loading

Reviews (11) · Last reviewed commit: "refactor(composer): name the mirror tag'..." · Reviewed by Greptile

@andrewpark-scaleai
andrewpark-scaleai requested a review from a team as a code owner November 9, 2026 22:58
andrewpark-scaleai added a commit that referenced this pull request Nov 10, 2026
…image in the mirror

The slow tier's times are from PR #580's run 42716509774, attempt 2, which built no image; the installer's from
the same run; the gateway's and unit tier's from main's run 42715765775, after #575 and #576.

Co-Authored-By: Claude <noreply@anthropic.com>
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/ci-shared-computer-images branch from 3f301be to d2329d8 Compare November 10, 2026 00:19
andrewpark-scaleai added a commit that referenced this pull request Nov 10, 2026
…image in the mirror

The slow tier's times are from PR #580's run 42716509774, attempt 2, which built no image; the installer's from
the same run; the gateway's and unit tier's from main's run 42715765775, after #575 and #576.

Co-Authored-By: Claude <noreply@anthropic.com>
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/ci-shared-computer-images branch from d2329d8 to 2c89b45 Compare November 10, 2026 00:50
andrewpark-scaleai added a commit that referenced this pull request Nov 10, 2026
…image in the mirror

The slow tier's times are from PR #580's run 42716509774, attempt 2, which built no image; the installer's from
the same run; the gateway's and unit tier's from main's run 42715765775, after #575 and #576.

Co-Authored-By: Claude <noreply@anthropic.com>
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/ci-shared-computer-images branch from 5d957be to c7f3ced Compare November 10, 2026 23:09
@andrewpark-scaleai
andrewpark-scaleai enabled auto-merge (squash) November 11, 2026 01:01
@AlexS-scale

Copy link
Copy Markdown
Contributor

Review summary

The PR works, and the tests are good. I found no bug. My concern is the size of the change against the gain.

The gain is small

  • Attempt 2 (full mirror): the slowest shard took 941s.
  • Main's median for the slowest shard, over 46 runs, is 940s.
  • The mirror cuts the slow tail (800s to 1198s). It does not cut the typical run.
  • Shard 8 still runs 865s of pytest. The plan says 596s. The durations table is out of date, and the mirror does not fix that.

Simpler options

  1. Do not add the prune job now.

    • The repository is public, and public GHCR storage is free.
    • The prune needs a package-admin right that is not yet granted.
    • This removes prune_computer_images.py, its workflow, its tests, and the seen-tag scheme (about 250 lines).
    • Speed is the same, because the mirror is the same.
    • If you need a prune later, delete the versions whose tag does not start with the current composition() prefix. That needs no seen tags.
  2. Fix the shard plan.

    • Refresh slow_durations.json from a full-mirror run. The PR already lists this as the next step.
    • Put tests that share an image on the same shard. Today 15 images are stored 25 times across 10 shards.
    • I did not measure this. It could give a shorter slowest shard than the mirror does, because shard 8 is 270s over its plan.
    • Please test it with one run before you decide.
  3. Use skopeo copy --all instead of the new copy_image code.

    • copy_image, has_blob, upload and the _send changes add about 75 lines of push code with its own token and redirect cases.
    • skopeo does the same copy. Check that it is on the runner image.
    • Question: why extend the httpx client? If the reason is that the copy must run inside the composer, say so in the PR.

Other

  • Move the slow_durations.json change (about 560 lines) to its own PR. It hides the real change.

Proposed order

Land the mirror and the composer copy. Drop the prune job for now. Do the durations refresh and the shard grouping in a separate PR, and compare the slowest shard time.

andrewpark-scaleai and others added 5 commits November 13, 2026 01:38
…ather than per-shard caches

The slow tier saved each runner's whole registry to the Actions cache, keyed by shard and week. The same
image landed on up to three shards, so the entries (1.6 to 6.3GB each) overflowed the repository's 10GB,
evicted each other, and most shards built every image from scratch.

- The composer copies an image of a key the env's image store lacks from AGENT_ENV_COMPUTER_IMAGE_MIRROR,
  read anonymously, before it would build it; a mirror it can't read leaves the image to be built.
- registry_api.copy_image copies an image between registries by its digest, sending only the blobs the
  target lacks and each manifest byte for byte.
- After its tests pass on main, each slow shard shares the images it built or copied to
  ghcr.io/scaleapi/agentenv-ci-computers under their keys and <key>-seen-<date>.
- computer-image-mirror.yml deletes, daily, the versions no run has used for 7 days, keeping a kept
  index's platform manifests.
- The test packages folder keeps its own per-shard weekly cache, saved only on main and named by its locks.
  prune_registry.py, which trimmed the registry cache, goes with that cache.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…fore its credentials go

Co-Authored-By: Claude <noreply@anthropic.com>
The mirror package is public, so GHCR stores it for free, and the prune
needed a package-admin right the repository doesn't hold. Removing it
removes the daily workflow, its script and tests, and the dated seen tag
each share wrote only so the prune could tell used images from unused
ones. Sharing and the composer's copy are unchanged.
The runner's registry has no computer repository when the shard composed
no image, so listing its tags answers 404. The share script now treats
that as nothing to share. It also drops comments that restated the step
names or the composer's lookup.
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/ci-shared-computer-images branch from c7f3ced to f7e3eee Compare November 13, 2026 02:00
@andrewpark-scaleai

Copy link
Copy Markdown
Contributor Author

Thanks, Alex. I took the proposed order.

  • Prune dropped (db51841). The workflow, prune_computer_images.py, its tests and the -seen- tags are gone. The PR no longer needs the package-admin grant. Your prefix rule is in the description in case a prune is ever needed.
  • Durations refresh moved out. The branch is rebased on main without that commit. test: split browser_computer_test's multi-env tests out and refresh the shard table #622 refreshes slow_durations.json and splits the slowest file. Grouping the tests that share an image onto one shard will be its own PR, checked in the shard simulator before a CI run.
  • copy_image stays. The composer copies in-process while it composes an env (composer.py _stored). That runs on developers' machines too, and skopeo isn't a dependency of the library. The share script reuses the same function, so there's one copy path. The description now says this.
  • On the gain, you're right. The full-mirror run's slowest shard took 941s against main's 940s median. The case for the PR is a steadier slowest shard, not a faster typical one. Since ci: run the slow tier's tests two at a time on each shard #600, main's slowest slow shard has a median of 1155s over its last 20 green runs (1000s to 1308s).
  • One fix from review (f7e3eee). The share step no longer fails on a shard that composed no image.

This branch's CI can't show mirror hits. Sharing runs only on main, and a composer change gives a new mirror prefix. The first main runs after merge are the measurement, and I'll post them here.

The diff is now 13 files, +438/−260, down from 16 files, +1194/−820.

Playbook check: deslop ✅ · no-comments ✅ (5 comments removed) · independent blind review ✅ (1 latent bug fixed, no blockers) · unit tests ✅ (mutation-checked) · CI ⏳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants