Repository navigation
ci: share the slow tier's computer images through a mirror registry rather than per-shard caches - #580
Conversation
3f301be to
d2329d8
Compare
d2329d8 to
2c89b45
Compare
5d957be to
c7f3ced
Compare
Review summaryThe PR works, and the tests are good. I found no bug. My concern is the size of the change against the gain. The gain is small
Simpler options
Other
Proposed orderLand the mirror and the composer copy. Drop the prune job for now. Do the durations refresh and the shard grouping in a separate PR, and compare the slowest shard time. |
…ather than per-shard caches The slow tier saved each runner's whole registry to the Actions cache, keyed by shard and week. The same image landed on up to three shards, so the entries (1.6 to 6.3GB each) overflowed the repository's 10GB, evicted each other, and most shards built every image from scratch. - The composer copies an image of a key the env's image store lacks from AGENT_ENV_COMPUTER_IMAGE_MIRROR, read anonymously, before it would build it; a mirror it can't read leaves the image to be built. - registry_api.copy_image copies an image between registries by its digest, sending only the blobs the target lacks and each manifest byte for byte. - After its tests pass on main, each slow shard shares the images it built or copied to ghcr.io/scaleapi/agentenv-ci-computers under their keys and <key>-seen-<date>. - computer-image-mirror.yml deletes, daily, the versions no run has used for 7 days, keeping a kept index's platform manifests. - The test packages folder keeps its own per-shard weekly cache, saved only on main and named by its locks. prune_registry.py, which trimmed the registry cache, goes with that cache. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…fore its credentials go Co-Authored-By: Claude <noreply@anthropic.com>
The mirror package is public, so GHCR stores it for free, and the prune needed a package-admin right the repository doesn't hold. Removing it removes the daily workflow, its script and tests, and the dated seen tag each share wrote only so the prune could tell used images from unused ones. Sharing and the composer's copy are unchanged.
The runner's registry has no computer repository when the shard composed no image, so listing its tags answers 404. The share script now treats that as nothing to share. It also drops comments that restated the step names or the composer's lookup.
c7f3ced to
f7e3eee
Compare
|
Thanks, Alex. I took the proposed order.
This branch's CI can't show mirror hits. Sharing runs only on main, and a composer change gives a new mirror prefix. The first main runs after merge are the measurement, and I'll post them here. The diff is now 13 files, +438/−260, down from 16 files, +1194/−820. Playbook check: deslop ✅ · no-comments ✅ (5 comments removed) · independent blind review ✅ (1 latent bug fixed, no blockers) · unit tests ✅ (mutation-checked) · CI ⏳ |
Why
The goal is a shorter Local backends run on the same 12 slow-tier runners. The run finishes when the slowest slow shard does, and that shard is usually rebuilding computer images another shard already built.
computers-<shard>-<week>-<tags>.What changes
The composer copies an image it would build from a mirror.
_storedlooks for a key's image in the env's image store, as before.AGENT_ENV_COMPUTER_IMAGE_MIRRORis set and the store lacks the image, the composer reads the image anonymously from the mirror and copies it into the store. It then reuses the image as if it had been built earlier.<composer digest>-<key>. The composer digest iscomposition(), which already covers the composer, the builder, the computers' build files and the protocol's computer package. So a change to the code that builds images misses the mirror and builds them again, and the build path stays tested.registry_api.copy_imagecopies an image between registries by its digest.Content-Length. They are never held in memory.The slow tier shares images through
ghcr.io/scaleapi/agentenv-ci-computers.share_computer_images.pydoes this asGITHUB_TOKEN, and sends no blob the mirror already holds. If anonymous reads fail, it warns that the package needs to be public.actions/cacherestore, the "keep only the computer images" step, and the save of the registry are gone.prune_registry.pyand its test, which only trimmed the registry cache, are removed.No prune. The package is public, so GHCR stores it for free. If storage ever needs trimming, a prune can delete every version whose tag doesn't start with the current
composition()digest, which needs no extra tags.Why
copy_imageand notskopeo copy --all. The composer copies an image in-process, while it composes an env. That runs on developers' machines as well as in CI, andskopeoisn't a dependency of the library. The share script reuses the same function, so the copy has one implementation.The durations table is in #622. The refresh of
slow_durations.jsonmoved there, so this diff holds only the mirror.Before merge
agentenv-ci-computers, and it's already readable without credentials.Measured on this PR
Run 42716509774 was measured twice. Attempt 1 started with an empty mirror and built everything. Attempt 2 found every image in the mirror.
Main's slowest slow shard has a median of 940s over its last 46 runs (800s to 1198s). Those runs mixed cache hits and misses from shard to shard. With the mirror, every shard reuses every image whose sources are unchanged, so a run stops depending on which cache entries survived. Shard 8 is still the slowest at 865s of pytest, against the 596s the durations table plans for it. Refreshing the table from a run with every image in the mirror is #622's job.
Verification
copy_imagesends only missing blobs and keeps each manifest's bytes.copy_imageresend blobs the target has.copy_imagecopied the published 2-platformagentenv-db-webindex from GHCR into a localregistry:2with the same digest, in 10.6s. A second copy, with the blobs already there, took 0.6s.share_computer_images.pybetween two local registries pushed each key and skippedindex-*tags.

Confidence Score: 5/5
The PR appears safe to merge, with no new findings or remaining blockers from the earlier threads.Summary
The PR replaces per-shard image caches with a shared GHCR mirror. The latest change names the existing tag-prefix length
MIRROR_PREFIX_LENGTHwithout changing its value.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Composer needs an image] --> B{Image store has it?} B -->|Yes| E[Reuse image] B -->|No| C{Mirror serves it?} C -->|Yes| D[Copy into image store] D --> E C -->|Missing or copy fails| F[Build image] E --> G[Run tests] F --> G G --> H[Passing main shards share images]Reviews (11) · Last reviewed commit: "refactor(composer): name the mirror tag'..." · Reviewed by Greptile