Skip to content

Keep the Docker image's toolchain layers stable across releases, and slim them - #439

Open
lelia wants to merge 7 commits into
mainfrom
lelia/ce-430-docker-layers
Open

lelia wants to merge 7 commits into
mainfrom
lelia/ce-430-docker-layers

Conversation

@lelia

@lelia lelia commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Every release currently rebuilds every layer of the Docker image, so upgrading socketdev/cli downloads nearly the whole image. This PR fixes that and removes about 1.1 GB of payload nothing runs.

Root cause of the per-release re-pull

ARG CLI_VERSION (with the other release ARGs) was declared at the top of the Dockerfile. BuildKit cache-keys every RUN after an ARG on that ARG's value, and CLI_VERSION changes on every release, so every toolchain layer missed the cache on every release build even when nothing in it changed. From the published amd64 manifests:

Upgrade Layers reused New compressed bytes
2.11.5 → 2.11.6 8 of 27 1,025 MB of 1,046
2.11.10 → 2.11.11 8 of 27 1,028 MB of 1,049

The base image digest was the same across 2.11.10 → 2.11.11, so the churn didn't come from upstream. The release ARGs are now declared just before the CLI install. tests/unit/test_dockerfile.py lists the ARGs allowed above the toolchain layers (GO_VERSION, JAVA_VERSION, DOTNET_VERSION, TARGETARCH, PIP_NO_CACHE_DIR) and fails on any other, so a new per-release ARG is caught as well. It fails against main's Dockerfile. Dockerfile is added to the Unit Tests push paths, since the suite now reads it.

Release build cache

Moving the ARGs only helps if a release can read the previous release's cache, and it can't today. release.yml caches with type=gha. Actions caches are scoped to the ref that wrote them, and each release builds on its own tag: the repository's BuildKit cache entries all sit under refs/tags/v2.11.x, a full set per release, none ever read by the next one. That is also why the Actions cache is pinned at its 10 GB limit. The release build now uses a registry cache at socketdev/cli:buildcache (mode=max), which any tag can read.

I checked the mechanism with a local registry and a multi-platform push. With the registry cache, two builds with different CLI_VERSION values produced identical layer digests for a RUN step that writes a timestamp, so the step was reused rather than re-run. An inline cache (cache-to: type=inline with cache-from pointing at the published image) did not hit after a multi-platform push, which is why this uses a dedicated cache tag. The cache tag is pushed to the same Docker Hub repository and isn't meant to be pulled.

Reachability inside the image (e2e-image.yml)

e2e-test.yml runs reachability on the runner, so nothing checked that the image's toolchains actually work for Coana. The new workflow builds the image once from the commit (reading the release cache, never writing it) and runs --reach inside it against the existing npm fixture and a new Rust fixture (smallvec 1.6.0, which has a known advisory). Each fixture's result is recorded, so one failure doesn't hide the other. It runs when the Dockerfile, the reachability launcher, or the e2e fixtures change, on push to main, and on demand. It isn't a required check.

Both e2e workflows now share one policy for the backend's intermittent empty results:

  • tests/e2e/run-with-retries.sh is the retry loop: up to three attempts while a probe fails, with retries surfaced as warnings. e2e-test.yml now calls it instead of its inline loop.
  • tests/e2e/reach_facts.py is the probe, and the one definition of the inconclusive backend result (a manifest uploaded, components orphaned, nothing to analyze). It replaces reach-facts-probe.sh, and validate-reachability.sh calls it instead of its own greps.

The image check asserts only what the image controls: the run completes, and Coana reaches a reachable or unreachable verdict for at least one vulnerability, which needs the ecosystem's toolchain to work. It doesn't assert which verdict.

Dependabot and Node majors

.github/dependabot.yml now ignores major updates to the node image. Dependabot doesn't know Node's release schedule, so it proposes odd-numbered majors, which never become LTS, and it can't check the reachability engine's minimum Node version. The floating node:24-alpine3.24 tag already picks up Node 24 minor and patch releases on every build. A Node major becomes a deliberate change, checked with e2e-image.yml.

Payload removed

  • Rust: rustup's minimal profile (rustc, cargo, rust-std) instead of default. Drops about 900 MB of offline HTML docs, plus rustfmt and clippy. Reachability only invokes cargo.
  • Caches: npm cache clean --force after the global install, and ARG PIP_NO_CACHE_DIR=1 so no pip install leaves a cache. As an ARG it applies only at build time, and its value never changes, so it doesn't affect layer caching.
  • apk update is gone; apk add --no-cache doesn't need it.
  • strace is no longer installed (it was referenced nowhere). The changelog notes apk add strace for anyone who relied on it.
  • .dockerignore: lists what the build context includes (package source, build metadata, entrypoint, and dist/ for Dockerfile.preview's wheel) instead of what it excludes, so new top-level directories stay out by default.

Verification

CI's dockerfile-smoke job builds this Dockerfile natively on amd64 and arm64 and runs its tool and TLS checks. The measurements below are from local linux/arm64 builds.

  • Size: 4,579 MB for socketdev/cli:2.11.11 vs 3,446 MB for this branch (uncompressed, arm64). /root/.rustup went from 1,693 to 753 MB, /root/.npm from 20 to 1 MB, /app from 11 to 2 MB.
  • Layer reuse: built with CLI_VERSION=2.11.10, then rebuilt with CLI_VERSION=2.11.11. Steps 2–17, everything through the locked dependency install, were CACHED. Only the CLI install and the small layers after it rebuilt. A real release also changes pyproject.toml/uv.lock, which adds the locked-dependency layer (about 21 MB). So an upgrade is roughly 25 MB uncompressed instead of about 1 GB compressed.
  • Toolchains: socketcli, cargo/rustc, go, java, dotnet, node, ruby/bundle, pyenv, uv and socket all run. cargo generate-lockfile, cargo metadata and cargo build work on a crate with a crates.io dependency.
  • Tests: 734 unit tests pass.

Rust reachability in the image is covered by e2e-image.yml on this PR. A throwaway run with node:26-alpine3.24 (#444, closed) also passed both fixtures, for when Node 26 is adopted.

Once the registry cache is in use, the old type=gha BuildKit entries (about 10 GB) can be deleted from the repository's Actions caches.

Bumps to 2.12.0, a minor version because the image drops binaries (strace, rustfmt, clippy) and changes how it is built and published. #438 also targets 2.12.0; whichever merges second rides that version, and needs a small merge in the changelog and in release.yml's build step.

Fixes CE-430

🤖 Generated with Claude Code


Note

Medium Risk
Changes the default Docker toolchain (minimal Rust, no strace) and layer layout that all CLI container users depend on; Rust reachability in the image was not fully re-verified in this PR.

Overview
Release 2.11.12 reshapes how the published socketdev/cli image is built so version bumps reuse cached toolchain layers instead of forcing a near-full re-download, and trims roughly a gigabyte of unused payload.

Layer caching: CLI_VERSION and the other release-specific build args move to just before the CLI install step. BuildKit invalidates every subsequent RUN when those args change; having them at the top previously busted cache for Go, Java, Rust, pyenv, locked deps, etc. on every release. A new unit test in test_dockerfile.py asserts release ARGs stay below toolchain installs, and Dockerfile is included in the CI paths filter on push.

Smaller image: Rust is installed with rustup’s minimal profile (no offline docs, rustfmt, or clippy); strace is dropped from the base packages; npm and pip caches are cleared or use --no-cache-dir; redundant apk update is removed. A new .dockerignore keeps tests, docs, and similar out of the build context while still allowing preview wheels from dist/.

Reviewed by Cursor Bugbot for commit 29a60c9. Configure here.

…slim them

CLI_VERSION was declared at the top of the Dockerfile. Every RUN after an ARG
is cache-keyed on its value, and CLI_VERSION changes on every release, so each
release rebuilt every toolchain layer even when nothing in them changed. Between
2.11.10 and 2.11.11, 1,028 MB of the 1,049 MB compressed amd64 image was new
layers, so every upgrade was close to a full pull. The release ARGs are now
declared just before the CLI install, and a unit test keeps them there.

The image also drops payload nothing runs:
- rustup's minimal profile replaces the default one, which added offline HTML
  docs, rustfmt and clippy. Reachability runs only cargo.
- npm and pip caches are no longer left in the image.
- strace is no longer installed.
- A .dockerignore keeps tests and other repo files out of the build context.
  dist/ stays in, since Dockerfile.preview copies the preview wheel from it.
@lelia

lelia commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 29a60c9. Configure here.

@lelia
lelia temporarily deployed to socket-firewall November 6, 2026 05:18 — with GitHub Actions Inactive
…e the image in CI

The release build cached layers with type=gha. Actions caches are scoped
to the ref that wrote them, and each release builds on its own tag, so no
release could read another's cache: every release rebuilt every layer
under a new digest, and each wrote about 2.6 GB of cache nobody read. A
registry cache at socketdev/cli:buildcache is readable from any tag, and
reusing its layers keeps their digests stable across releases.

e2e-image.yml builds the image from the commit, reusing that cache
read-only, and runs --reach inside it against the npm fixture and a new
Rust fixture. It follows e2e-test.yml's retry policy for the backend's
intermittent empty results. Validation asserts what the image controls:
the run completes and Coana reaches a reachable or unreachable verdict
for at least one vulnerability, which needs the ecosystem's toolchain to
work. It does not assert which verdict, since that depends on Coana and
the backend.
@lelia
lelia temporarily deployed to socket-firewall November 6, 2026 05:31 — with GitHub Actions Inactive
@socket-security

socket-security Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedgithub/​docker/​setup-buildx-action@​f87e5991a6d7451dcb8d9637bfbc97413f497069100100100100100

View full report

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[agent] Simplification pass (quality only, not a bug hunt). Five suggestions and one comment-wording note, all inline. Looked fine: the Dockerfile parser in the test, the validator's own logic, the minimal Rust profile with cargo --version, npm cache clean, dropping apk update, adding Dockerfile to the unit-test push paths, and the .dockerignore entries (none are dead).

Comment thread .github/workflows/e2e-image.yml Resolved
Comment thread .github/workflows/e2e-image.yml Resolved
Comment on lines +82 to +83
if args.probe:
return 0 if alert_states(args.fixture) else 1

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[agent] --probe repeats tests/e2e/reach-facts-probe.sh. This script could be the single implementation, with reach-facts-probe.sh reduced to uv run python tests/e2e/<script>.py <fixture> --probe. The script would then want a name that isn't specific to the image.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ede86cf. tests/e2e/reach_facts.py (renamed, so the name isn't image-specific) is the single implementation of the probe. reach-facts-probe.sh is gone: e2e-test.yml's retry-probe and validate-reachability.sh call uv run python tests/e2e/reach_facts.py <fixture> --probe directly, and the image job runs the same script with the image's Python.

Comment thread Dockerfile Resolved
Comment thread tests/unit/test_dockerfile.py Resolved
Comment thread .dockerignore Resolved
Comment thread Dockerfile Resolved
lelia added 4 commits November 6, 2026 17:07
The reachability engine sets a minimum Node version, and Dependabot doesn't
know Node's release schedule, so it would propose odd-numbered majors, which
never get LTS. The floating node tag already picks up minor and patch releases.
… the Dockerfile guards

- tests/e2e/run-with-retries.sh is the one retry loop for the backend's
  intermittent empty results, used by e2e-test.yml and e2e-image.yml.
- tests/e2e/reach_facts.py is the one probe and the one definition of the
  inconclusive backend result, replacing reach-facts-probe.sh and the
  image-only validator. validate-reachability.sh calls it.
- e2e-image.yml builds the image once and runs every fixture against it,
  recording each fixture's result so one failure doesn't hide another.
- The Dockerfile test lists the ARGs allowed above the toolchain layers and
  fails on any other, so a new per-release ARG is caught too.
- PIP_NO_CACHE_DIR as a build ARG replaces the per-call --no-cache-dir flags.
- .dockerignore lists what the build context includes instead of what it
  excludes.
…ayers

# Conflicts:
#	CHANGELOG.md
#	pyproject.toml
#	socketsecurity/__init__.py
#	uv.lock
@lelia
lelia deployed to socket-firewall November 6, 2026 23:18 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
socket-firewall — 965638cc Deployed Nov 6, 2026 by lelia via python-sfw-smoke-enterprise #571
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants