diff --git a/.claude/skills/local-dev/SKILL.md b/.claude/skills/local-dev/SKILL.md index 3872ca1446..dbebd8bb1e 100644 --- a/.claude/skills/local-dev/SKILL.md +++ b/.claude/skills/local-dev/SKILL.md @@ -173,7 +173,9 @@ fails its first step. ## Docker -Each server has a `Dockerfile`, and no gate builds them. The build context +Each server has a `Dockerfile`. The gate's `image-smoke` stage builds and +boots the image of each server a change touches (`npm run image-smoke -- --all` +for all seven; `docs/quality-gate.md`). The build context differs by language. The TypeScript images build from the **repo root**, because they copy the root `tsconfig.json`: `docker build -t mcp/ -f src//Dockerfile .`. The Python images diff --git a/.claude/skills/pre-push-gate/SKILL.md b/.claude/skills/pre-push-gate/SKILL.md index 8231084a4f..76bd790e12 100644 --- a/.claude/skills/pre-push-gate/SKILL.md +++ b/.claude/skills/pre-push-gate/SKILL.md @@ -275,7 +275,7 @@ of the server's stderr. Re-run a subset with ### `interface-diff` -The last stage diffs the MCP interface (capabilities, tools, prompts, +This stage diffs the MCP interface (capabilities, tools, prompts, resources, resource templates) of each server the change touches, in this checkout's build, against the same server built from the merge-base with `origin/v2/main`, and prints one report per server. Each report says why that @@ -308,6 +308,39 @@ it compares against in CI, and why it is a script rather than an action, is in [`docs/quality-gate.md`](../../../docs/quality-gate.md) and the header of `scripts/interface-diff.mjs`. +### `image-smoke` + +The last stage builds the Docker image of each server the change touches, +boots it with `docker run -i`, lists its tools, and (for `everything` and +`memory`) stops it with `docker stop`. A change that touches no image prints +`nothing to build` and passes without Docker. What selects an image is in +[`docs/quality-gate.md`](../../../docs/quality-gate.md). A failure is +`image-smoke — FAIL ` and the reason: + +- **"no Docker daemon answered"**: the change touches an image and Docker is + not running. Start it and re-run; the stage never skips. +- **"`docker build …` failed"**, with the tail of the build log: the + Dockerfile no longer builds from this checkout. The usual causes are a file + the Dockerfile copies that moved or is excluded by its + `Dockerfile.dockerignore`, a lockfile out of step with a manifest + (`npm ci`, `uv sync --locked`), or, offline, a layer Docker has not cached. +- **"Connection closed"** or a timeout, with the container's stderr: the + image built but its entrypoint did not start the server. Run it by hand: + `docker run -i --rm mcp-image-smoke/:local`. +- **"tools/list does not include …"**: as for `smoke`; the tool the image is + checked for is in `IMAGES` in `scripts/image-smoke.mjs`. +- **"`docker stop` had to kill it"**: the server did not exit on the image's + stop signal. Check the Dockerfile's `STOPSIGNAL` and the server's signal + handlers. +- **"/data holds […] after the stop"**: `memory` left a lock or temporary + file behind, or lost the write. +- **A new Dockerfile fails `test:scripts`** with "every Dockerfile in the + repo has an image entry": add it to `IMAGES`. + +Re-run it on its own with `npm run image-smoke`, `-- --server ` for one +image, `-- --all` for all seven, or `-- --list` to see what it would select +and why. + ## Waiting on the lease A gate that starts with diff --git a/.github/workflows/image-smoke.yml b/.github/workflows/image-smoke.yml new file mode 100644 index 0000000000..6836aecad5 --- /dev/null +++ b/.github/workflows/image-smoke.yml @@ -0,0 +1,144 @@ +# Builds and boots the Docker image of each server a change touches (#5598). +# +# Every server ships a Dockerfile, and before this workflow nothing built one: +# an image that no longer built, could not find its server, or had to be killed +# by `docker stop` reached a release unnoticed, and the release verification +# found it by hand. The check is `npm run image-smoke` +# (scripts/image-smoke.mjs, whose header holds what it asserts and why). The +# same script is a `local:gate` stage, as `scripts/lib/workflow-gate.test.mjs` +# requires of every npm script a workflow runs. +# +# - An image is built only when the change touches it: its server's directory, +# a root file it copies, a package it installs from the root lockfile, or +# the image tooling itself (every image). A `paths` filter cannot say "the +# lockfile changed in a way that reaches this server", so the workflow runs +# on every change and the `changes` job asks the script which images to +# build (`--list`, the same selection the gate makes). `image` fans out over +# that list, one matrix leg per image. This is the interface diff's shape +# (everything-mcp-diff.yml), for the same reasons. +# - The base is the PR's base, or the branch as it was before a push. +# - A manual run (`workflow_dispatch`) builds every image, changed or not: the +# way to check all seven on a branch, for a release verification or after a +# base image moves. +# - Untrusted code (the PR's Dockerfiles and the selection script) runs with a +# read-only token and no persisted credentials. No job here holds a +# credential, pushes an image or logs in to a registry. +# +# The version PRs are opened with the workflow token, whose events start no +# `pull_request` run here. They change version metadata only, and nothing +# dispatches this workflow for them: GitHub dispatches only a workflow whose +# file is on the default branch, which this one is not until a milestone merge. +name: Image smoke + +on: + push: + branches: [main, v2/main] + # No branch filter: PRs to v2/main and to main both run, and so does a PR + # stacked on another PR's branch. The base is the PR's own base either way. + pull_request: + workflow_dispatch: + +permissions: + contents: read + +# Superseded runs on the same PR are cancelled. Every other run gets a group +# of its own, so a push's run is never dropped behind two queued ones. +concurrency: + group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + changes: + name: Choose the images to build + timeout-minutes: 10 + runs-on: ubuntu-latest + outputs: + images: ${{ steps.list.outputs.images }} + steps: + - name: Checkout repository + uses: actions/checkout@v6 + with: + # The base is found in history, so all of it is needed. + fetch-depth: 0 + # The event's own commit (the merge commit for a PR), not a branch + # ref that a later push could move before this step runs. + ref: ${{ github.sha }} + persist-credentials: false + + # No install: the selection imports nothing outside Node. + - uses: actions/setup-node@v7 + with: + node-version: 24 + + - name: List the images the change touches + id: list + env: + EVENT: ${{ github.event_name }} + PR_BASE: ${{ github.event.pull_request.base.sha }} + BEFORE: ${{ github.event.before }} + run: | + case "$EVENT" in + # The checkout is the PR's merge commit, so its base tip is + # exactly what the PR changes. + pull_request) base="$PR_BASE" ;; + # The branch as it was before this push; a new branch has none. + push) + if [ -n "$BEFORE" ] && [ "$BEFORE" != 0000000000000000000000000000000000000000 ]; then + base="$BEFORE" + else + base=$(git rev-parse HEAD^) + fi ;; + workflow_dispatch) base="" ;; + *) echo "unexpected event: $EVENT" >&2; exit 1 ;; + esac + # The reasons go to the log on stderr; stdout is the JSON list. + if [ -n "$base" ]; then + echo "Comparing against $base" + images=$(npm run --silent image-smoke -- --base "$base" --list) + else + images=$(npm run --silent image-smoke -- --all --list) + fi + echo "Images to build: $images" + echo "images=$images" >> "$GITHUB_OUTPUT" + { + if [ "$images" = "[]" ]; then + echo "The change touches no server's image; there is nothing to build." + else + echo "Building: $(jq -r 'map("`" + .server + "`") | join(", ")' <<<"$images")" + fi + } >> "$GITHUB_STEP_SUMMARY" + + image: + name: Image ${{ matrix.server }} + needs: changes + if: needs.changes.outputs.images != '[]' + # A cold build is about a minute; the slack is for a slow registry pull. + timeout-minutes: 20 + runs-on: ubuntu-latest + strategy: + # One image's failure must not hide another's verdict. + fail-fast: false + matrix: + include: ${{ fromJSON(needs.changes.outputs.images) }} + steps: + - name: Checkout repository + uses: actions/checkout@v6 + with: + ref: ${{ github.sha }} + persist-credentials: false + + - uses: actions/setup-node@v7 + with: + node-version: 24 + cache: npm + + # Only for the MCP client the smoke drives the container with, a root + # devDependency. The image builds from the checkout's source inside + # Docker, so no workspace is built here. + - name: Install dependencies + run: npm ci --ignore-scripts + + - name: Build, boot and stop the image + env: + SERVER: ${{ matrix.server }} + run: npm run image-smoke -- --server "$SERVER" diff --git a/AGENTS.md b/AGENTS.md index c090e1a5d0..0648e4ced1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -53,7 +53,7 @@ servers/ ├── .claude/skills/ On-demand procedures (see the Skills index above) ├── .claude/settings.json The SessionStart hook that checks this session's instructions are current ├── .changeset/ Pending changesets for the TypeScript servers, and the changesets config -├── scripts/ The pre-push gate (gate-lease, smoke-servers, validate-py, coverage-py, interface-diff), its guards (verify-*), +├── scripts/ The pre-push gate (gate-lease, smoke-servers, validate-py, coverage-py, interface-diff, image-smoke), its guards (verify-*), │ the skills tooling, release tooling (npm-publish-guard, prepare-python-release, │ pack-and-verify, release-manifest, release-notes), and the issue-filing sweeps │ (dependency-refresh, dependabot-alerts, sdk-watch), and the session-start @@ -65,6 +65,7 @@ servers/ │ version-packages.yml (the changesets PR), prepare-python-release.yml (the CalVer PR), │ everything-mcp-diff.yml (each changed server's interface diff on PRs), │ mcp-interface-diff.yml (a dispatch-only placeholder for that file's new name, #5463), +│ image-smoke.yml (each changed server's Docker image, built and booted), │ main-source.yml (a PR into main must bring nothing that is not on v2/main), │ and the scheduled sweeps that file issues: dependency-refresh.yml, dependabot-alerts.yml, sdk-watch.yml ├── .github/ISSUE_TEMPLATE/ Bug and feature issue forms; config.yml routes security and new servers away @@ -139,14 +140,17 @@ TypeScript workspace's per-file coverage gate), `validate:py` (each Python server's locked sync, `ruff check`, `ruff format --check`, pyright, pytest and build), `coverage:py` (each Python server's per-file coverage gate), `verify:skills:cli`, `smoke` (every server booted over each -transport it implements), and `interface-diff` (the MCP interface of each -server the change touches, diffed against the base it branched from). The stage-by-stage +transport it implements), `interface-diff` (the MCP interface of each +server the change touches, diffed against the base it branched from), and +`image-smoke` (the Docker image of each server the change touches, built, +booted and stopped; it needs a running Docker daemon when a change touches +one). The stage-by-stage reference, and what CI runs where, is [`docs/quality-gate.md`](./docs/quality-gate.md); diagnosing a red stage is the `pre-push-gate` skill. - **`npm run validate` and the per-package commands above are the inner loop, not a substitute.** They skip the other language, the pinned skills validator, - the boot smoke and the interface diff. + the boot smoke, the interface diff and the image smoke. - **Format before committing**: `npm run format` at the root (TypeScript), and `uv run --frozen ruff format .` in a Python server you changed. The gate only checks formatting; it never rewrites files. diff --git a/docs/quality-gate.md b/docs/quality-gate.md index 9dcee6551c..887ea7cf0d 100644 --- a/docs/quality-gate.md +++ b/docs/quality-gate.md @@ -13,7 +13,7 @@ out; everything else points here. | --- | --- | --- | --- | | Inner loop | `npm run validate -w src/`, `npm run validate:py -- ` | Your machine | While iterating on one server | | Pre-push gate | `npm run local:gate` | Your machine, under a lease | Before every push | -| CI | `.github/workflows/typescript.yml`, `python.yml`, `everything-mcp-diff.yml`; `dco.yml`; `main-source.yml` | GitHub, on every push and pull request (`everything-mcp-diff.yml` diffs only the servers the change touches); `dco.yml` on pull requests only; `main-source.yml` on pull requests into `main` only, and not mirrored by a gate stage, since no local push targets `main` | Before merge | +| CI | `.github/workflows/typescript.yml`, `python.yml`, `everything-mcp-diff.yml`, `image-smoke.yml`; `dco.yml`; `main-source.yml` | GitHub, on every push and pull request (`everything-mcp-diff.yml` diffs, and `image-smoke.yml` builds, only the servers the change touches); `dco.yml` on pull requests only; `main-source.yml` on pull requests into `main` only, and not mirrored by a gate stage, since no local push targets `main` | Before merge | **`npm run local:gate` runs every check CI runs.** CI splits the same checks into parallel jobs on a fresh install; the gate runs them in sequence on your @@ -58,6 +58,7 @@ in this order, and the first failure stops the run. | 8 | `verify:skills:cli` | `claude plugin validate` on `.claude/skills`, at the pinned CLI version | `typescript.yml` → **Root guards** | | 9 | `smoke` | Every server boots over each transport it implements and answers one tool call | `typescript.yml` → **Boot smoke** | | 10 | `interface-diff` | For each server the change touches, its MCP interface (capabilities, tools, prompts, resources, resource templates) in this checkout's build, diffed against the same server built from the base. It fails only when a diff cannot run; a changed interface is reported, not failed | `everything-mcp-diff.yml` → **Choose the servers to diff**, then **Diff \** (one leg each, and a PR comment each) | +| 11 | `image-smoke` | For each server the change touches, its Docker image: it builds, a client lists its tools over `docker run -i`, and `docker stop` ends `everything` and `memory` without killing them, `memory` leaving no lock file behind | `image-smoke.yml` → **Choose the images to build**, then **Image \** (one leg each) | Notes on the stages: @@ -100,9 +101,10 @@ Notes on the stages: when a server's environment is missing or behind its lockfile, since `uv sync --locked` then downloads packages (and the pinned interpreter, if `uv` does not have it); `coverage:py` begins with the same sync, which is a - no-op once `validate:py` has run; and `interface-diff` needs it for the - base's `npm ci` and `uv sync`. With warm caches those are the only stages that can fail - offline. + no-op once `validate:py` has run; `interface-diff` needs it for the + base's `npm ci` and `uv sync`; and `image-smoke` needs it for a build whose + layers Docker has not cached (base images, `npm ci`, `uv sync`, `apt-get`). + With warm caches those are the only stages that can fail offline. - **`smoke` launches what a user launches**: the built `dist/index.js` for a TypeScript server, the console script through `uv run --no-sync` for a Python one. stdio and Streamable HTTP for all seven; HTTP+SSE as well for @@ -133,6 +135,28 @@ Notes on the stages: `mcp-server-diff` changes. A change that touches no server diffs nothing and passes. `-- --server ` or `-- --all` diffs servers regardless, and `-- --list` prints the selection without diffing. +- **`image-smoke` needs a running Docker daemon, but only when the change + touches an image** (#5598). It selects as `interface-diff` does + (`selectImages` in `scripts/image-smoke.mjs`): a file under `src//`; + for a TypeScript server, the root `.npmrc` or `tsconfig.json`, which its + image copies, or a `package-lock.json` edit that changes a package the + server installs; and every image when the image tooling changes. A change + that touches none builds nothing and passes with no Docker at all. One that + does, with no daemon answering, **fails rather than skips**: start Docker + and re-run. It builds from the working tree, with the command each + Dockerfile documents (the repository root as the context for a TypeScript + server, the server's directory for a Python one), under the tag + `mcp-image-smoke/:local`, which each run reuses. A build with warm + layers takes a few seconds per image. `-- --server ` or `-- --all` + builds images regardless, and `-- --list` prints the selection without + building. In CI a manual run of `image-smoke.yml` builds all seven. +- **`image-smoke` asserts a clean stop only where an image has one**: + `everything` and `memory`, the two whose Dockerfiles declare a `STOPSIGNAL`. + A container Docker has to kill exits 137, and that fails. For `memory` it + also makes a write first and checks that the graph's directory holds the + graph and nothing else after the stop, so the lock #5485 removes on a + signal is gone. The other five images are killed at the timeout today + (#5611) and are not checked for it. - `python.yml` also has a **Build \** job per server that re-runs pyright and `uv build` and uploads the built distribution. It checks nothing stage 6 does not. diff --git a/package.json b/package.json index 979811b59b..b0e081e0b9 100644 --- a/package.json +++ b/package.json @@ -44,10 +44,11 @@ "verify:dco": "node scripts/verify-dco.mjs", "smoke": "node scripts/smoke-servers.mjs", "interface-diff": "node scripts/interface-diff.mjs", + "image-smoke": "node scripts/image-smoke.mjs", "pack:verify": "node scripts/pack-and-verify.mjs", "release:notes": "node scripts/release-notes.mjs", "local:gate": "node scripts/gate-lease.mjs npm run local:gate:stages", - "local:gate:stages": "npm run verify:install-fresh && npm run verify:dco && npm run validate && npm run coverage && npm run validate:py && npm run coverage:py && npm run verify:skills:cli && npm run smoke && npm run interface-diff" + "local:gate:stages": "npm run verify:install-fresh && npm run verify:dco && npm run validate && npm run coverage && npm run validate:py && npm run coverage:py && npm run verify:skills:cli && npm run smoke && npm run interface-diff && npm run image-smoke" }, "dependencies": { "@modelcontextprotocol/server-everything": "*", diff --git a/scripts/image-smoke.mjs b/scripts/image-smoke.mjs new file mode 100644 index 0000000000..cdfde5a4a8 --- /dev/null +++ b/scripts/image-smoke.mjs @@ -0,0 +1,657 @@ +#!/usr/bin/env node +// The image smoke (#5598): build the Docker image of each server a change +// touches, start it the way a client configuration does (`docker run -i`), +// and ask it for its tools over stdio. +// +// Why it exists: every server ships a `Dockerfile`, and until this script +// nothing built one. The boot smoke runs the checkout and `pack:verify` the +// publish artifact, so a Dockerfile that no longer builds, an image whose +// entrypoint cannot find the server, or a container that `docker stop` has to +// kill all passed every check and reached a release, where they were found by +// hand (the v2.5.0 release verification, #5572). +// +// What it checks, per image: +// +// 1. It builds, with the command its Dockerfile and README document: the +// repository root as the context for a TypeScript server (an npm +// workspace, whose lockfile is the root's), the server's own directory +// for a Python one. +// 2. It boots: an MCP client connects over the container's stdio, lists the +// tools, and finds the one the table names. +// 3. Where the table says the image stops on its own (`cleanStop`), +// `docker stop` ends it before Docker's timeout. A container Docker has +// to SIGKILL exits 137, and the server in it ran none of its cleanup. +// 4. For `memory`, whose cleanup removes the lock file a write holds +// (#5485): after a write and that stop, the graph's directory holds the +// graph and nothing else, so no lock or temporary save file was left. +// +// `cleanStop` is in the table, not read from the image's `STOPSIGNAL`: the +// regression worth catching is that line being dropped, which would also drop +// a check derived from it. The other five images do not stop on their own +// today (#5611), so nothing is asserted of them. +// +// Which images it builds is decided from the files changed since the base, +// as the interface diff decides which servers to diff: a server's own +// directory, the root files its image copies, a change to the packages it +// resolves in the root lockfile, or a change to this tooling (every image). +// The root `package.json` is copied into the TypeScript images too, but it is +// left out of the selection: almost every tooling change edits it (a new npm +// script), and the part of it an image depends on, `overrides`, reaches the +// image only through the lockfile, which is read. CI's `changes` job asks +// this script for the list (`--list`) and fans out one leg per image +// (`--server`); the gate runs it bare. Both go through the same selection. +// +// A change that touches no image builds nothing and needs no Docker. One that +// does needs a running Docker daemon, in the gate as in CI, and fails without +// one rather than skipping: a skipped check that prints a line nobody reads is +// how the gap this closes went unnoticed. A build with a cold layer cache +// also needs the network (base images, `npm ci`, `uv sync`, `apt-get`). +// +// The MCP client is imported only when an image is booted, so `--list` runs +// before any `npm ci`, as the interface diff's does. +// +// Usage: +// npm run image-smoke # base: merge-base with origin/v2/main +// npm run image-smoke -- --base # any commit, tag or branch +// npm run image-smoke -- --server # this image, changed or not (repeatable) +// npm run image-smoke -- --all # every image, changed or not +// npm run image-smoke -- --list # print the selected images as JSON; build nothing + +import { spawnSync } from "node:child_process"; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + rmSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { changesSince, resolveBase } from "./interface-diff.mjs"; +import { lockfileClosure } from "./lib/interface-diff-servers.mjs"; + +const repoRoot = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + "..", +); + +/** How long one MCP request may take, container start included. */ +export const REQUEST_TIMEOUT_MS = 120_000; + +/** The grace period `docker stop` gives a container: Docker's own default. */ +export const STOP_TIMEOUT_SECONDS = 10; + +/** What a container exits with when Docker had to SIGKILL it (128 + 9). */ +export const KILLED_EXIT_CODE = 137; + +/** + * Every server's image: how it is run for the boot, and what proves it is up. + * + * `args` are the server's arguments, after the image name. `env` is set with + * `docker run -e`. `tool` must be in `tools/list`. `cleanStop` says + * `docker stop` must end the container without killing it. `write` is a tool + * call made before the stop, and `leaves` the directory inside the container + * that must then hold exactly those files. + * + * Listed by hand, because the tool to look for is a fact about each server. + * `findUnlistedImages` keeps the list complete: a `src//Dockerfile` + * with no entry here fails the script tests. + */ +export const IMAGES = [ + { name: "everything", language: "ts", tool: "echo", cleanStop: true }, + { + name: "filesystem", + language: "ts", + // It refuses to start without an allowed directory; the image has /tmp. + args: ["/tmp"], + tool: "list_allowed_directories", + }, + { + name: "memory", + language: "ts", + // /data is the directory the image prepares for a graph kept outside + // dist; nothing is mounted over it, so the graph dies with the container. + env: { MEMORY_FILE_PATH: "/data/memory.jsonl" }, + tool: "read_graph", + cleanStop: true, + write: { + name: "create_entities", + arguments: { + entities: [ + { name: "image-smoke", entityType: "probe", observations: ["up"] }, + ], + }, + }, + leaves: { dir: "/data", files: ["memory.jsonl"] }, + }, + { name: "sequentialthinking", language: "ts", tool: "sequentialthinking" }, + { name: "fetch", language: "py", tool: "fetch" }, + { name: "git", language: "py", tool: "git_status" }, + { name: "time", language: "py", tool: "get_current_time" }, +]; + +/** Files whose change selects every image: this tooling itself. */ +export const TOOLING_PATHS = [ + "scripts/image-smoke.mjs", + ".github/workflows/image-smoke.yml", +]; + +/** Root files every TypeScript image copies, besides the manifests. */ +export const TS_WIDE_PATHS = [".npmrc", "tsconfig.json"]; + +/** + * The server directories under `srcDir` that hold a `Dockerfile` and that + * `IMAGES` does not list. + * + * @param {string} srcDir + * @param {{ name: string }[]} [images] + * @returns {string[]} + */ +export function findUnlistedImages(srcDir, images = IMAGES) { + const listed = new Set(images.map((i) => i.name)); + return readdirSync(srcDir, { withFileTypes: true }) + .filter( + (entry) => + entry.isDirectory() && + existsSync(path.join(srcDir, entry.name, "Dockerfile")), + ) + .map((entry) => entry.name) + .filter((name) => !listed.has(name)) + .sort(); +} + +/** + * Parse the command line. + * + * @param {string[]} argv + * @returns {{ base: string | null, servers: string[], all: boolean, + * list: boolean }} + */ +export function parseArgs(argv) { + const opts = { base: null, servers: [], all: false, list: false }; + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i]; + const [flag, inline] = arg.startsWith("--") ? arg.split(/=(.*)/s) : [arg]; + const value = () => { + const v = inline ?? argv[++i]; + if (v === undefined || v === "") throw new Error(`${flag} needs a value`); + return v; + }; + const bare = () => { + if (inline !== undefined) throw new Error(`${flag} takes no value`); + return true; + }; + if (flag === "--base") opts.base = value(); + else if (flag === "--server") opts.servers.push(value()); + else if (flag === "--all") opts.all = bare(); + else if (flag === "--list") opts.list = bare(); + else throw new Error(`unknown argument: ${arg}`); + } + if (opts.all && opts.servers.length > 0) + throw new Error("--all and --server cannot be combined"); + const known = new Set(IMAGES.map((i) => i.name)); + const unknown = opts.servers.filter((s) => !known.has(s)); + // A typo must not read as "nothing to build, so it passed". + if (unknown.length > 0) + throw new Error( + `unknown server(s): ${unknown.join(", ")}. Known: ${[...known].join(", ")}.`, + ); + return opts; +} + +/** Whether two lockfile closures lock different packages. */ +function closuresDiffer(a, b) { + if (a.size !== b.size) return true; + for (const [location, print] of a) if (b.get(location) !== print) return true; + return false; +} + +/** + * The images a change can have broken, in `IMAGES` order, each with the + * reason it was selected. + * + * @param {{ changedFiles: string[], baseLock: object | null, + * headLock: object | null, images?: typeof IMAGES }} input + * `changedFiles` are repository-relative, `/`-separated paths. + * @returns {{ server: string, language: string, reason: string }[]} + */ +export function selectImages({ + changedFiles, + baseLock, + headLock, + images = IMAGES, +}) { + const changed = new Set(changedFiles); + const tooling = TOOLING_PATHS.find((p) => changed.has(p)); + const selected = []; + for (const { name, language } of images) { + const dir = `src/${name}/`; + const own = changedFiles.filter((f) => f.startsWith(dir)); + let reason = tooling ? `\`${tooling}\` (the image tooling) changed` : null; + if (!reason && own.length > 0) + reason = + own.length === 1 + ? `\`${own[0]}\` changed` + : `${own.length} files under \`${dir}\` changed`; + if (!reason && language === "ts") { + const wide = TS_WIDE_PATHS.find((p) => changed.has(p)); + if (wide) + reason = `\`${wide}\`, which every TypeScript image copies, changed`; + else if ( + changed.has("package-lock.json") && + closuresDiffer( + lockfileClosure(baseLock, `src/${name}`), + lockfileClosure(headLock, `src/${name}`), + ) + ) + reason = "`package-lock.json` changes a package it installs"; + } + if (reason) selected.push({ server: name, language, reason }); + } + return selected; +} + +/** + * The images a run covers, each with the reason: every one (`--all`), the + * named ones (`--server`), or else the ones the change touches. + * + * @param {ReturnType} opts + * @param {() => Parameters[0]} changes read lazily, so + * naming the images needs no git history + * @returns {{ server: string, language: string, reason: string }[]} + */ +export function chooseImages(opts, changes) { + const named = (reason, keep) => + IMAGES.filter(keep).map(({ name, language }) => ({ + server: name, + language, + reason, + })); + if (opts.all) return named("every image was asked for (`--all`)", () => true); + if (opts.servers.length > 0) + return named("it was named (`--server`)", (i) => + opts.servers.includes(i.name), + ); + return selectImages(changes()); +} + +/** + * The tag an image is built under. One per server, reused by every run, so + * the images do not pile up and one can be run by hand afterwards. A run never + * boots by this tag: another checkout's run can move it between this run's + * build and its boot, and the boot would then pass on that checkout's image. + * The boot uses the image ID the build wrote (`--iidfile`). + */ +export function imageTag(name) { + return `mcp-image-smoke/${name}:local`; +} + +/** + * How an image is built: `docker`'s arguments and the directory to run it in. + * These are the commands each Dockerfile and README give. + * + * @param {(typeof IMAGES)[number]} image + * @param {string} [root] + * @returns {{ args: string[], cwd: string }} + */ +export function buildCommand(image, root = repoRoot) { + const tag = imageTag(image.name); + return image.language === "ts" + ? { + args: ["build", "-f", `src/${image.name}/Dockerfile`, "-t", tag, "."], + cwd: root, + } + : { + args: ["build", "-t", tag, "."], + cwd: path.join(root, "src", image.name), + }; +} + +/** + * `docker`'s arguments for the boot: `run -i`, as a client configuration has + * it, under a name so the container can be stopped and inspected afterwards + * (hence no `--rm`; the check removes it). + * + * @param {(typeof IMAGES)[number]} image + * @param {string} container + * @param {string} [ref] the image to run: the ID this run built + * @returns {string[]} + */ +export function runArgs(image, container, ref = imageTag(image.name)) { + return [ + "run", + "-i", + "--name", + container, + ...Object.entries(image.env ?? {}).flatMap(([k, v]) => ["-e", `${k}=${v}`]), + ref, + ...(image.args ?? []), + ]; +} + +/** + * Why a stopped container's exit code fails the clean-stop check, or `null` + * when it passes. Any code but Docker's kill is the server's own exit: 0, or + * 128 plus the signal it handled. + * + * @param {number} exitCode + * @returns {string | null} + */ +export function judgeStop(exitCode) { + if (!Number.isInteger(exitCode)) + return "its exit code could not be read after `docker stop`"; + return exitCode === KILLED_EXIT_CODE + ? `\`docker stop\` had to kill it after ${STOP_TIMEOUT_SECONDS}s (exit ${KILLED_EXIT_CODE}): the server did not exit on the image's stop signal, so it ran none of its cleanup. Check the Dockerfile's STOPSIGNAL and the server's signal handlers.` + : null; +} + +/** + * Why the files a stopped container left fail the check, or `null`. + * + * @param {string[]} found + * @param {{ dir: string, files: string[] }} leaves + * @returns {string | null} + */ +export function judgeLeftovers(found, leaves) { + const want = [...leaves.files].sort(); + const got = [...found].sort(); + return got.length === want.length && got.every((f, i) => f === want[i]) + ? null + : `${leaves.dir} holds [${got.join(", ")}] after the stop, not exactly [${want.join(", ")}]: a write was lost, or a lock or temporary file was left behind.`; +} + +const tail = (text) => + (text ?? "").trim().split("\n").slice(-25).join("\n").trim(); + +/** Run `docker` to completion and hand back what happened; never throws. */ +function docker(args, cwd = repoRoot) { + const res = spawnSync("docker", args, { + cwd, + encoding: "utf8", + maxBuffer: 64 * 1024 * 1024, + }); + return { + status: res.error ? null : res.status, + stdout: res.stdout ?? "", + stderr: res.error ? String(res.error.message) : (res.stderr ?? ""), + }; +} + +/** This process's environment, as the string map the transport takes. */ +export function inheritedEnv(base = process.env) { + return Object.fromEntries( + Object.entries(base).filter(([, value]) => typeof value === "string"), + ); +} + +/** The signals that interrupt a run, and what a process they end exits with. */ +export const INTERRUPT_EXIT_CODES = { SIGINT: 130, SIGTERM: 143 }; + +/** + * Connect an MCP client to a container started with `dockerArgs`. + * + * @param {string[]} dockerArgs + * @returns {Promise<{ listTools: () => Promise, + * callTool: (call: object) => Promise<{ isError: boolean, text: string }>, + * stderr: () => string, close: () => Promise }>} + */ +async function connect(dockerArgs) { + const { Client } = await import("@modelcontextprotocol/client"); + const { StdioClientTransport } = + await import("@modelcontextprotocol/client/stdio"); + const options = { timeout: REQUEST_TIMEOUT_MS }; + const client = new Client({ name: "image-smoke", version: "0.0.0" }); + const transport = new StdioClientTransport({ + command: "docker", + args: dockerArgs, + // The transport's default environment is a short allowlist that leaves + // out DOCKER_HOST, DOCKER_CONTEXT and the TLS settings, so `docker run` + // would reach a different daemon from the one `docker()` built on. + env: inheritedEnv(), + stderr: "pipe", + }); + let stderr = ""; + transport.stderr?.on("data", (chunk) => (stderr += chunk)); + const session = { + listTools: async () => + (await client.listTools(undefined, options)).tools.map((t) => t.name), + callTool: async (call) => { + const result = await client.callTool(call, options); + return { + isError: result.isError === true, + text: (result.content ?? []) + .filter((c) => c.type === "text") + .map((c) => c.text) + .join("\n"), + }; + }, + stderr: () => stderr, + close: () => client.close().catch(() => {}), + }; + try { + await client.connect(transport, options); + } catch (err) { + await session.close(); + throw new Error( + `${err?.message ?? err}${tail(stderr) ? `\n--- container stderr ---\n${tail(stderr)}` : ""}`, + { cause: err }, + ); + } + return session; +} + +/** + * Build, boot and stop one image. Resolves when it passes; rejects with the + * reason when it does not. Always removes the container it started. + * + * @param {(typeof IMAGES)[number]} image + * @param {{ docker?: typeof docker, connect?: typeof connect, + * log?: (line: string) => void, root?: string, + * proc?: Pick }} [deps] replaced + * in tests + */ +export async function checkImage(image, deps = {}) { + const run = deps.docker ?? docker; + const open = deps.connect ?? connect; + const log = deps.log ?? console.log; + const proc = deps.proc ?? process; + + const build = buildCommand(image, deps.root); + const scratch = mkdtempSync(path.join(tmpdir(), "mcp-image-smoke-")); + const iidFile = path.join(scratch, "image-id"); + let imageId; + try { + // The context (`.`) stays last; the ID file is how the boot below runs + // exactly what this build produced, whatever the shared tag points at. + const built = run( + [...build.args.slice(0, -1), "--iidfile", iidFile, "."], + build.cwd, + ); + if (built.status !== 0) + throw new Error( + `\`docker ${build.args.join(" ")}\` failed (${built.status ?? "could not run"}):\n${tail(built.stderr) || tail(built.stdout)}`, + ); + imageId = existsSync(iidFile) ? readFileSync(iidFile, "utf8").trim() : ""; + if (!imageId) + throw new Error( + `\`docker ${build.args.join(" ")}\` succeeded but wrote no image ID, so there is nothing to boot.`, + ); + } catch (err) { + rmSync(scratch, { recursive: true, force: true }); + throw err; + } + log( + `image-smoke — built ${imageTag(image.name)} (${imageId.replace(/^sha256:/, "").slice(0, 12)})`, + ); + + const container = `mcp-image-smoke-${image.name}-${process.pid}-${Date.now()}`; + let session; + // An interrupt (Ctrl-C, a cancelled gate) ends the process without running + // the `finally` below, and the container has no `--rm`, so it would be left + // behind, still running where the image ignores the signal `docker run` + // forwards. Remove it first, then exit as the signal would have. + const interrupts = Object.entries(INTERRUPT_EXIT_CODES).map( + ([signal, code]) => [ + signal, + () => { + run(["rm", "-f", container]); + proc.exit(code); + }, + ], + ); + for (const [signal, handler] of interrupts) proc.once(signal, handler); + try { + session = await open(runArgs(image, container, imageId)); + try { + const tools = await session.listTools(); + if (!tools.includes(image.tool)) + throw new Error( + `tools/list does not include \`${image.tool}\` (got: ${tools.join(", ")})`, + ); + if (image.write) { + const result = await session.callTool(image.write); + if (result.isError) + throw new Error( + `\`${image.write.name}\` returned an error result: ${result.text}`, + ); + } + } catch (err) { + const stderr = tail(session.stderr()); + throw new Error( + `${err?.message ?? err}${stderr ? `\n--- container stderr ---\n${stderr}` : ""}`, + { cause: err }, + ); + } + log(`image-smoke — booted; tools/list has \`${image.tool}\``); + + if (image.cleanStop) { + const stopped = run([ + "stop", + "-t", + String(STOP_TIMEOUT_SECONDS), + container, + ]); + if (stopped.status !== 0) + throw new Error(`\`docker stop\` failed:\n${tail(stopped.stderr)}`); + const inspected = run([ + "inspect", + "-f", + "{{.State.ExitCode}}", + container, + ]); + const exitCode = + inspected.status === 0 ? Number(inspected.stdout.trim()) : NaN; + const verdict = judgeStop(exitCode); + if (verdict) throw new Error(verdict); + log(`image-smoke — stopped on its own (exit ${exitCode})`); + } + + if (image.leaves) { + const left = path.join(scratch, "left"); + mkdirSync(left); + // `docker cp` reads a stopped container's filesystem; `/.` copies the + // directory's contents rather than the directory. + const copied = run(["cp", `${container}:${image.leaves.dir}/.`, left]); + if (copied.status !== 0) + throw new Error( + `could not read ${image.leaves.dir} from the stopped container:\n${tail(copied.stderr)}`, + ); + const verdict = judgeLeftovers(readdirSync(left), image.leaves); + if (verdict) throw new Error(verdict); + log( + `image-smoke — ${image.leaves.dir} holds only ${image.leaves.files.join(", ")}`, + ); + } + } finally { + await session?.close(); + run(["rm", "-f", container]); + for (const [signal, handler] of interrupts) proc.off(signal, handler); + rmSync(scratch, { recursive: true, force: true }); + } +} + +/** + * @param {string[]} [argv] + * @param {{ changes?: () => Parameters[0], + * docker?: typeof docker, check?: typeof checkImage }} [deps] replaced in + * tests + * @returns {Promise} the exit code + */ +export async function main(argv = process.argv.slice(2), deps = {}) { + let opts; + try { + opts = parseArgs(argv); + } catch (err) { + console.error(`image-smoke — ${err.message}`); + return 2; + } + + let selected; + try { + selected = chooseImages( + opts, + deps.changes ?? (() => changesSince(resolveBase(opts.base))), + ); + } catch (err) { + console.error(`image-smoke — ${err.message}`); + return 1; + } + for (const s of selected) + console.error(`image-smoke — ${s.server}: selected because ${s.reason}`); + + if (opts.list) { + console.log( + JSON.stringify( + selected.map(({ server, language }) => ({ server, language })), + ), + ); + return 0; + } + if (selected.length === 0) { + console.log( + "image-smoke — the change touches no server's image; nothing to build.", + ); + return 0; + } + + const run = deps.docker ?? docker; + const daemon = run(["version", "--format", "{{.Server.Version}}"]); + if (daemon.status !== 0) { + console.error( + `image-smoke — FAIL no Docker daemon answered (\`docker version\`: ${tail(daemon.stderr) || "no output"}).\n` + + `The change touches ${selected.map((s) => s.server).join(", ")}, so ${selected.length === 1 ? "its image has" : "their images have"} to build and boot before it is pushed. Start Docker and run \`npm run image-smoke\` again.`, + ); + return 1; + } + + const check = deps.check ?? checkImage; + const failures = []; + for (const { server } of selected) { + const image = IMAGES.find((i) => i.name === server); + const started = Date.now(); + console.log(`image-smoke — ${server} …`); + try { + await check(image, { docker: deps.docker }); + console.log(`image-smoke — ok ${server} (${Date.now() - started}ms)`); + } catch (err) { + failures.push(server); + console.error(`image-smoke — FAIL ${server}\n${err?.message ?? err}\n`); + } + } + + if (failures.length > 0) { + console.error( + `image-smoke — ${failures.length} of ${selected.length} failed: ${failures.join(", ")}`, + ); + return 1; + } + console.log( + `image-smoke — OK (${selected.length} image${selected.length === 1 ? "" : "s"} built and booted)`, + ); + return 0; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) + process.exit(await main()); diff --git a/scripts/image-smoke.test.mjs b/scripts/image-smoke.test.mjs new file mode 100644 index 0000000000..c9ead7aabd --- /dev/null +++ b/scripts/image-smoke.test.mjs @@ -0,0 +1,626 @@ +// Tests for the image smoke (#5598). Building and booting the images is what +// `npm run image-smoke` itself does, as a gate stage and in CI, so it is not +// repeated here. What is pinned is what a green run could otherwise hide: a +// Dockerfile missing from the table, a change that selects no image when it +// should, a typo'd name passing as "nothing to build", the build and run +// commands drifting from the documented ones, a killed container read as a +// clean stop, and a missing Docker daemon read as a pass. `checkImage` and +// `main` are driven with a fake `docker` and a fake MCP session. +// Run via `npm run test:scripts`. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { EventEmitter } from "node:events"; +import { + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { + IMAGES, + INTERRUPT_EXIT_CODES, + KILLED_EXIT_CODE, + TOOLING_PATHS, + buildCommand, + checkImage, + chooseImages, + findUnlistedImages, + imageTag, + inheritedEnv, + judgeLeftovers, + judgeStop, + main, + parseArgs, + runArgs, + selectImages, +} from "./image-smoke.mjs"; + +const repoRoot = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + "..", +); +const byName = (name) => IMAGES.find((i) => i.name === name); +const names = (selected) => selected.map((s) => s.server); +const noLocks = { baseLock: null, headLock: null }; + +/** Run `fn` with the console silenced, handing back what it printed. */ +async function quietly(t, fn) { + const lines = []; + for (const method of ["log", "error"]) + t.mock.method(console, method, (line) => lines.push(String(line))); + const result = await fn(); + return { result, output: lines.join("\n") }; +} + +test("every Dockerfile in the repo has an image entry", () => { + assert.deepEqual(findUnlistedImages(path.join(repoRoot, "src")), []); + assert.equal(IMAGES.length, 7); +}); + +test("findUnlistedImages names a Dockerfile the table lacks", (t) => { + const src = mkdtempSync(path.join(tmpdir(), "image-smoke-src-")); + t.after(() => rmSync(src, { recursive: true, force: true })); + for (const [dir, file] of [ + ["time", "Dockerfile"], + ["newserver", "Dockerfile"], + ["nodocker", "package.json"], + ]) { + mkdirSync(path.join(src, dir)); + writeFileSync(path.join(src, dir, file), ""); + } + assert.deepEqual(findUnlistedImages(src), ["newserver"]); +}); + +test("an image that declares a stop signal is checked for a clean stop", () => { + // The table, not the Dockerfile, decides what is asserted, so a deleted + // STOPSIGNAL cannot delete the check with it. This pins the other + // direction: an image that gains a STOPSIGNAL gains the check in the same + // change. Today that is `everything` and `memory`; the rest are #5611. + const declared = IMAGES.filter((image) => + /^STOPSIGNAL\s/m.test( + readFileSync( + path.join(repoRoot, "src", image.name, "Dockerfile"), + "utf8", + ), + ), + ).map((image) => image.name); + assert.deepEqual(declared, ["everything", "memory"]); + for (const name of declared) assert.equal(byName(name).cleanStop, true, name); +}); + +test("parseArgs reads every flag, in either spelling", () => { + const defaults = { base: null, servers: [], all: false, list: false }; + assert.deepEqual(parseArgs([]), defaults); + assert.deepEqual(parseArgs(["--base", "abc", "--list"]), { + ...defaults, + base: "abc", + list: true, + }); + assert.deepEqual(parseArgs(["--server", "git", "--server=time"]), { + ...defaults, + servers: ["git", "time"], + }); + assert.deepEqual(parseArgs(["--all"]), { ...defaults, all: true }); +}); + +test("parseArgs refuses an unknown flag or server, or a missing value", () => { + assert.throws(() => parseArgs(["--push"]), /unknown argument/); + assert.throws(() => parseArgs(["--base"]), /--base needs a value/); + assert.throws(() => parseArgs(["--all=yes"]), /--all takes no value/); + assert.throws(() => parseArgs(["--all", "--server", "git"]), /combined/); + // A typo must not read as "nothing to build". + assert.throws(() => parseArgs(["--server", "tyme"]), /unknown server/); +}); + +test("a change under a server's directory selects that image only", () => { + const selected = selectImages({ + changedFiles: ["src/memory/Dockerfile", "README.md"], + ...noLocks, + }); + assert.deepEqual(names(selected), ["memory"]); + assert.match(selected[0].reason, /src\/memory\/Dockerfile/); + assert.match( + selectImages({ + changedFiles: ["src/git/a.py", "src/git/b.py"], + ...noLocks, + })[0].reason, + /2 files under `src\/git\/`/, + ); +}); + +test("a change that touches no image selects none", () => { + assert.deepEqual( + selectImages({ + changedFiles: ["AGENTS.md", "package.json", "scripts/smoke-servers.mjs"], + ...noLocks, + }), + [], + ); +}); + +test("a root file the TypeScript images copy selects the four of them", () => { + for (const file of [".npmrc", "tsconfig.json"]) + assert.deepEqual( + names(selectImages({ changedFiles: [file], ...noLocks })), + ["everything", "filesystem", "memory", "sequentialthinking"], + file, + ); +}); + +test("the tooling's own files select every image", () => { + for (const file of TOOLING_PATHS) { + assert.ok(existsOnDisk(file), `${file} must exist`); + const selected = selectImages({ changedFiles: [file], ...noLocks }); + assert.equal(selected.length, IMAGES.length, file); + assert.match(selected[0].reason, /image tooling/); + } +}); + +function existsOnDisk(file) { + try { + readFileSync(path.join(repoRoot, file)); + return true; + } catch { + return false; + } +} + +test("a lockfile change selects only the images that install the changed package", () => { + const lock = (zodVersion) => ({ + packages: { + "": { workspaces: ["src/*"] }, + "src/memory": { dependencies: { zod: "^4" } }, + "src/filesystem": { dependencies: { minimatch: "^11" } }, + "node_modules/zod": { version: zodVersion }, + "node_modules/minimatch": { version: "11.0.1" }, + }, + }); + const changedFiles = ["package-lock.json"]; + assert.deepEqual( + names( + selectImages({ + changedFiles, + baseLock: lock("4.0.0"), + headLock: lock("4.0.1"), + }), + ), + ["memory"], + ); + // A rewrite that reaches no server's packages selects nothing. + assert.deepEqual( + selectImages({ + changedFiles, + baseLock: lock("4.0.0"), + headLock: lock("4.0.0"), + }), + [], + ); +}); + +test("chooseImages: --all, --server, or the change, read only when needed", () => { + const never = () => assert.fail("the change must not be read"); + assert.equal(chooseImages(parseArgs(["--all"]), never).length, IMAGES.length); + assert.deepEqual( + names(chooseImages(parseArgs(["--server", "time"]), never)), + ["time"], + ); + assert.deepEqual( + names( + chooseImages(parseArgs([]), () => ({ + changedFiles: ["src/fetch/uv.lock"], + ...noLocks, + })), + ), + ["fetch"], + ); +}); + +test("an image builds with the command its Dockerfile documents", () => { + // TypeScript: the repository root is the context. Python: the server's + // directory is. + assert.deepEqual(buildCommand(byName("memory"), "/repo"), { + args: [ + "build", + "-f", + "src/memory/Dockerfile", + "-t", + "mcp-image-smoke/memory:local", + ".", + ], + cwd: "/repo", + }); + assert.deepEqual(buildCommand(byName("git"), "/repo"), { + args: ["build", "-t", "mcp-image-smoke/git:local", "."], + cwd: path.join("/repo", "src", "git"), + }); + for (const image of IMAGES) { + const dockerfile = readFileSync( + path.join(repoRoot, "src", image.name, "Dockerfile"), + "utf8", + ); + // The TypeScript Dockerfiles say so themselves; keep the table honest. + assert.equal( + dockerfile.includes(`-f src/${image.name}/Dockerfile`), + image.language === "ts", + image.name, + ); + } +}); + +test("an image is run the way a client runs it: `docker run -i`, stdio", () => { + assert.deepEqual(runArgs(byName("filesystem"), "c1"), [ + "run", + "-i", + "--name", + "c1", + "mcp-image-smoke/filesystem:local", + "/tmp", + ]); + assert.deepEqual(runArgs(byName("memory"), "c2"), [ + "run", + "-i", + "--name", + "c2", + "-e", + "MEMORY_FILE_PATH=/data/memory.jsonl", + "mcp-image-smoke/memory:local", + ]); +}); + +test("judgeStop fails only a container Docker killed, or one it cannot read", () => { + assert.equal(judgeStop(0), null); + assert.equal(judgeStop(130), null); + assert.equal(judgeStop(143), null); + assert.match(judgeStop(KILLED_EXIT_CODE), /had to kill it/); + assert.match(judgeStop(NaN), /could not be read/); +}); + +test("judgeLeftovers wants exactly the files named", () => { + const leaves = { dir: "/data", files: ["memory.jsonl"] }; + assert.equal(judgeLeftovers(["memory.jsonl"], leaves), null); + assert.match( + judgeLeftovers(["memory.jsonl", "memory.jsonl.lock"], leaves), + /memory\.jsonl\.lock/, + ); + assert.match(judgeLeftovers([], leaves), /a write was lost/); +}); + +/** + * A fake `docker` that records each call and answers from `answers`, keyed by + * the subcommand; anything unlisted succeeds with no output. `cp` writes + * `files` into its destination, as the real one copies a directory out. + */ +function fakeDocker(answers = {}, files = ["memory.jsonl"]) { + const calls = []; + const docker = (args, cwd) => { + calls.push({ args, cwd }); + // A build that succeeds writes the ID of what it built. + if (args[0] === "build" && (answers.build?.status ?? 0) === 0) + writeFileSync( + args[args.indexOf("--iidfile") + 1], + answers.imageId ?? "sha256:built-here\n", + ); + if (args[0] === "cp" && answers.cp === undefined) + for (const file of files) writeFileSync(path.join(args[2], file), ""); + return { status: 0, stdout: "", stderr: "", ...answers[args[0]] }; + }; + return { docker, calls, ran: () => calls.map((c) => c.args[0]) }; +} + +/** A fake MCP session over a container. */ +function fakeSession({ tools, isError = false } = {}) { + const state = { closed: 0, called: [] }; + const connect = async () => ({ + listTools: async () => tools, + callTool: async (call) => { + state.called.push(call.name); + return { isError, text: "refused" }; + }, + stderr: () => "server stderr line", + close: async () => void (state.closed += 1), + }); + return { connect, state }; +} + +const quiet = { log: () => {} }; + +test("checkImage builds, boots, stops and reads back memory's directory", async () => { + const { docker, calls, ran } = fakeDocker({ + inspect: { stdout: "130\n" }, + }); + const { connect, state } = fakeSession({ tools: ["read_graph"] }); + await checkImage(byName("memory"), { + docker, + connect, + root: "/repo", + ...quiet, + }); + assert.deepEqual(ran(), ["build", "stop", "inspect", "cp", "rm"]); + assert.equal(calls[0].cwd, "/repo"); + const container = calls[1].args.at(-1); + assert.match(container, /^mcp-image-smoke-memory-/); + assert.match(calls[3].args[1], /:\/data\/\.$/); + assert.deepEqual(calls[4].args, ["rm", "-f", container]); + assert.deepEqual(state.called, ["create_entities"]); + assert.equal(state.closed, 1); +}); + +test("checkImage boots the image it built, by ID, not whatever the tag points at", async () => { + // Another checkout's run can retag `mcp-image-smoke/:local` between + // this run's build and its boot. + const { docker, calls } = fakeDocker(); + let booted; + const connect = async (args) => { + booted = args; + return fakeSessionFor(["get_current_time"]); + }; + await checkImage(byName("time"), { docker, connect, ...quiet }); + assert.deepEqual(calls[0].args.slice(0, 3), [ + "build", + "-t", + "mcp-image-smoke/time:local", + ]); + assert.equal(calls[0].args.at(-3), "--iidfile"); + assert.equal(calls[0].args.at(-1), "."); + assert.equal(booted.at(-1), "sha256:built-here"); + assert.ok(!booted.includes("mcp-image-smoke/time:local")); +}); + +test("checkImage fails a build that reports no image ID", async () => { + const { docker, ran } = fakeDocker({ imageId: "" }); + const { connect } = fakeSession({ tools: [] }); + await assert.rejects( + checkImage(byName("time"), { docker, connect, ...quiet }), + /wrote no image ID/, + ); + assert.deepEqual(ran(), ["build"]); +}); + +test("checkImage does not stop-check an image the table does not say stops", async () => { + const { docker, ran } = fakeDocker(); + const { connect } = fakeSession({ tools: ["get_current_time"] }); + await checkImage(byName("time"), { docker, connect, ...quiet }); + assert.deepEqual(ran(), ["build", "rm"]); +}); + +test("checkImage fails an image that does not build, and starts nothing", async () => { + const { docker, ran } = fakeDocker({ + build: { status: 1, stderr: "ERROR: failed to solve" }, + }); + const { connect, state } = fakeSession({ tools: [] }); + await assert.rejects( + checkImage(byName("git"), { docker, connect, ...quiet }), + /docker build -t mcp-image-smoke\/git:local \.` failed \(1\):\nERROR: failed to solve/, + ); + assert.deepEqual(ran(), ["build"]); + assert.equal(state.closed, 0); +}); + +test("checkImage fails an image whose server does not list its tool, with its stderr", async () => { + const { docker, ran } = fakeDocker(); + const { connect, state } = fakeSession({ tools: ["other"] }); + await assert.rejects( + checkImage(byName("everything"), { docker, connect, ...quiet }), + /tools\/list does not include `echo` \(got: other\)\n--- container stderr ---\nserver stderr line/, + ); + // The container is removed and the session closed all the same. + assert.deepEqual(ran(), ["build", "rm"]); + assert.equal(state.closed, 1); +}); + +test("checkImage fails an image whose container never answers", async () => { + const { docker, ran } = fakeDocker(); + const connect = async () => { + throw new Error("Connection closed"); + }; + await assert.rejects( + checkImage(byName("fetch"), { docker, connect, ...quiet }), + /Connection closed/, + ); + assert.deepEqual(ran(), ["build", "rm"]); +}); + +test("checkImage fails a write the server refuses", async () => { + const { docker } = fakeDocker(); + const { connect } = fakeSession({ tools: ["read_graph"], isError: true }); + await assert.rejects( + checkImage(byName("memory"), { docker, connect, ...quiet }), + /`create_entities` returned an error result: refused/, + ); +}); + +test("checkImage fails a container Docker had to kill", async () => { + const { docker, ran } = fakeDocker({ + inspect: { stdout: `${KILLED_EXIT_CODE}\n` }, + }); + const { connect } = fakeSession({ tools: ["echo"] }); + await assert.rejects( + checkImage(byName("everything"), { docker, connect, ...quiet }), + /had to kill it/, + ); + assert.deepEqual(ran(), ["build", "stop", "inspect", "rm"]); +}); + +test("checkImage fails when the stop, the inspect or the copy cannot run", async () => { + const { connect } = fakeSession({ tools: ["read_graph"] }); + for (const [answers, message] of [ + [ + { stop: { status: 1, stderr: "no such container" } }, + /docker stop` failed/, + ], + [{ inspect: { status: 1 } }, /exit code could not be read/], + [ + { inspect: { stdout: "0" }, cp: { status: 1, stderr: "denied" } }, + /could not read \/data from the stopped container:\ndenied/, + ], + ]) + await assert.rejects( + checkImage(byName("memory"), { + docker: fakeDocker(answers).docker, + connect, + ...quiet, + }), + message, + ); +}); + +test("checkImage fails memory when a lock file outlives the stop", async () => { + const { docker } = fakeDocker({ inspect: { stdout: "130" } }, [ + "memory.jsonl", + "memory.jsonl.lock", + ]); + const { connect } = fakeSession({ tools: ["read_graph"] }); + await assert.rejects( + checkImage(byName("memory"), { docker, connect, ...quiet }), + /memory\.jsonl\.lock/, + ); +}); + +test("main: a bad argument exits 2", async (t) => { + const { result, output } = await quietly(t, () => main(["--nope"])); + assert.equal(result, 2); + assert.match(output, /unknown argument/); +}); + +test("main: a change that cannot be read exits 1", async (t) => { + const { result, output } = await quietly(t, () => + main([], { + changes: () => { + throw new Error("origin/v2/main is not available"); + }, + }), + ); + assert.equal(result, 1); + assert.match(output, /origin\/v2\/main is not available/); +}); + +test("main --list prints the selection as JSON and touches no Docker", async (t) => { + const { docker, calls } = fakeDocker(); + const { result, output } = await quietly(t, () => + main(["--list"], { + docker, + changes: () => ({ changedFiles: ["src/time/Dockerfile"], ...noLocks }), + }), + ); + assert.equal(result, 0); + assert.match(output, /\[\{"server":"time","language":"py"\}\]/); + assert.equal(calls.length, 0); +}); + +test("main: a change that touches no image passes without Docker", async (t) => { + const { docker, calls } = fakeDocker({ version: { status: 1 } }); + const { result, output } = await quietly(t, () => + main([], { + docker, + changes: () => ({ changedFiles: ["README.md"], ...noLocks }), + }), + ); + assert.equal(result, 0); + assert.match(output, /nothing to build/); + assert.equal(calls.length, 0); +}); + +test("main: a selected image with no Docker daemon fails; it is never skipped", async (t) => { + const checked = []; + const { result, output } = await quietly(t, () => + main(["--server", "git", "--server", "time"], { + docker: fakeDocker({ + version: { status: 1, stderr: "Cannot connect to the Docker daemon" }, + }).docker, + check: async (image) => void checked.push(image.name), + }), + ); + assert.equal(result, 1); + assert.match(output, /no Docker daemon answered/); + assert.match(output, /Cannot connect to the Docker daemon/); + assert.match(output, /git, time, so their images have/); + assert.deepEqual(checked, []); +}); + +test("main checks every selected image, and one failure does not stop the rest", async (t) => { + const checked = []; + const deps = (failing) => ({ + docker: fakeDocker().docker, + check: async (image) => { + checked.push(image.name); + if (failing.includes(image.name)) throw new Error("did not build"); + }, + }); + const failed = await quietly(t, () => main(["--all"], deps(["memory"]))); + assert.equal(failed.result, 1); + assert.equal(checked.length, IMAGES.length); + assert.match(failed.output, /FAIL {2}memory\ndid not build/); + assert.match(failed.output, /1 of 7 failed: memory/); + + const passed = await quietly(t, () => main(["--server", "time"], deps([]))); + assert.equal(passed.result, 0); + assert.match(passed.output, /OK \(1 image built and booted\)/); +}); + +test("the container is started with this process's whole environment", () => { + // DOCKER_HOST and DOCKER_CONTEXT choose the daemon; the transport's default + // environment drops them, so the boot would miss the daemon the build used. + assert.deepEqual( + inheritedEnv({ DOCKER_HOST: "tcp://x:1", PATH: "/bin", GONE: undefined }), + { DOCKER_HOST: "tcp://x:1", PATH: "/bin" }, + ); +}); + +test("an interrupt removes the container and exits as the signal would", async () => { + for (const [signal, code] of Object.entries(INTERRUPT_EXIT_CODES)) { + const { docker, calls } = fakeDocker(); + const proc = Object.assign(new EventEmitter(), { + exited: [], + exit(n) { + this.exited.push(n); + }, + }); + let interrupted; + const connect = async (args) => { + // The container is up; the interrupt arrives mid-boot. + interrupted = args[args.indexOf("--name") + 1]; + proc.emit(signal); + return fakeSessionFor(["get_current_time"]); + }; + await checkImage(byName("time"), { docker, connect, proc, ...quiet }); + assert.deepEqual(proc.exited, [code], signal); + assert.deepEqual(calls[1].args, ["rm", "-f", interrupted], signal); + // Normal completion unregisters both handlers. + for (const name of Object.keys(INTERRUPT_EXIT_CODES)) + assert.equal(proc.listenerCount(name), 0, name); + } +}); + +test("a failed check unregisters its interrupt handlers too", async () => { + const proc = Object.assign(new EventEmitter(), { exit() {} }); + const { connect } = fakeSession({ tools: ["other"] }); + await assert.rejects( + checkImage(byName("time"), { + docker: fakeDocker().docker, + connect, + proc, + ...quiet, + }), + ); + assert.equal(proc.listenerCount("SIGINT"), 0); + assert.equal(proc.listenerCount("SIGTERM"), 0); +}); + +/** A bare session that lists `tools`. */ +function fakeSessionFor(tools) { + return { + listTools: async () => tools, + callTool: async () => ({ isError: false, text: "" }), + stderr: () => "", + close: async () => {}, + }; +} + +test("imageTag is one fixed tag per server", () => { + assert.equal(imageTag("time"), "mcp-image-smoke/time:local"); +}); diff --git a/scripts/interface-diff.mjs b/scripts/interface-diff.mjs index 4b21fd10f3..07699b64eb 100644 --- a/scripts/interface-diff.mjs +++ b/scripts/interface-diff.mjs @@ -325,8 +325,11 @@ function run(command, args, cwd, env) { const git = (...args) => run("git", args, repoRoot).trim(); -/** The commit to compare against, as a full SHA. */ -function resolveBase(base) { +/** + * The commit to compare against, as a full SHA. Exported for the image smoke + * (`scripts/image-smoke.mjs`), which selects from the same change. + */ +export function resolveBase(base) { if (base) return git("rev-parse", "--verify", `${base}^{commit}`); try { git("rev-parse", "--verify", DEFAULT_BASE_BRANCH); @@ -342,9 +345,10 @@ function resolveBase(base) { * What changed between `sha` and this checkout as it stands: the files * (committed, staged, unstaged and untracked alike, since the head probed is * the working tree's build, both sides of a rename included) and the root - * lockfile on each side (`null` where there is none). + * lockfile on each side (`null` where there is none). Exported for the image + * smoke, as `resolveBase` is. */ -function changesSince(sha) { +export function changesSince(sha) { // NUL-delimited, so a path holding a non-ASCII character, a tab or a // newline comes back as written rather than C-quoted (`"src/time/…`), which // would match no server's directory. diff --git a/scripts/lib/workflow-gate.test.mjs b/scripts/lib/workflow-gate.test.mjs index c08ae56186..9aa870ccf1 100644 --- a/scripts/lib/workflow-gate.test.mjs +++ b/scripts/lib/workflow-gate.test.mjs @@ -393,6 +393,7 @@ describe("the gate's name", () => { "verify:skills:cli", "smoke", "interface-diff", + "image-smoke", ]) { it(`runs \`${stage}\``, () => { assert.ok(