Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 46 additions & 3 deletions .github/workflows/recover-protocol-cdn.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Recover protocol release on CDN
name: Recover release artifacts on CDN

on:
workflow_dispatch:
Expand All @@ -7,7 +7,19 @@ on:
description: Exact stable semver to recover (for example, 3.0.22)
required: true
type: string

surfaces:
description: >-
protocol = the four signed tuple files only. all = also the immutable
schemas/<version>/ and compliance/<version>/ trees.
required: false
default: protocol
type: choice
options:
- protocol
- all

# One recovery per version at a time, whatever the requested surfaces, so two
# dispatches cannot interleave reads and conditional writes on the same objects.
concurrency:
group: recover-protocol-cdn-${{ inputs.version }}
cancel-in-progress: false
Expand All @@ -21,18 +33,23 @@ permissions:

jobs:
recover:
name: Recover immutable protocol tuple
name: Recover immutable release artifacts
runs-on: ubuntu-latest
steps:
- name: Checkout repository history
uses: actions/checkout@v7.0.1
with:
fetch-depth: 0

# The tag checked out below carries its own copies of these scripts, and
# an old release predates them entirely. Recovery logic must come from
# the current reviewed branch, never from the artifact being recovered.
- name: Preserve current workflow publication fence
run: |
node scripts/check-release-state.cjs current
cp scripts/check-release-state.cjs "$RUNNER_TEMP/check-release-state.cjs"
cp scripts/check-protocol-bundle-parity.cjs "$RUNNER_TEMP/check-protocol-bundle-parity.cjs"
cp scripts/recover-cdn-version-trees.sh "$RUNNER_TEMP/recover-cdn-version-trees.sh"

- name: Install cosign
uses: sigstore/cosign-installer@v3
Expand Down Expand Up @@ -164,3 +181,29 @@ jobs:
--certificate-identity-regexp '^https://github\.com/adcontextprotocol/adcp/\.github/workflows/release\.yml@refs/heads/.*$' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
"${VERSION}.tgz"

# The signed tuple above is the only cryptographic authority available
# for a historical version; R2 objects are unsigned. The script below
# proves the trees it publishes are byte-for-byte the trees inside that
# tarball before it writes anything, so one cosign verification covers
# every object. See RELEASING.md "Recovering a release's CDN artifacts".
- name: Upload only missing immutable schema and compliance objects
if: inputs.surfaces == 'all'
env:
VERSION: ${{ inputs.version }}
FENCE_SCRIPT: ${{ runner.temp }}/check-release-state.cjs
PARITY_SCRIPT: ${{ runner.temp }}/check-protocol-bundle-parity.cjs
BUCKET: ${{ vars.ADCP_ARTIFACT_R2_BUCKET || 'adcp-artifacts' }}
R2_ACCOUNT_ID: ${{ secrets.R2_ACCOUNT_ID || secrets.CLOUDFLARE_ACCOUNT_ID }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID || secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY || secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
run: bash "$RUNNER_TEMP/recover-cdn-version-trees.sh" upload

- name: Verify recovered public CDN trees
if: inputs.surfaces == 'all'
env:
VERSION: ${{ inputs.version }}
FENCE_SCRIPT: ${{ runner.temp }}/check-release-state.cjs
PARITY_SCRIPT: ${{ runner.temp }}/check-protocol-bundle-parity.cjs
run: bash "$RUNNER_TEMP/recover-cdn-version-trees.sh" verify
96 changes: 94 additions & 2 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -424,8 +424,100 @@ Live unscoped bulk backfills are disabled. `--dry-run` remains available for
inventory; it is not release authorization. Historical CDN recovery must use
an existing approved tag/release, a current tested branch fence, unchanged
committed bytes, and conditional missing-object creation. The stable-only
`recover-protocol-cdn.yml` recovers four protocol files, not schemas/compliance
and not RC releases.
`recover-protocol-cdn.yml` is the single dispatch for that; it does not
recover RC releases.

### Recovering a release's CDN artifacts

A stable release whose objects never reached R2 does not announce itself. The
worker's pinned-semver fallback resolves `/schemas/3.0.28/...` to the newest
published release at-or-below it on the same line, so the request returns 200
with an *older release's* bytes. Only two signals distinguish that from a real
hit: the response carries `cache-control: public, no-cache, must-revalidate`
instead of the immutable policy an exact directory gets, and
`build-schema-discovery-pointers.cjs` refuses to advertise the release, so the
deploy log carries `::warning title=Schema versions missing from the CDN`
naming the gap and the line's alias stays pinned one release back (#8318,
#8324, #8347).

Nothing in the routine paths can close that gap. `release.yml` publishes only
the version in the current checkout's `package.json`; `--latest-only` excludes
every semver path; live bulk backfill is disabled. Recovery is a separate,
maintainer-dispatched authority:

```
gh workflow run recover-protocol-cdn.yml --ref main \
-f version=<exact stable semver> -f surfaces=all
```

`surfaces` defaults to `protocol` (the four signed tuple files, the original
scope of this workflow). `all` additionally recovers the immutable
`schemas/<version>/` and `compliance/<version>/` trees through
`scripts/recover-cdn-version-trees.sh`. The run:

- checks out tag `v<version>` and requires its `package.json` to hold exactly
that version. The fence, the parity checker and the recovery script are
copied from the dispatched branch before the tag checkout overwrites them, so
recovery logic never comes from the artifact being recovered;
- verifies the committed signed tuple with `cosign verify-blob` against the
`release.yml` identity and runs `check-release-state.cjs published`, which
binds the tag to its GitHub Release and to the committed tuple bytes. Both
happen on the protocol path, which always runs, so tree recovery inherits
them;
- proves that the committed `dist/schemas/<v>` and `dist/compliance/<v>` trees
are byte-for-byte the `schemas/` and `compliance/` trees inside that signed
tarball (`scripts/check-protocol-bundle-parity.cjs`). R2 objects are
unsigned, so this is what makes the release signature cover them. cosign's
identity regexp binds the signing workflow, not the version, so the bundle's
`adcp-<version>/` root directory is the load-bearing version binding. The
only tolerated divergence is the bounded build timestamp
`build-protocol-tarball.cjs` re-pins in `schemas/manifest.json` and
`compliance/index.json` — the one field the signature does not cover, so it
is constrained by shape and size instead. Any other difference, extra file or
missing file fails the run before a single write;
- reads every published byte with `git cat-file blob`, never the working tree
and never `git archive`, whose output is filtered by the tag's own
`.gitattributes` (`export-subst` splices commit text into the bytes). It
refuses any path that is not a regular blob with an unambiguous name and a
known published content type, and re-checks each file's digest immediately
before its write;
- writes conditionally (`If-None-Match: *`) and only where the object is
confirmed missing. An existing object must already match byte-for-byte *and*
carry the right content type; an inconclusive existence check aborts rather
than reading as absent. Nothing is overwritten or deleted, and the root
discovery pointers stay with the deploy path;
- re-checks the current-branch publication fence before and after every object,
so a branch move mid-run stops publication;
- then fetches every recovered object from the public origin and requires the
bytes, the immutable cache policy **and** the `ETag` to match. The ETag is
the decisive one: R2 reports a single-part object's ETag as the body's hex
MD5, so it proves a bucket object with exactly those bytes answered. Bytes
and cache policy alone cannot, because on a miss the worker proxies to the
Fly origin, which serves the same committed tree and stamps the same
immutable policy for an exact pinned hit
(`server/src/schemas-middleware.ts`).

The dispatch is idempotent and safe to re-run: a complete recovery reports
every object as already verified.

Operational notes:

- **It is slow.** Recovery fences the current branch before and after every
object, exactly as release publication does, so ~770 objects means ~2,300
`git ls-remote` round trips plus the S3 calls. Budget tens of minutes, and
expect a transient failure to abort mid-run; re-dispatch resumes safely.
- **A branch move fails a successful run.** Anything merging to `main` during
the upload trips the fence in the verification step, so the job can go red
after publication actually succeeded. Re-dispatch to confirm.
- **The line alias moves as soon as the first object lands.** The worker
resolves `v3`/`v3.0` from the R2 prefix listing, and a prefix exists after
the first write, so mid-run those aliases already resolve to the version
being recovered while its tree is incomplete. Objects not yet uploaded miss
R2 and fall through to the origin. This is the same progressive-publication
window `release.yml` has for a new release, not a new one, but it is a window
— prefer running recovery when the line is quiet.
- The discovery pointers advance on the next successful `main` deploy that sees
the new prefix, not as part of recovery.

### Explicit recovery of a superseded release merge

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@
"test:build-schemas-hoist-by-identity": "node --test --test-force-exit --test-timeout=30000 tests/build-schemas-hoist-by-identity.test.cjs",
"test:build-schemas-async-response-refs": "node --test --test-force-exit --test-timeout=30000 tests/build-schemas-async-response-refs.test.cjs",
"test:build-schemas-vendor-metric-uniqueness": "node --test --test-force-exit --test-timeout=30000 tests/build-schemas-vendor-metric-uniqueness.test.cjs",
"test:release-workflow": "node tests/release-workflow-immutability.test.cjs && node --test tests/release-publication-ordering.test.cjs tests/maintenance-release-publication.test.cjs tests/changesets-fence.test.cjs tests/release-supersession.test.cjs tests/skip-withdrawn-release.test.cjs && node tests/promote-release-candidate.test.cjs && node tests/published-compliance-versions.test.cjs && node --test --test-timeout=30000 tests/cdn-artifacts.test.cjs tests/artifact-cdn-worker.test.cjs tests/schema-discovery-pointers.test.cjs",
"test:release-workflow": "node tests/release-workflow-immutability.test.cjs && node --test tests/release-publication-ordering.test.cjs tests/maintenance-release-publication.test.cjs tests/changesets-fence.test.cjs tests/release-supersession.test.cjs tests/skip-withdrawn-release.test.cjs && node tests/promote-release-candidate.test.cjs && node tests/published-compliance-versions.test.cjs && node --test --test-timeout=30000 tests/cdn-artifacts.test.cjs tests/artifact-cdn-worker.test.cjs tests/schema-discovery-pointers.test.cjs && node --test --test-timeout=120000 tests/recover-cdn-version-trees.test.cjs",
"promote:rc": "node scripts/promote-release-candidate.mjs",
"test:immutable-release-artifacts": "node tests/immutable-release-artifacts.test.cjs",
"test:runtime-artifacts": "node --test --test-force-exit --test-timeout=30000 tests/runtime-artifact-pruning.test.cjs",
Expand Down
196 changes: 196 additions & 0 deletions scripts/check-protocol-bundle-parity.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
#!/usr/bin/env node
// Prove that a release's committed schema and compliance trees are exactly the
// trees inside that release's signed protocol tarball.
//
// CDN recovery (RELEASING.md "Recovering a release's CDN artifacts") republishes
// hundreds of individual objects for a historical version. Nothing in R2 is
// signed, so the only cryptographic authority available is the cosign signature
// over `dist/protocol/<version>.tgz`. `build-protocol-tarball.cjs` stages that
// tarball by copying `dist/schemas/<version>` and `dist/compliance/<version>`
// verbatim, so proving tree parity makes that one signature transitively cover
// every object recovery is about to write — every byte except the value of one
// bounded timestamp field in the two files named below, which the bundle build
// re-pins and which is therefore constrained by shape and size rather than by
// the signature.
//
// Usage:
// node check-protocol-bundle-parity.cjs <version> <released-root> <bundle-root>
//
// released-root directory holding dist/schemas/<version> and
// dist/compliance/<version>, materialized from the tag's
// Git blobs
// bundle-root the extracted `adcp-<version>/` directory from the tarball
//
// Exits non-zero, with no partial success, on any divergence.

const fs = require("node:fs");
const path = require("node:path");

const STABLE_SEMVER = /^\d+\.\d+\.\d+$/;
// Bounded fractional seconds. The published bytes for the two files below are
// the committed ones, so this field's value is the only thing the release
// signature does not cover; an unbounded fraction would let a release PR
// commit a multi-megabyte "timestamp" that passes parity and is then
// unfixable under the no-overwrite rule.
const ISO_TIMESTAMP = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,9})?Z$/;
// Belt and braces on the same bound, in bytes rather than shape.
const MAX_PINNED_DELTA_BYTES = 64;

// The only divergence `build-protocol-tarball.cjs` is allowed to introduce.
// `pinGeneratedAt` rewrites the staged copy's build timestamp, which is written
// by a later build step than the one that wrote the committed tree. Its sibling
// `pinPublishedVersion` only runs when the bundle is republished under a
// different version, which a release build never does, so nothing else may
// differ. Keyed by surface so a compliance file named manifest.json, or a
// schema file named index.json, gets no exemption.
const PINNED_BUILD_TIMESTAMP_FILES = new Map([
["schemas", "manifest.json"],
["compliance", "index.json"],
]);

function fail(message) {
throw new Error(message);
}

// Reject anything that is not a plain file or directory, so a symlink cannot
// smuggle bytes from outside the compared tree into the comparison or the
// upload that follows it.
function walk(root, surface) {
const files = [];
const visit = (relative) => {
const absolute = path.join(root, relative || ".");
const metadata = fs.lstatSync(absolute);
if (metadata.isSymbolicLink())
fail(`Symlinked ${surface} entry: ${relative}`);
if (metadata.isDirectory()) {
for (const name of fs.readdirSync(absolute).sort())
visit(relative ? path.join(relative, name) : name);
return;
}
if (!metadata.isFile()) fail(`Unsupported ${surface} entry: ${relative}`);
files.push(relative);
};
visit("");
if (files.length === 0) fail(`Empty ${surface} tree: ${root}`);
return files;
}

function parsePinnedDocument(bytes, label) {
let document;
try {
document = JSON.parse(bytes.toString("utf8"));
} catch {
return fail(`Pinned build-timestamp file is not JSON: ${label}`);
}
if (
!document ||
typeof document !== "object" ||
Array.isArray(document) ||
typeof document.generated_at !== "string" ||
!ISO_TIMESTAMP.test(document.generated_at)
)
fail(`Pinned build-timestamp file has no ISO generated_at: ${label}`);
return document;
}

// Byte equality after substituting exactly the one value the bundle build is
// allowed to rewrite. Deep JSON equality would also accept reordered keys and
// reformatted bytes; this does not.
function matchesExceptPinnedTimestamp(releasedBytes, bundleBytes, label) {
if (Math.abs(releasedBytes.length - bundleBytes.length) > MAX_PINNED_DELTA_BYTES)
fail(`Pinned build-timestamp file differs by more than a timestamp: ${label}`);
const released = parsePinnedDocument(releasedBytes, label);
const bundle = parsePinnedDocument(bundleBytes, label);
const needle = Buffer.from(
`"generated_at": ${JSON.stringify(bundle.generated_at)}`,
"utf8",
);
const replacement = Buffer.from(
`"generated_at": ${JSON.stringify(released.generated_at)}`,
"utf8",
);
const first = bundleBytes.indexOf(needle);
if (first === -1) return false;
if (bundleBytes.indexOf(needle, first + needle.length) !== -1)
fail(`Ambiguous generated_at occurrences in ${label}`);
const normalized = Buffer.concat([
bundleBytes.subarray(0, first),
replacement,
bundleBytes.subarray(first + needle.length),
]);
return normalized.equals(releasedBytes);
}

function compareSurface(surface, releasedRoot, bundleRoot) {
for (const [label, root] of [
[`released ${surface}`, releasedRoot],
[`bundled ${surface}`, bundleRoot],
])
if (!fs.existsSync(root)) fail(`Missing ${label} tree: ${root}`);

const released = walk(releasedRoot, `released ${surface}`);
const bundle = walk(bundleRoot, `bundled ${surface}`);
const bundled = new Set(bundle);
const expected = new Set(released);
const extra = bundle.filter((file) => !expected.has(file));
const missing = released.filter((file) => !bundled.has(file));
if (missing.length || extra.length)
fail(
`Bundled ${surface} tree does not match the released tree` +
(missing.length ? `; missing ${missing.slice(0, 5).join(", ")}` : "") +
(extra.length ? `; unexpected ${extra.slice(0, 5).join(", ")}` : ""),
);

const pinned = PINNED_BUILD_TIMESTAMP_FILES.get(surface);
for (const file of released) {
const releasedBytes = fs.readFileSync(path.join(releasedRoot, file));
const bundleBytes = fs.readFileSync(path.join(bundleRoot, file));
if (releasedBytes.equals(bundleBytes)) continue;
const label = `${surface}/${file}`;
if (
file === pinned &&
matchesExceptPinnedTimestamp(releasedBytes, bundleBytes, label)
)
continue;
fail(`Released ${label} differs from the signed protocol bundle`);
}
return released.length;
}

function checkBundleParity(version, releasedRoot, bundleRoot) {
if (!STABLE_SEMVER.test(version || ""))
fail("Bundle parity requires an exact stable semver version.");
const counts = {};
for (const surface of ["schemas", "compliance"]) {
counts[surface] = compareSurface(
surface,
path.join(releasedRoot, "dist", surface, version),
path.join(bundleRoot, surface),
);
}
return counts;
}

if (require.main === module) {
const [version, releasedRoot, bundleRoot] = process.argv.slice(2);
if (!version || !releasedRoot || !bundleRoot) {
console.error(
"Usage: check-protocol-bundle-parity.cjs <version> <released-root> <bundle-root>",
);
process.exitCode = 2;
} else {
try {
const counts = checkBundleParity(version, releasedRoot, bundleRoot);
console.log(
`Signed protocol bundle ${version} covers ${counts.schemas} schema and ${counts.compliance} compliance files.`,
);
} catch (error) {
console.error(
`::error::${error.message}\nRecovery stopped: the released trees are not the signed bundle's trees, so the release signature does not cover them. See RELEASING.md.`,
);
process.exitCode = 1;
}
}
}

module.exports = { checkBundleParity };
Loading
Loading