Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
1a25341
feat(normalization): define bounded worker protocol and profile contr…
ernestprovo23 Nov 14, 2026
514b7bf
feat(normalization): validate derived evidence and mediate supplied c…
ernestprovo23 Nov 14, 2026
0b3c010
feat(normalizer): add bounded recursive parser worker (DSE-718)
ernestprovo23 Nov 14, 2026
d92d74a
fix(normalizer): qualify embedded content and conservative format ref…
ernestprovo23 Nov 14, 2026
0c95354
feat(normalization): supervise immutable parser with bounded Linux Do…
ernestprovo23 Nov 14, 2026
7163c76
fix(normalization): qualify Docker controls and cumulative child CPU …
ernestprovo23 Nov 14, 2026
9a2885c
fix(normalization): bound coordinator allocations and preserve strict…
ernestprovo23 Nov 14, 2026
2aba529
fix(normalizer): inspect reconstructed structured text views (DSE-718)
ernestprovo23 Nov 14, 2026
cd2062e
feat(normalization): qualify Linux execution and document finite cove…
ernestprovo23 Nov 14, 2026
65748b5
fix(normalizer): preserve evidence while removing C1 controls (DSE-718)
ernestprovo23 Nov 14, 2026
8bd2b3b
fix(normalization): revalidate public evidence and bind original text…
ernestprovo23 Nov 14, 2026
bce07d6
fix(normalizer): inspect bounded CommonMark semantic token views (DSE…
ernestprovo23 Nov 14, 2026
4726c0b
test(normalization): freeze adversarial evidence and runtime regressions
ernestprovo23 Nov 14, 2026
7429c3f
fix(normalization): bind live inspection to original MCP body bytes
ernestprovo23 Nov 14, 2026
42972ae
test(normalization): qualify every supported ingress and format for v…
ernestprovo23 Nov 14, 2026
81953d7
fix(normalizer): package project and parser license notices
ernestprovo23 Nov 14, 2026
f9cf474
fix(normalizer): refuse duplicate critical MIME headers (DSE-718)
ernestprovo23 Nov 14, 2026
de34061
docs: align v2.3.0 changelog and historical publisher check
ernestprovo23 Nov 14, 2026
c872932
test(normalizer): isolate cleanup assertions to the current container
ernestprovo23 Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/normalization.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
name: Normalization qualification

on:
pull_request:
branches: [main]
push:
branches: [main]

permissions:
contents: read

concurrency:
group: normalizer-${{ github.ref }}
cancel-in-progress: true

jobs:
qualify:
name: Linux isolated normalization
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install coordinator and standalone parser test closure
run: |
pip install --require-hashes -r requirements-dev.lock
pip install --require-hashes -r components/normalizer/requirements.lock
pip install --no-deps -e .
- name: Build exact immutable worker
run: |
docker build --iidfile "$RUNNER_TEMP/normalizer-image" -f components/normalizer/Dockerfile .
echo "NORMALIZER_TEST_IMAGE=$(cat "$RUNNER_TEMP/normalizer-image")" >> "$GITHUB_ENV"
- name: Audit independent parser closure
run: |
python -m venv "$RUNNER_TEMP/parser-audit"
"$RUNNER_TEMP/parser-audit/bin/pip" install "pip-audit==2.9.0"
"$RUNNER_TEMP/parser-audit/bin/python" -m pip_audit --disable-pip --require-hashes -r components/normalizer/requirements.lock
- name: Qualify actual sandbox and every ingress
run: bash scripts/qualify_normalizer.sh "$NORMALIZER_TEST_IMAGE"
- name: Deterministic normalization property corpus
run: pytest -q tests/fuzz/test_fuzz_normalization.py --hypothesis-seed=0
14 changes: 13 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,17 @@ Streamable HTTP; the v0.3 `guard` proxy adds deterministic runtime *result* insp

## [Unreleased]

## [2.3.0] — 2026-11-14

### Added

- DSE-718 opt-in Linux supplied-content normalization: independent isolated parser
worker; all nine ingress kinds; bounded text/web/data, MIME, ZIP and conservative
text-PDF extraction; raw and normalized inspection, labeled provenance, immutable
worker IDs, atomic guard/offline refusal, `normalize` evidence CLI and dedicated
real-sandbox qualification. Office/OCR, external retrieval and non-Linux worker
execution are unsupported. No whole-ATK or arbitrary-format claim is made.

### Fixed

- Both dependency locks pin `opentelemetry-api` 1.45.2 to address
Expand Down Expand Up @@ -582,7 +593,8 @@ v1-strong foundation. Highlights of the 0.3.0 → 1.0.0 arc:
declared surface, RFC 8785 (JCS) + SHA-256 canonicalization, SARIF output, and a live
integrity-gate workflow with committed `clean.warden.lock`.

[Unreleased]: https://github.com/DataScience-EngineeringExperts/mcp-warden/compare/v2.2.0...HEAD
[Unreleased]: https://github.com/DataScience-EngineeringExperts/mcp-warden/compare/v2.3.0...HEAD
[2.3.0]: https://github.com/DataScience-EngineeringExperts/mcp-warden/compare/v2.2.0...v2.3.0
[2.2.0]: https://github.com/DataScience-EngineeringExperts/mcp-warden/compare/v2.1.0...v2.2.0
[2.1.0]: https://github.com/DataScience-EngineeringExperts/mcp-warden/compare/v2.0.0...v2.1.0
[1.1.0]: https://github.com/DataScience-EngineeringExperts/mcp-warden/compare/v1.0.1...v1.1.0
Expand Down
16 changes: 16 additions & 0 deletions DOCUMENTATION_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,22 @@ implementations (Python reference, zero-dependency TypeScript) prove it in CI.
| [`packages/lock-ts/`](packages/lock-ts/README.md) | `@mcp-warden/lock` — verify-only TypeScript: hand-written JCS, SHA-256, capability + skeleton derivation, drift classifier; `npm test` runs the same corpus |
| [`.github/workflows/integrity-gate.yml`](.github/workflows/integrity-gate.yml) `conformance` job | Runs both harnesses; mutation proof flips one hex char and requires BOTH to fail |

## Supplied-content normalization (DSE-718)

| Artifact | Purpose |
|---|---|
| [`docs/CONTENT_NORMALIZATION.md`](docs/CONTENT_NORMALIZATION.md) | Finite nine-ingress supplied-content contract, format/provenance limits, Linux sandbox, budgets, refusal/migration and qualification |
| [`components/normalizer/README.md`](components/normalizer/README.md) | Standalone parser component, independent hash lock, pinned runtime and conservative format profiles |
| [`src/mcp_warden/normalization_models.py`](src/mcp_warden/normalization_models.py) | Frozen finite profiles and evidence types |
| [`src/mcp_warden/normalization.py`](src/mcp_warden/normalization.py) | Trusted coordinator validates worker evidence and derives untrusted envelopes |
| [`src/mcp_warden/normalization_sandbox.py`](src/mcp_warden/normalization_sandbox.py) | Fixed immutable-image Linux Docker supervisor; bounded IPC and fail-closed cleanup |
| [`src/mcp_warden/normalization_ingress.py`](src/mcp_warden/normalization_ingress.py) | All nine supplied-payload adapters; no external retrieval |
| [`src/mcp_warden/normalization_inspection.py`](src/mcp_warden/normalization_inspection.py) | Raw/normalized shared catalog and supplied MCP container checks |
| [`src/mcp_warden/normalization_guard.py`](src/mcp_warden/normalization_guard.py) | Atomic opt-in live frame mediation before legacy handling |
| [`src/mcp_warden/normalization_trace.py`](src/mcp_warden/normalization_trace.py) | Bounded, explicit-direction offline trace loading |
| [`scripts/qualify_normalizer.sh`](scripts/qualify_normalizer.sh) | Actual-image qualification with independent builder QA claims |
| [`.github/workflows/normalization.yml`](.github/workflows/normalization.yml) | Dedicated Python 3.12 Linux parser/sandbox/every-ingress/fuzz qualification |

## Agent gates (`deploy-gate` + `auth audit` — DSE-1257 / DSE-1258)

Two fail-closed gates extending warden past MCP-surface integrity into the adjacent
Expand Down
22 changes: 19 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# mcp-warden

Last Updated: 2026-11-06
Last Updated: 2026-11-14

[![CI](https://github.com/DataScience-EngineeringExperts/mcp-warden/actions/workflows/integrity-gate.yml/badge.svg)](https://github.com/DataScience-EngineeringExperts/mcp-warden/actions/workflows/integrity-gate.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
Expand All @@ -21,8 +21,8 @@ operators can check production OIDC without uploading via [`RELEASING.md`](RELEA

`mcp-warden-cli` is owned by the [TheDataExperts PyPI organization](https://pypi.org/org/TheDataExperts/).
The company-repository publisher passed an authentication-only check on 2026-11-06;
no package was uploaded and PyPI remains at 2.2.0. The next normal release must
still prove OIDC upload and provenance; see the [publishing status](RELEASING.md#ownership-and-publishing-status--2026-11-06).
that check uploaded no package. The next normal release must prove OIDC upload
and provenance; see the [publishing status](RELEASING.md#ownership-and-publishing-status--2026-11-06).

The proposed next direction is a protocol-neutral **Warden**: human-approved tool versions and bounded actions, with untrusted inputs kept separate from authority.
The [upgrade plan and checkpoint proposal](docs/plans/2026-10-02-tool-integrity-upgrade.md)
Expand Down Expand Up @@ -183,6 +183,21 @@ automatically — so the use cases are sequenced by leverage:
- **Agent-framework integrator** *(post-launch).* Enforce that only warden-locked servers
register in a LangGraph-style orchestrator — one integration locks an entire downstream ecosystem.

### Isolated supplied-content inspection (DSE-718)

The opt-in Linux normalizer processes supplied text/web/data, email, ZIP and a
conservative text-PDF subset through one engine across all nine declared ingress
families. A separately hash-locked parser image runs without network, credentials
or host-data mounts. The coordinator preserves raw-byte evidence and creates
untrusted derived envelopes with labeled source precision; it grants no delivery
authority. Office/OCR, external retrieval and non-Linux worker execution refuse.

`normalize <file> --media-type <MIME> --source-kind <kind> --normalizer-image <sha256:id>`
emits evidence and redacted findings. `guard` and `inspect` accept the same
`--normalizer-image` option for atomic, fail-closed inspection using a qualified
immutable local image. Without it, their narrower legacy contracts remain.
See [the supported profiles, sandbox qualification and limits](docs/CONTENT_NORMALIZATION.md).

---

## What it does
Expand Down Expand Up @@ -395,6 +410,7 @@ run the gate only on push:
| `mcp-warden policy eval <file> <sample.json> [--lock F]` | Evaluate one sample call | **non-zero on a deny verdict** (CI assertion) |
| `mcp-warden guard <server-cmd...> [--lock F] [--policy F] [--no-block-* / --allow-exfil-domain] [--block-inject-phrase] [--audit-only] [--strict] [--sarif F] [--record T]` | **(v0.3)** Transparent stdio proxy: inspects `tools/call` results + arguments at runtime. **Deterministic tier blocks by default**; opt out per-category with `--no-block-<category>` or fully with `--audit-only`. `--strict` fails CLOSED on an internal inspection error (exit `3`) | child's exit code; `3` on a `--strict` abort; otherwise never breaks the session |
| `mcp-warden inspect <trace.jsonl> [--lock F] [--sarif F]` | **(v0.2)** Offline analyzer over a recorded JSON-RPC session — same `WRD-RES-*` catalog as `guard` (always report-only) | non-zero on any BLOCK-tier finding; 2 on read error |
| `mcp-warden normalize <file> --media-type MIME --source-kind KIND --normalizer-image sha256:ID` | Linux supplied-content normalization/inspection through an independently qualified isolated worker; emits evidence, not extracted content. `guard`/`inspect` also accept `--normalizer-image`; see [`docs/CONTENT_NORMALIZATION.md`](docs/CONTENT_NORMALIZATION.md) | 0 no blocking finding; 1 blocking finding; 2 refusal/error |
| `mcp-warden lock rotate <lock> [--approver ID] [--actor ID] [--note T] [--json]` | **(v0.3)** Re-attest provenance on an existing baseline without re-capturing the surface; `overall_digest` stays **byte-identical** (WARDEN_LOCK_SCHEMA §8.2). Fails closed on a tampered/inconsistent lock | 0 on success, 2 on missing/invalid/tampered lock |
| `mcp-warden lock sign <lock> [--coordinate C] [--identity-token T]` | **(DSE-1530)** Sigstore-sign an existing **unsigned** lock and write `<lockname>.sigstore` beside it — for attesters that capture offline and sign afterwards. v2 statement (binds the coordinate) with `--coordinate`, v1 otherwise; `overall_digest` unchanged. Refuses an existing sidecar or an already-signed lock | 0 on success; **1 on signing failure (fail closed, no partial sidecar)**; 2 on missing/tampered lock, bad coordinate, existing signature, or missing `mcp-warden[sigstore]` |
| `mcp-warden diff <lock-a> <lock-b> [--json] [--sarif F] [--no-provenance] [--exit-code]` | **(v0.3)** Offline, **redacted** viewer over the drift engine: renders integrity drift between two existing locks (A=baseline, B=current) + a separate informational provenance section. Never re-captures and never prints raw `server.command`/`args` (secret-safe) | 0 (viewer); with `--exit-code`, 1 on **integrity** drift only; 2 on missing/invalid lock |
Expand Down
29 changes: 25 additions & 4 deletions SYSTEM_CONTEXT_DIAGRAM.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# mcp-warden — System Context Diagram

Last Updated: 2026-11-06
Last Updated: 2026-11-14

**CLI 2.0.0 / schema level 4** extends capture/lock/check to complete tool annotations and output
**CLI 2.3.0 / schema level 4** includes complete tool annotations and output
schemas, with structural output drift and Python/TypeScript parity. The existing
tools/list gate compares those commitments for v4 locks; legacy locks retain narrower
runtime coverage and require re-pin for the new fields. Annotations grant no authority.
Expand Down Expand Up @@ -60,6 +60,27 @@ logic) plus a separate informational provenance section. It never prints raw
> enter an existing code-scanning pipeline with no new plumbing. Runtime capability
> brokering remains out of scope (DSE-725). See [`docs/AGENT_GATES.md`](docs/AGENT_GATES.md).

> **Supplied-content normalization (DSE-718)** is an opt-in Linux worker path,
> separate from the base CLI parser dependencies. All nine input families use
> the same engine over supplied bytes; no external connectors fetch content.
> The coordinator creates digest-bound untrusted lineage/maps and validates
> bounded output; PDP/PEP remains the delivery authority. The optional guard
> wrapper inspects every complete server frame before legacy delivery, forces
> strict refusal, and has no in-process fallback. See
> [`docs/CONTENT_NORMALIZATION.md`](docs/CONTENT_NORMALIZATION.md).

```mermaid
flowchart LR
supplied[Supplied bytes: nine ingress families] --> capture[Exact-byte ingress envelope]
capture --> coord[normalization.py: trusted evidence coordinator]
coord --> sandbox[normalization_sandbox.py: Linux isolated immutable worker]
sandbox --> parser[components/normalizer: bounded recursive parsers]
parser --> validate[Bounded untrusted manifest validation]
validate --> evidence[Derived untrusted envelope and source-map digest]
evidence --> inspection[Raw and normalized shared inspection]
inspection --> gate[Existing policy and delivery boundary]
```

> **`doctor` (DSE-1516)** is the zero-config entry point and sits at the far left of the
> picture: it reads the MCP client configs already on the developer's machine (Claude
> Code, Claude Desktop, Cursor, VS Code, Windsurf, Codex — a documented, closed set with a
Expand Down Expand Up @@ -104,8 +125,8 @@ logic) plus a separate informational provenance section. It never prints raw
> the company-repository publisher subsequently passed the 2026-11-06
> authentication-only check ([run 42224199909](https://github.com/DataScience-EngineeringExperts/mcp-warden/actions/runs/42224199909)).
> `mcp-warden-cli` now belongs to the [TheDataExperts PyPI organization](https://pypi.org/org/TheDataExperts/).
> No upload occurred; PyPI remains at 2.2.0. End-to-end OIDC upload and provenance
> still need proof on the next normal release. See [`RELEASING.md`](RELEASING.md).
> That authentication-only check uploaded no package. End-to-end OIDC upload
> and provenance require proof from the next normal release. See [`RELEASING.md`](RELEASING.md).

---

Expand Down
13 changes: 13 additions & 0 deletions components/normalizer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Official Python 3.12.16 slim-bookworm linux/amd64 manifest, reviewed 2026-11-14.
FROM python:3.12.16-slim-bookworm@sha256:0f086103b026048cc50cb5da25ec435c023c5241a83fc44daea593377abc61c4

COPY components/normalizer/requirements.lock /worker/requirements.lock
RUN python -m pip install --no-cache-dir --no-compile --require-hashes \
--only-binary=:all: --no-deps -r /worker/requirements.lock
COPY components/normalizer/*.py /worker/
COPY src/mcp_warden/normalization_ipc.py /worker/normalization_ipc.py
COPY LICENSE /worker/LICENSE
COPY components/normalizer/THIRD_PARTY_NOTICES.md /worker/THIRD_PARTY_NOTICES.md
WORKDIR /worker
USER 10001:10001
ENTRYPOINT ["/usr/bin/env", "-i", "/usr/local/bin/python3", "-I", "-B", "/worker/worker.py"]
Loading
Loading