Skip to content

fix(container-gateway): answer 404 when an inspect fails the label check - #2078

Merged
shahar1 merged 2 commits into
apache:mainfrom
shahar1:fix/container-gateway-not-found-404
Nov 14, 2026
Merged

shahar1 merged 2 commits into
apache:mainfrom
shahar1:fix/container-gateway-not-found-404

Conversation

@shahar1

@shahar1 shahar1 commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • An untyped docker inspect <image> asks for a container of that name first (GET /containers/<name>/json) and tries the image only after a 404. The gateway answered 403, so the CLI stopped there. A tool that checks for an image this way concluded the image was missing. Apache Airflow's breeze then tried to rebuild a CI image that was already present, and the gateway refuses that build as well.
  • A failed label check on an inspect of a container, pod, volume or network now answers 404, for a missing name and for another project's resource alike, with the same body. Every other failed label check stays 403, an exec inspect included. The gateway still forwards nothing that it refused before.
  • The 404 needs a 2xx or a 404 from the daemon for the label check. When the daemon is unreachable or answers with an error, the inspect keeps the 403, so an outage never reads as "not found".
  • The gateway is still not an existence oracle. The rule in relay.py exists so that a client cannot tell "no such resource" from "another project's". Both cases get one answer per route, before and after this change.
  • The troubleshooting entry no longer lists the untyped docker inspect among the workflows that cannot pass the gateway. The README and the two specs say where the 404 applies.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • tools/container-gateway: uv run pytest tests gives 805 passed, 5 deselected. ruff check and mypy src pass.
  • New test test_inspect_of_a_foreign_or_missing_resource_is_the_same_404 (containers, volumes, networks) fails on all three cases without the fix. It asserts 404 for a foreign name and for a missing name, that the two bodies are identical once the name is substituted, and that an owned container still inspects with 200.
  • New test test_inspect_is_403_when_the_backend_cannot_settle_the_label_check fails on the first commit and passes on the second.
  • prek run workspace-pytest --from-ref upstream/main --to-ref HEAD passes. lychee, markdownlint, typos, ruff, mypy, spec-validate and skill-and-tool-validate pass on the changed files. isolated-setup-fingerprint re-stamped isolated_fingerprint.py.
  • Live, Docker 29.9.3 on Linux, patched gateway on its own run directory (run on the first commit; not repeated after the second, which changes only the outage case):
    • docker inspect ghcr.io/apache/airflow/main/ci/python3.11 returns the image. With the released gateway it fails with label-check: … does not belong to this project.
    • docker start <missing name> still gets the 403 label-check refusal.
    • breeze shell no longer tries to build the CI image when the image exists.
  • Not run: anything against podman (not installed here), and macOS.
  • Two review rounds by a second model (Copilot CLI, gpt-6.1-sol, read-only). Round 1 found the outage case, a test that passed if every inspect answered 404, and doc wording that covered exec inspect. All three are fixed in the second commit. Round 2 found nothing.
  • Two hooks fail in my checkout for reasons outside this change, and I skipped only those two for the commit: symlink-lint (old directories under an untracked .worktrees/) and check-doc-sync (an untracked eval folder). Both failures are the same without this change.

RFC-AI-0004 compliance

  • Sandbox — no new host access. No request that was refused is now forwarded: the change is the status code of one refusal.
  • Vendor neutrality — breeze appears only as the example of a tool that hits the refusal, as in the existing troubleshooting entry.

Linked issues

Refs #1810, which documented this refusal and left the 404 out.

Notes for reviewers (optional)

  • This reverses a note I wrote in fix(agent-isolation): hint on container-gateway policy refusals #1810: "A 404 instead of 403 for missing or foreign containers. It would reverse the no-detection decision in relay.py and still not unblock breeze." The first half was wrong for a 404 given to both cases. The no-detection decision needs the two cases to look the same. It does not need the status to be 403.
  • Why inspect only. The docker CLI, compose and buildx probe for a name with an inspect and carry on only after a 404. A refused start, stop, remove or exec keeps the 403, so a policy refusal of an action still reads as a refusal.
  • This does not make breeze run through the gateway. BuildKit and the socket bind-mount are still refused, as the troubleshooting entry says. On my machine breeze shell now passes the image check and stops later on external volume "airflow-pycache-volume" not found. I infer that the volume was created outside the gateway and has no project label; I did not inspect it on the daemon.
  • An untyped inspect of a name that exists nowhere now walks the CLI's whole list of object types and ends on the plugins endpoint, which the gateway denies (denied-endpoint). The exit code is still non-zero. I left that alone.
  • Left out: libpod /exists. It is the same kind of existence probe and still answers 403. I have no podman here to test a change against, so it is a possible follow-up.
  • Left out, separate concern: the README's "Run it" example (container-gateway --project .) lacks the serve subcommand and fails with invalid choice.

Generated-by: Claude Code (Fable 5.2)

🤖 Generated with Claude Code

An untyped `docker inspect <image>` asks for a container of that name
first and tries the image only after a 404. The gateway answered 403, so
the CLI stopped there and the tool calling it concluded the image was
missing. Apache Airflow's breeze then tried to rebuild an image that was
already present, and the build is refused too.

An inspect of a name that is not one of the project's resources now
answers 404, for a missing name and for another project's alike, with
the same body. The two cases stay indistinguishable, so the gateway is
still not an existence oracle. Every other failed label check stays 403.

Signed-off-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
Generated-by: Claude Code (Fable 5.2)
@shahar1 shahar1 added family:setup setup-* skills family:tools tools/* capability:fix Implement code change to resolve an issue capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) substrate:sandbox Tool substrate: agent isolation, egress control, settings audit labels Nov 14, 2026
@github-actions github-actions Bot added family:docs Docs, MISSION.md, READMEs family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json labels Nov 14, 2026
…spect

The 404 for a failed label check on an inspect also covered a backend
that was unreachable or answered with an error. An outage then read as
"not found", and a client could go on to create a resource that exists.
The 404 now needs a 2xx or a 404 from the backend; anything else is
refused with 403, as before.

The docs name the routes the 404 applies to: an inspect of a container,
pod, volume or network. An exec inspect keeps the 403.

Signed-off-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
Generated-by: Claude Code (Fable 5.2)

@Kaap10 Kaap10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Very clean and targeted fix for the untyped docker inspect probing behavior.

A few strong details in the implementation:

  • Existence oracle protection preserved: Ensuring foreign and missing resources return the identical 404 response body keeps the gateway's non-detection property intact.
  • Outage boundary: Keeping 403 when the daemon cannot settle the check (ConnectionRefusedError / unreachable backend) prevents outages from masquerading as "not found".
  • Comprehensive test coverage: Parameterized test test_inspect_of_a_foreign_or_missing_resource_is_the_same_404 and outage case test_inspect_is_403_when_the_backend_cannot_settle_the_label_check thoroughly pin down the contract.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.*

@shahar1
shahar1 merged commit 0a04290 into apache:main Nov 14, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:fix Implement code change to resolve an issue capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json family:docs Docs, MISSION.md, READMEs family:setup setup-* skills family:tools tools/* substrate:sandbox Tool substrate: agent isolation, egress control, settings audit

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants