Skip to content

feat(proxy): judge streamed ext_proc bodies whole; verify agentgateway and Envoy Gateway - #112

Open
CharmingGroot wants to merge 6 commits into
feat/gateway-integrationfrom
feat/ext-proc-modes
Open

CharmingGroot wants to merge 6 commits into
feat/gateway-integrationfrom
feat/ext-proc-modes

Conversation

@CharmingGroot

Copy link
Copy Markdown
Owner

Stacked on #110.

#110's ext_proc server required BUFFERED bodies. agentgateway defaults to fullDuplexStreamed, and Envoy Gateway offers Streamed, so a pasu attached with their defaults refused every completion. This PR accepts both streamed modes without letting an unjudged byte through, and verifies pasu on two more data planes.

Streamed body modes (eb1876a)

pasu still sees the whole body before any of it goes on:

mode while the body arrives once whole a refused streamed completion
BUFFERED (one message) pass / rewrite / refuse clean 403
STREAMED each piece answered by clearing it the last piece carries the whole body reset with no body byte (headers were already downstream), or a clean 403 if the refusal beats them
FULL_DUPLEX_STREAMED nothing answered; response headers withheld too whole body sent back in ≤64 KiB pieces clean 403
  • Size limits apply to the whole held body.
  • A held body that ends in trailers is refused.
  • NONE, BUFFERED_PARTIAL and GRPC are still refused (500) when they would hide a body pasu must judge.
  • Bodies pasu does not judge stream through as they arrive.

Tests: 20 new tests in tests/ext_proc_streamed_e2e.rs. Each of 8 mutants is caught: hold, clear, header withholding, per-piece limit, trailer check, accepting BUFFERED_PARTIAL, skipping the final replacement, judging each piece alone.

On Envoy 1.38, with a slow chunked client:

  • Streamed text arrives byte-identical.
  • A denied streamed tool call delivers no body byte.
  • A resident number split across request pieces never reaches the provider.
  • --redact delivers [REDACTED:ko-rrn].
  • With clearing disabled, 763 bytes containing tool_calls leaked.

Misconfiguration refusals (0a19c96, 5329c7a)

  • The 500 now names every wrong body mode, in Envoy, Envoy Gateway and agentgateway spelling. Before, it named one, in Envoy's.
  • It carries "type":"pasu_misconfigured" instead of pasu_policy_block, so an alert on blocks neither fires on a configuration error nor hides one.

agentgateway 1.5.0 (385e2fe, 60e2292)

integrations/agentgateway/config.yaml has been tested in both buffered and the default fullDuplexStreamed:

  • A denied tool call, streamed or not, is a clean 403 with no tool-call byte.
  • /v1/embeddings and a resident-number prompt are refused.
  • failureMode: failClosed (the default) with pasu stopped is a 500.
  • failOpen forwarded everything, the denied tool call included. The docs say never.
  • The default maxBufferSize is 2 MiB.
  • One difference in full duplex: the request headers go upstream before the body is judged. A refused prompt therefore opens a request at the provider; its body never arrives. Use buffered if the provider must not see it at all. The same applies to Envoy's two streamed modes, and is documented there.

Envoy Gateway v1.9.2 on kind (aabbbfc)

integrations/envoy-gateway/ contains manifests: pasu, an EnvoyExtensionPolicy, and a ClientTrafficPolicy buffer limit. The 32 KiB default refused a 100 KiB prompt.

With Buffered bodies everything fails closed, including pasu scaled to 0. Three settings fail open, and pasu cannot detect them because it is never asked:

  • processingMode left out
  • failOpen: true
  • shadowMode: true

The README and the manifests name all three. Envoy Gateway was measured before the streamed-mode support; its Streamed rows say so and were not re-run.

Verification

cargo test (224), clippy -D warnings with default features and without, and fmt --check all pass.

Not verified here:

  • The MSRV 1.86 toolchain (CI's job will check it).
  • Envoy Gateway with streamed modes.
  • An HTTPS backend behind agentgateway.

🤖 Generated with Claude Code

…whole

The ext_proc server refused any body that arrived in pieces, and
agentgateway defaults to FULL_DUPLEX_STREAMED while Envoy Gateway offers
Streamed. Both are now accepted without letting an unjudged byte through:
pasu holds a body it judges until its last piece.

- STREAMED: each held piece is answered with clear_body, so Envoy forwards
  nothing; the last piece is replaced with the whole body, or refused.
- FULL_DUPLEX_STREAMED: held pieces get no answer, and neither do the
  response headers, until the body is whole; it then goes back as streamed
  body responses of at most 64 KiB, the last ending the stream. With no
  message timeout in this mode, withholding the headers is what keeps a
  refusal a clean local reply.
- max_request_bytes / max_response_bytes bound the held body, not a piece.
- misconfigured() accepts BUFFERED, STREAMED and FULL_DUPLEX_STREAMED and
  still refuses NONE, BUFFERED_PARTIAL (and GRPC) for a judged route. A
  held body ending in trailers is refused. No mode_override is sent over a
  streamed mode.
- Bodies pasu does not judge (passthrough, non-2xx, RequestOnly responses,
  prompts with no inspector) stream through untouched; full duplex echoes
  them piece by piece. BUFFERED behaves exactly as before.

Measured with Envoy 1.38 (distroless) and a mock provider: a streamed
shell tool call is a clean 403 under full duplex, and under STREAMED 200
headers then a reset with zero body bytes at the client; streamed text
arrives byte-identical; a Korean RRN split across slow request pieces is
refused in both modes with zero body bytes reaching the provider, and
--redact delivers it redacted. Reverting the clear/hold logic leaks 763
bytes of the tool call to the client under STREAMED.

tests/ext_proc_streamed_e2e.rs drives a Session per mode and applies each
answer as Envoy would, asserting on the bytes that would reach the
provider or the client; every mutant of the hold, clear, header-withhold,
limit, trailer and mode checks fails at least one test.

Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
agentgateway's extProc policy defaults both body modes to
fullDuplexStreamed. It reports those modes to pasu, so the default is
refused with a 500 naming the setting before the request is forwarded;
with buffered bodies a denied tool call, streamed or not, is a 403 and
no tool-call byte reaches the client. Unknown endpoints and a prompt
with a resident registration number are refused without reaching the
provider, and a stopped pasu is a 500 under the default
failureMode: failClosed.

Measured against agentgateway 1.5.0 (standalone) and a plain-HTTP mock
provider. The example also raises maxBufferSize, whose 2 MiB default
refuses larger buffered bodies.

Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
… spelling

The misconfiguration refusal named only the first wrong setting, in Envoy's
snake_case. Found verifying agentgateway 1.5.0, whose default leaves both
body modes unusable and whose options are spelled differently: an operator
fixed one, hit the next 500, and had to translate both. The refusal now
lists every wrong mode and gives the Envoy Gateway and agentgateway names.

Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
With STREAMED/FULL_DUPLEX_STREAMED support, agentgateway 1.5.0's default
fullDuplexStreamed now judges as buffered does: a denied tool call, streamed
or not, is a clean 403 with no tool-call byte at the client, and a refused
prompt's body never reaches the provider. One difference is documented: in
full duplex the request headers go upstream before the body is judged, so
the provider sees a request open and abandoned. Use buffered where it must
not see the request at all.

Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
integrations/envoy-gateway/ holds the manifests run on a kind cluster
against Envoy Gateway v1.9.2: pasu as a Deployment + Service, an
EnvoyExtensionPolicy with both bodies Buffered, a ClientTrafficPolicy
buffer limit, and a Gateway/HTTPRoute to a mock provider.

Through the Gateway, text passed; a shell tool call was refused with no
tool-call byte at the client, streamed or not; /v1/embeddings and an RRN
prompt never reached the provider; the provider saw accept-encoding:
identity while the client asked for gzip. pasu at zero replicas was a
500. Streamed, BufferedPartial, FullDuplexStreamed and a missing response
block were each pasu's 500 naming the setting, so Envoy Gateway does
report protocol_config.

The 32 KiB default buffer limit refused a 100 KiB prompt (413) and a
256 KiB completion (500); the ClientTrafficPolicy alone lifted both.

Three settings fail open and the README names them: no processingMode
(Envoy Gateway then sends pasu nothing), failOpen: true, and shadowMode.

Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
The 500 for a data plane that would never show pasu a body carried the same
error type as a policy refusal, so an alert on pasu_policy_block fired on a
configuration mistake and a block dashboard counted one. Raised verifying
Envoy Gateway.

Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant