Repository navigation
feat(proxy): judge streamed ext_proc bodies whole; verify agentgateway and Envoy Gateway - #112
Open
CharmingGroot wants to merge 6 commits into
Open
CharmingGroot wants to merge 6 commits into
CharmingGroot wants to merge 6 commits into
Conversation
…whole The ext_proc server refused any body that arrived in pieces, and agentgateway defaults to FULL_DUPLEX_STREAMED while Envoy Gateway offers Streamed. Both are now accepted without letting an unjudged byte through: pasu holds a body it judges until its last piece. - STREAMED: each held piece is answered with clear_body, so Envoy forwards nothing; the last piece is replaced with the whole body, or refused. - FULL_DUPLEX_STREAMED: held pieces get no answer, and neither do the response headers, until the body is whole; it then goes back as streamed body responses of at most 64 KiB, the last ending the stream. With no message timeout in this mode, withholding the headers is what keeps a refusal a clean local reply. - max_request_bytes / max_response_bytes bound the held body, not a piece. - misconfigured() accepts BUFFERED, STREAMED and FULL_DUPLEX_STREAMED and still refuses NONE, BUFFERED_PARTIAL (and GRPC) for a judged route. A held body ending in trailers is refused. No mode_override is sent over a streamed mode. - Bodies pasu does not judge (passthrough, non-2xx, RequestOnly responses, prompts with no inspector) stream through untouched; full duplex echoes them piece by piece. BUFFERED behaves exactly as before. Measured with Envoy 1.38 (distroless) and a mock provider: a streamed shell tool call is a clean 403 under full duplex, and under STREAMED 200 headers then a reset with zero body bytes at the client; streamed text arrives byte-identical; a Korean RRN split across slow request pieces is refused in both modes with zero body bytes reaching the provider, and --redact delivers it redacted. Reverting the clear/hold logic leaks 763 bytes of the tool call to the client under STREAMED. tests/ext_proc_streamed_e2e.rs drives a Session per mode and applies each answer as Envoy would, asserting on the bytes that would reach the provider or the client; every mutant of the hold, clear, header-withhold, limit, trailer and mode checks fails at least one test. Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
agentgateway's extProc policy defaults both body modes to fullDuplexStreamed. It reports those modes to pasu, so the default is refused with a 500 naming the setting before the request is forwarded; with buffered bodies a denied tool call, streamed or not, is a 403 and no tool-call byte reaches the client. Unknown endpoints and a prompt with a resident registration number are refused without reaching the provider, and a stopped pasu is a 500 under the default failureMode: failClosed. Measured against agentgateway 1.5.0 (standalone) and a plain-HTTP mock provider. The example also raises maxBufferSize, whose 2 MiB default refuses larger buffered bodies. Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
… spelling The misconfiguration refusal named only the first wrong setting, in Envoy's snake_case. Found verifying agentgateway 1.5.0, whose default leaves both body modes unusable and whose options are spelled differently: an operator fixed one, hit the next 500, and had to translate both. The refusal now lists every wrong mode and gives the Envoy Gateway and agentgateway names. Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
With STREAMED/FULL_DUPLEX_STREAMED support, agentgateway 1.5.0's default fullDuplexStreamed now judges as buffered does: a denied tool call, streamed or not, is a clean 403 with no tool-call byte at the client, and a refused prompt's body never reaches the provider. One difference is documented: in full duplex the request headers go upstream before the body is judged, so the provider sees a request open and abandoned. Use buffered where it must not see the request at all. Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
integrations/envoy-gateway/ holds the manifests run on a kind cluster against Envoy Gateway v1.9.2: pasu as a Deployment + Service, an EnvoyExtensionPolicy with both bodies Buffered, a ClientTrafficPolicy buffer limit, and a Gateway/HTTPRoute to a mock provider. Through the Gateway, text passed; a shell tool call was refused with no tool-call byte at the client, streamed or not; /v1/embeddings and an RRN prompt never reached the provider; the provider saw accept-encoding: identity while the client asked for gzip. pasu at zero replicas was a 500. Streamed, BufferedPartial, FullDuplexStreamed and a missing response block were each pasu's 500 naming the setting, so Envoy Gateway does report protocol_config. The 32 KiB default buffer limit refused a 100 KiB prompt (413) and a 256 KiB completion (500); the ClientTrafficPolicy alone lifted both. Three settings fail open and the README names them: no processingMode (Envoy Gateway then sends pasu nothing), failOpen: true, and shadowMode. Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
The 500 for a data plane that would never show pasu a body carried the same error type as a policy refusal, so an alert on pasu_policy_block fired on a configuration mistake and a block dashboard counted one. Raised verifying Envoy Gateway. Signed-off-by: Ho Geun Choi <ohyes9711@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #110.
#110's ext_proc server required
BUFFEREDbodies. agentgateway defaults tofullDuplexStreamed, and Envoy Gateway offersStreamed, so a pasu attached with their defaults refused every completion. This PR accepts both streamed modes without letting an unjudged byte through, and verifies pasu on two more data planes.Streamed body modes (
eb1876a)pasu still sees the whole body before any of it goes on:
BUFFEREDSTREAMEDFULL_DUPLEX_STREAMEDNONE,BUFFERED_PARTIALandGRPCare still refused (500) when they would hide a body pasu must judge.Tests: 20 new tests in
tests/ext_proc_streamed_e2e.rs. Each of 8 mutants is caught: hold, clear, header withholding, per-piece limit, trailer check, acceptingBUFFERED_PARTIAL, skipping the final replacement, judging each piece alone.On Envoy 1.38, with a slow chunked client:
--redactdelivers[REDACTED:ko-rrn].tool_callsleaked.Misconfiguration refusals (
0a19c96,5329c7a)"type":"pasu_misconfigured"instead ofpasu_policy_block, so an alert on blocks neither fires on a configuration error nor hides one.agentgateway 1.5.0 (
385e2fe,60e2292)integrations/agentgateway/config.yamlhas been tested in bothbufferedand the defaultfullDuplexStreamed:/v1/embeddingsand a resident-number prompt are refused.failureMode: failClosed(the default) with pasu stopped is a 500.failOpenforwarded everything, the denied tool call included. The docs say never.maxBufferSizeis 2 MiB.bufferedif the provider must not see it at all. The same applies to Envoy's two streamed modes, and is documented there.Envoy Gateway v1.9.2 on kind (
aabbbfc)integrations/envoy-gateway/contains manifests: pasu, anEnvoyExtensionPolicy, and aClientTrafficPolicybuffer limit. The 32 KiB default refused a 100 KiB prompt.With
Bufferedbodies everything fails closed, including pasu scaled to 0. Three settings fail open, and pasu cannot detect them because it is never asked:processingModeleft outfailOpen: trueshadowMode: trueThe README and the manifests name all three. Envoy Gateway was measured before the streamed-mode support; its
Streamedrows say so and were not re-run.Verification
cargo test(224),clippy -D warningswith default features and without, andfmt --checkall pass.Not verified here:
🤖 Generated with Claude Code