Skip to content

test(ci): retain saved-provider routing request metadata - #1803

Open
lilly-luo wants to merge 2 commits into
mainfrom
isaac/lilly.luo/cuj7-routing-evidence-20261120
Open

lilly-luo wants to merge 2 commits into
mainfrom
isaac/lilly.luo/cuj7-routing-evidence-20261120

Conversation

@lilly-luo

@lilly-luo lilly-luo commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What did you change, and why?

CUJ7's saved-provider Codex routing test failed with a gateway 400 wrapping Smart Router returned HTTP 422 (job, request ID 85cf5768-2dbd-4bc6-946d-64da672a3514), while the same test passed on the previous PR head and recent main. Its request recorder held the router inputs only in memory, so the uploaded artifacts could not establish which recipe and offered models produced the rejection.

Before checking each matched routing response, retain its router name, offered model/harness pairs, status and gateway request ID through the existing redacted artifact writer. Exclude prompts, extra option/selector fields, arbitrary headers and response bodies. Preserve every success assertion and all timeouts; this captures evidence for backend correlation rather than retrying or concealing the rejection. Both existing Claude and Codex saved-provider journeys share the check.

How do you know it works?

  • All 298 combined integration-contract and offline CUJ-evidence tests pass, including a new rejected-route case that still raises while retaining the expected metadata and excluding private prompt/header/error/extra-field content.
  • Ruff check and format check pass for tests and the integration runner; diff whitespace and secret hooks pass.
  • Local tests use existing installed application dependencies plus test tools installed through the internal package proxy; no lockfile changes. Initial subprocess import failure was resolved by exposing the checkout and installed dependencies in the disposable test virtualenv.
  • Live CUJ7 passed all 10 tests on executable-change head c86b046 (final 32ab7bb only clarifies the capture lifecycle) (job). Downloaded artifact 14636088792 and verified Claude and Codex routing files contain only the intended metadata with HTTP 200 and gateway request IDs. The negative-control test covers rejected responses; the live HTTP 422 was not reproduced.
  • Existing PRs Add routing diagnostics for transient CI timeouts #1642 and test(ci): retain native Codex request diagnostics #1661 cover production/native routing diagnostics for the separate HTTP 429 failures.

tests/README.md is unchanged.

Checklist before requesting review

  • Comment /ug-review and address the comments. Completed UG review returned APPROVE with zero findings (run 44753980691). Polly found no blockers/security issues; its documentation note was addressed in 32ab7bb and answered in the discussion.
  • Ordinary PR CI runs integration checks; /integration-test will not be posted, as instructed.

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

/ug-review

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Local validation completed: 298 combined integration-contract and CUJ-evidence cases passed. Ruff check/format passed for tests and scripts/run_integration.py. The new negative-control case still rejects HTTP 400 and confirms prompt, credential/header, response-body and extra-field contents do not enter the routing artifact. Live 422 reproduction and artifact delivery remain for PR CI.


This comment was generated with GitHub MCP.

@github-actions

Copy link
Copy Markdown
Contributor

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

@github-actions

Copy link
Copy Markdown
Contributor

UG review recommendation

RECOMMENDATION: APPROVE

Reasons:

  • The review completed with no findings on the changed lines.

Complete review of 2 files; no findings.

Advisory only: this recommendation never approves or merges the pull request. The ready-to-approve label tracks it.

@github-actions github-actions Bot added the ready-to-approve Ready for maintainer approval label Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

None found.

Security vulnerabilities

None found. The added artifact uses explicit field allowlisting and the existing redacted writer; prompts, arbitrary headers and response bodies are excluded.

Non-blocking notes

  • tests/e2e_cuj/README.md:295: can we clarify that capture covers responses reached by validation? Completion-wait failures or an earlier failed assertion can prevent later artifacts.

Approach

The approach is sound: reuse the existing request/response recorder and redacted artifact writer without changing routing, retries, success assertions or timeouts. No new abstraction is needed.

Summary

Diff size (computed): 3 files; +55 / -1 lines (56 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 3 55 1 100.0%
Documentation 0 0 0 0.0%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 0 0 0 0.0%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

Retains routing metadata before saved-provider response validation, adds a focused offline regression case, and documents the artifact. The changes are proportional to the diagnostic goal. No concrete code consolidation is needed; the documentation qualification above would align its promise with the capture lifecycle.

Tests

Test-by-test assessment
Test / case Behavior protected Layer Needed? Action / rationale
tests/test_cuj_evidence.py::test_failed_saved_provider_route_retains_metadata_without_prompt_or_credentials A rejected response writes allowlisted metadata before raising, while excluding private fields. Offline component keep Directly protects the regression using the real artifact writer. Existing session tests separately cover credential redaction; no duplicate setup framework is needed.
tests/e2e_cuj/test_cuj7_model_discovery.py::_assert_saved_provider_routing Records each reached request/response pair before existing validation. Shared E2E assertion helper keep Reuses the existing correlation boundary for both journeys and preserves their assertions.
tests/e2e_cuj/test_cuj7_model_discovery.py::TestUnmanagedModelDiscovery::test_ug_claude_saved_provider_smart_routing Claude’s saved-provider routing and completed native interaction. Live E2E keep Distinct Claude process/TUI boundary; the shared helper adds evidence without another journey.
tests/e2e_cuj/test_cuj7_model_discovery.py::TestUnmanagedModelDiscovery::test_ug_codex_saved_provider_smart_routing Codex’s saved-provider routing and completed native interaction. Live E2E keep Distinct Codex native boundary; existing success coverage remains intact.
tests/e2e_cuj/README.md — routing-evidence documentation Describes retained fields and excluded private content. Documentation keep Appropriate location; qualify capture coverage as noted above.

Existing happy-path journeys and the new failure case cover the scoped change without another live test. Live reproduction, actual gateway request-ID correlation and uploaded failure-artifact delivery remain unverified.

The focused offline test could not reach collection because rpds-py==2026.6.3 was unavailable in the offline cache; no pytest pass is claimed. To verify locally, run uv run pytest tests/test_cuj_evidence.py tests/test_cuj_session.py. In an authorized live run, inspect the uploaded routing-request-*.json files for the expected metadata and absence of private content.

Scope

All changes support retaining saved-provider routing evidence. No unrelated changes were found. This adds diagnostics, not a fix for the backend routing rejection.


Automated review by Polly · workflow run

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Live CUJ7 validation passed all 10 tests on c86b046. I downloaded artifact14636088792 and verified both saved-provider Claude and Codex journeys delivered routing-request JSON containing only router_name, model/harness pairs, status_code and a gateway request_id; both responses were HTTP200. The formerly failing Codex case selected gpt-6-luna through recipe m2-r315-quality-20260929. Failure-path artifact preservation remains covered by the offline negative control. Broad CI currently has an unrelated Windows Codex desktop Get-AppxPackage timeout before inference; it will be rerun after the workflow finishes.


This comment was generated with GitHub MCP.

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Addressed Polly's documentation note in 32ab7bb: the CUJ guide now states that capture occurs after task completion, as responses reach validation, and that earlier failures can prevent later artifacts. No executable code changed after the reviewed c86b046 head. That head passed all 10 live CUJ7 cases and delivered the verified metadata artifact; the documentation push starts a fresh CI run, including the previously timed-out Windows package query.


This comment was generated with GitHub MCP.

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Final-head CI completed with five direct failures, all confirmed upstream HTTP429 rate limits: Anthropic relay Cowork on macOS/Windows, Claude relay full journeys on Linux/Windows, and native Codex managed routing. The CUJ7 routing change is unchanged from its live-validated executable head. I requested a failed-only rerun of run44755740715 on the same 32ab7bb commit; passing jobs are preserved. No tests, timeouts, or success conditions were weakened.


This comment was generated with GitHub MCP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants