Repository navigation
test(codex): cover desktop skill and MCP execution - #1757
andy-xu-db wants to merge 5 commits into
Conversation
|
/ug-review |
|
Stamping to unblock - I think there were some test issues which we discussed on slack. Recommend to fix those before merging |
| deadline = time.monotonic() + 60 | ||
| while not completed_task_models(session, "codex", marker): | ||
| assert time.monotonic() < deadline, "The app never recorded the completed skill turn" | ||
| time.sleep(1) | ||
| assert_desktop_skill_mcp_session(session, query, marker, MANAGED_CODEX_MODEL) |
| assert len(sessions) == 1, f"Expected only the desktop session, got {list(sessions)}" | ||
| assert_skill_mcp_turn(next(iter(sessions.values())), query, marker, model) |
There was a problem hiding this comment.
| def test_native_mcp_receipt_proves_execution_including_code_mode(records): | ||
| # Native receipts establish execution even when functions.exec hides the direct model call. |
UG review recommendationRECOMMENDATION: APPROVE Reasons:
Findings considered: 1 (severities: low: 1); see the inline review comments. Advisory only: this recommendation never approves or merges the pull request. The |
Polly AI ReviewBlocking issuesNone found. Security vulnerabilitiesNone found in the changed code. Non-blocking notes
ApproachShared evidence parsing with platform-specific GUI journeys is sound. Combining each journey with baseline inference would save an app launch but couple baseline coverage to MCP access. Separate cases are reasonable. SummaryDiff size (computed): 7 files; +325 / -7 lines (332 changed text lines); 0 binary files.
Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements. ChangesAdds local skill and MCP execution journeys for Codex Desktop on macOS and Windows, shared evidence helpers, offline validation, collection contracts, and coverage documentation. No application code or dependencies change. The size is proportional to the GUI boundaries and evidence checks. Shared parsing already avoids duplication; no substantial removal is justified. TestsTest-by-test assessmentThe frozen snapshot’s checksum was verified. Tests were not executed during this review. The reported offline passes are author-provided evidence; native GUI execution, actual MCP receipt schema, code-mode execution, and CI identity access remain unverified. Native receipts correlate tool arguments and results, but do not establish raw gateway HTTP status or downstream usage attribution.
Existing baseline inference journeys remain unchanged and retain their diagnostic/model-picker checks. Timeout rollout capture is present in both new journeys. No additional live failure matrix is needed; parser failures belong in the offline cases. To verify native behavior, run both required desktop jobs and macOS harness-latest in authorized CI. Confirm both cases are selected per lane, then inspect ScopeAll changes support desktop local-skill/MCP execution coverage or its documentation. No unrelated changes were found. UC skill download and connector sign-in remain explicitly outside scope. The supplied base/head objects were unavailable locally, so their relationship to the checksum-verified snapshot was not independently verified. Automated review by Polly · workflow run |
|
/ug-review |
| deadline = time.monotonic() + timeout | ||
| path = app.session.home / ".codex/config.toml" | ||
| while True: | ||
| mode = tomllib.loads(path.read_text(encoding="utf-8")).get("windows", {}).get("sandbox") |
There was a problem hiding this comment.
what happens if the app rewrites config.toml while we poll? a partial read raises TOMLDecodeError (missing file raises FileNotFoundError) and ends the 180s wait right away. can we treat both as "not ready yet" and keep polling?
Suggestion:
| mode = tomllib.loads(path.read_text(encoding="utf-8")).get("windows", {}).get("sandbox") | |
| try: | |
| config = tomllib.loads(path.read_text(encoding="utf-8")) | |
| except (FileNotFoundError, tomllib.TOMLDecodeError): | |
| config = {} | |
| mode = config.get("windows", {}).get("sandbox") |
Polly AI ReviewBlocking issuesNone identified. Security vulnerabilitiesNone identified. Registration forwards authentication environment names rather than credential values. No demonstrated approval-boundary bypass was found. Non-blocking notesNone. ApproachThe approach is sound: reuse the existing MCP config writer, share desktop orchestration, and reserve native journeys for behavior that component tests cannot establish. No concrete simplification clearly preserves the same coverage with less setup. SummaryDiff size (computed): 12 files; +675 / -26 lines (701 changed text lines); 0 binary files.
Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements. ChangesAdds local authored-skill and MCP execution coverage to Codex Desktop on macOS and Windows, with the registration and submission fixes needed to complete those journeys. The diff also includes shared automation helpers, regression coverage, diagnostics, and coverage documentation. No dependencies or generated assets change. The size is justified by the two native platform boundaries and their evidence-validation logic. No clearly redundant scenario or removable scaffolding was established. TestsTest-by-test assessmentAssessment is static. Tests were not executed, and the reported CI results and native artifacts were not independently verified. Both trusted review policies were read in full. The referenced helper catalog was unavailable; existing helpers and nearest scenarios were inspected instead. The native receipts correlate request arguments, successful responses, and the completed model turn. They do not independently verify gateway-side usage records or attribution; this PR does not claim that coverage. No concrete missing scenario was established after checking existing coverage.
No tests are removed. Existing inference assertions remain, including Windows model-picker and macOS failed-click coverage. For manual confirmation on each supported desktop, run the documented journey and verify that the draft is submitted visibly, only the expected search receives Allow once, and the hidden token appears after completion. On Windows, also verify visible sandbox setup and persisted elevated mode. ScopeAll changes support the stated desktop skill/MCP outcome or the registration, submission, and diagnostics needed to exercise it reliably. No unrelated changes or unresolved scope questions were found. Automated review by Polly · workflow run |
e615203 to
3b12e72
Compare
Add native macOS and Windows skill journeys with hidden reference inputs, verified tool approval and successful same-turn MCP receipts. Preserve the validated desktop setup and explicit Send fixes.
3b12e72 to
adefaaf
Compare
|
/ug-review |
| "test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_completes_a_gateway_turn", | ||
| "test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_skill_calls_mcp", |
| assert _collect_integration_cases(_ci_marker_expression(job, "codex")) == { | ||
| "test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_completes_a_gateway_turn" | ||
| "test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_completes_a_gateway_turn", | ||
| "test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_skill_calls_mcp", |
Polly AI ReviewBlocking issues
Security vulnerabilities
Non-blocking notes
ApproachThe native journeys establish boundaries mocks cannot: desktop input, skill loading, approval, and native MCP receipts. A concrete simplification is placing receipt parsing beside the existing helpers in SummaryDiff size (computed): 10 files; +732 / -23 lines (755 changed text lines); 0 binary files.
Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements. ChangesThe PR adds desktop skill/MCP journeys, OAuth setup and cleanup, native evidence validation, platform input support, component checks, and coverage documentation. No production implementation, dependency, or generated-file changes were found. The breadth supports the intended cross-platform outcome. Shared journey helpers avoid duplicating most logic; consolidating native receipt parsing with the existing Codex evidence helpers would reduce maintenance overhead. TestsTest-by-test assessmentThe reported 190 passing checks were not independently executed or collected at the supplied head. Native macOS/Windows execution and current-head OAuth/keyring cleanup remain unverified. Offline function probes reproduced the approval and prompt-correlation gaps described above. Existing Codex interaction helpers and neighboring evidence tests were inspected. The referenced
Native receipts would establish local tool-call/result evidence, not independently verify gateway HTTP status/headers or downstream usage attribution. Those remain unverified. To verify the fixes, extend the existing component scenarios with the two synthetic triggers above. Then run each desktop journey on its target OS with browser OAuth available, checking the actual approval’s server/tool/query, completed-turn evidence, and profile logout/keyring cleanup. ScopeAll changed files support desktop skill/MCP execution or its submission, setup, diagnostics and documentation. No clearly unrelated changes were found. The blocking issue is CI readiness for the intended journey, not an unrelated outcome. Automated review by Polly · workflow run |
Polly AI Review skipped
Push a commit before requesting another review. |
|
/ug-review |
| "--skip-upgrade", | ||
| timeout=240, | ||
| ) | ||
| session.run("mcp", "add", "--agents", "codex", "--names", MCP_SERVICE, timeout=240) |
|
/ug-review |
| session = live_session | ||
| profile = configure_databricks_token_profile(session, workspace) | ||
| marker = fresh_id() | ||
| query = f"Databricks Unity Catalog documentation {fresh_id()}" | ||
| write_search_skill(session.home, query, marker) | ||
| use_managed_config_fixture(session, "managed_workspace_default") |
🥞 Stacked PR
Use this link to review incremental changes.
Summary
Add skill/MCP journeys to the existing macOS and Windows Codex desktop tests. Both reuse the suite's existing CI credential in a disposable Databricks CLI token profile before configuring Codex and adding
system.ai.web_search. This PR targetsmainand has no production changes or dependency on #1788.The public
databricks configurecommand saves the token from redacted stdin;databricks current-user meverifies the profile. UG selects its existing--profile ... --use-patsaved-token mode, also used by the Cowork connector tests. The desktop session has no inherited bearer override, so the MCP proxy must load the credential from the profile. No new PAT is minted, and no browser login or OAuth cache is fabricated.Coverage
system.ai.gpt-5-6-sol, plus successful native MCP receipts carrying the exact query in that same turn. Check every matching call.Validation
c1235eb7ec6e93f4c03a8eb125abcac8777e4d49. Native macOS/Windows CI is pending. Earlier passes and login timeouts do not validate this saved-token setup.src/ucode/mcp.py.Limitations
This covers adding and using MCPs with an authenticated saved-token profile. The token comes from the same CI authentication setup as inference;
--use-patselects UG's existing static-token reader even when the supplied bearer was minted through OAuth. Browser sign-in, CLI OAuth-cache lookup, and token refresh are outside coverage. Token expiry fails the journey. The profile exists only in the disposable home and is excluded from uploaded evidence.UC skill downloads, connector-specific browser sign-in, and search ranking are outside coverage. The macOS app may auto-update after installation. Selecting the desktop root when child rollouts exist remains a follow-up.