Skip to content

test(codex): cover desktop skill and MCP execution - #1757

Open
andy-xu-db wants to merge 5 commits into
mainfrom
andy/codex-desktop-mcp-skills-e2e
Open

andy-xu-db wants to merge 5 commits into
mainfrom
andy/codex-desktop-mcp-skills-e2e

Conversation

@andy-xu-db

@andy-xu-db andy-xu-db commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

🥞 Stacked PR

Use this link to review incremental changes.


Summary

Add skill/MCP journeys to the existing macOS and Windows Codex desktop tests. Both reuse the suite's existing CI credential in a disposable Databricks CLI token profile before configuring Codex and adding system.ai.web_search. This PR targets main and has no production changes or dependency on #1788.

The public databricks configure command saves the token from redacted stdin; databricks current-user me verifies the profile. UG selects its existing --profile ... --use-pat saved-token mode, also used by the Cowork connector tests. The desktop session has no inherited bearer override, so the MCP proxy must load the credential from the profile. No new PAT is minted, and no browser login or OAuth cache is fabricated.

Coverage

  • Invoke a local authored skill whose reference contains a random query and completion token absent from the GUI prompt.
  • Require the token on screen and in a completed Codex Desktop/Databricks turn on system.ai.gpt-5-6-sol, plus successful native MCP receipts carrying the exact query in that same turn. Check every matching call.
  • Verify the rendered draft, click Send, and require native submission. Approve only the expected server, tool, and query through Allow once.
  • Complete Windows sandbox setup through visible controls and verify the app saved elevated mode; missing or partial config reads wait within the existing deadline. One visible readiness recheck handles its restart error before submission.
  • Retain the original inference cases. Component checks cover evidence validation, setup failures, removal of inherited credential overrides, and redaction of token stdin/stdout/stderr.

Validation

  • 191 passed in desktop evidence and integration-contract component checks.
  • Ruff check/format and diff checks passed; the real CLI saved an inert token at the isolated default profile location in a local setup check.
  • Head: c1235eb7ec6e93f4c03a8eb125abcac8777e4d49. Native macOS/Windows CI is pending. Earlier passes and login timeouts do not validate this saved-token setup.
  • Verified no production diff from the base, including src/ucode/mcp.py.

Limitations

This covers adding and using MCPs with an authenticated saved-token profile. The token comes from the same CI authentication setup as inference; --use-pat selects UG's existing static-token reader even when the supplied bearer was minted through OAuth. Browser sign-in, CLI OAuth-cache lookup, and token refresh are outside coverage. Token expiry fails the journey. The profile exists only in the disposable home and is excluded from uploaded evidence.

UC skill downloads, connector-specific browser sign-in, and search ranking are outside coverage. The macOS app may auto-update after installation. Selecting the desktop root when child rollouts exist remains a follow-up.

@andy-xu-db
andy-xu-db requested a review from masonc08 November 18, 2026 21:45
@andy-xu-db

Copy link
Copy Markdown
Collaborator Author

/ug-review

@masonc08

Copy link
Copy Markdown
Collaborator

Stamping to unblock - I think there were some test issues which we discussed on slack. Recommend to fix those before merging

Comment on lines +121 to +125
deadline = time.monotonic() + 60
while not completed_task_models(session, "codex", marker):
assert time.monotonic() < deadline, "The app never recorded the completed skill turn"
time.sleep(1)
assert_desktop_skill_mcp_session(session, query, marker, MANAGED_CODEX_MODEL)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintainability · low
can we move this poll loop into assert_desktop_skill_mcp_session? the macOS and Windows tests copy the same setup and wait loop, so the two can drift apart.

Comment on lines +75 to +76
assert len(sessions) == 1, f"Expected only the desktop session, got {list(sessions)}"
assert_skill_mcp_turn(next(iter(sessions.values())), query, marker, model)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · medium
can we avoid requiring exactly one rollout? if Codex starts a subagent anyway (the skill only asks it not to), this fails even though the turn and MCP receipt are fine. AGENTS.md says not to pin subagent counts. could we pick the Codex Desktop root session instead?

Comment on lines +66 to +67
def test_native_mcp_receipt_proves_execution_including_code_mode(records):
# Native receipts establish execution even when functions.exec hides the direct model call.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · low
no record here models a functions.exec code-mode turn; it's just a duplicate receipt. can we rename this to say it allows extra successful calls, or add a real code-mode record shape?

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UG review recommendation

RECOMMENDATION: APPROVE

Reasons:

  • The only finding (tests/integration/test_ug_codex_desktop_windows.py, lines 116-121) is low-severity maintainability feedback.
  • It asks to move the profile/skill/configure/mcp setup, the prompt string and the 60s completed-turn wait into utils/codex_desktop.py, because that code is copied from the macOS test. This is a refactoring suggestion, not a defect.
  • No finding points to data loss, credential or cross-workspace leakage, broken workflows, a behavioral bug, unsafe rollout, or missing regression coverage.
  • The duplication risk can be handled in a follow-up refactor without blocking this change.

Findings considered: 1 (severities: low: 1); see the inline review comments.

Advisory only: this recommendation never approves or merges the pull request. The ready-to-approve label tracks it.

@github-actions github-actions Bot added the ready-to-approve Ready for maintainer approval label Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

None found.

Security vulnerabilities

None found in the changed code.

Non-blocking notes

  • tests/integration/utils/codex_desktop.py:60: can we require one exact-query receipt while allowing successful refined-query calls, and extend test_native_mcp_receipt_proves_execution_including_code_mode to cover this?

Approach

Shared evidence parsing with platform-specific GUI journeys is sound. Combining each journey with baseline inference would save an app launch but couple baseline coverage to MCP access. Separate cases are reasonable.

Summary

Diff size (computed): 7 files; +325 / -7 lines (332 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 7 325 7 100.0%
Documentation 0 0 0 0.0%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 0 0 0 0.0%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

Adds local skill and MCP execution journeys for Codex Desktop on macOS and Windows, shared evidence helpers, offline validation, collection contracts, and coverage documentation. No application code or dependencies change.

The size is proportional to the GUI boundaries and evidence checks. Shared parsing already avoids duplication; no substantial removal is justified.

Tests

Test-by-test assessment

The frozen snapshot’s checksum was verified. Tests were not executed during this review. The reported offline passes are author-provided evidence; native GUI execution, actual MCP receipt schema, code-mode execution, and CI identity access remain unverified. Native receipts correlate tool arguments and results, but do not establish raw gateway HTTP status or downstream usage attribution.

Test / case Behavior protected Layer Needed? Action / rationale
tests/README.md — coverage table and desktop coverage notes Documents supported journeys and exclusions Documentation keep Records coverage boundaries without treating offline checks as native execution.
tests/integration/README.md — journey, index, and harness-latest updates Documents prerequisites, execution evidence, and platform selection Documentation keep Supports reproduction and explains local-skill versus UC-download coverage.
tests/integration/test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_skill_calls_mcp Public MCP registration, skill discovery, visible token, and correlated completed managed-model turn Native GUI E2E keep macOS Accessibility, native app behavior, and MCP transport require this boundary.
tests/integration/test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_skill_calls_mcp Equivalent journey through the packaged Windows app Native GUI E2E keep Windows UI Automation and configuration are distinct from macOS and parser coverage.
tests/integration/utils/codex_desktop.py — constants, write_search_skill, assert_skill_mcp_turn, assert_desktop_skill_mcp_session Generates hidden reference inputs and validates same-turn completion and receipts Shared setup/evidence helper keep Reuses existing rollout/model parsing; allow refined-query successes while retaining one required exact-query receipt.
tests/test_codex_desktop_evidence.py::records Supplies explicit metadata, context, receipt, and completion inputs Unit fixture keep Small offline input fixture; does not simulate a live service.
tests/test_codex_desktop_evidence.py::test_skill_values_are_only_in_the_reference_file Query and completion token remain outside SKILL.md Filesystem unit keep Protects the hidden-input premise independently of model behavior.
tests/test_codex_desktop_evidence.py::test_native_mcp_receipt_proves_execution_including_code_mode Accepts native receipts without direct-call records and permits multiple successes Parser unit keep Extend the second receipt with a different successful query; actual code-mode execution remains unverified.
tests/test_codex_desktop_evidence.py::test_wrong_origin_model_turn_or_answer_fails — eight variants Rejects wrong origin/provider/model, mismatched turns, incomplete events, and nonexact answers Parser unit keep Each variant protects a distinct false-positive boundary.
tests/test_codex_desktop_evidence.py::test_every_matching_mcp_receipt_must_succeed — eleven variants Rejects wrong receipt identity/query, incomplete or failed calls, errors, and empty results Parser unit keep Fault inputs belong offline; retain rejection when no exact-query receipt exists.
tests/test_codex_desktop_evidence.py::test_success_cannot_hide_another_failed_matching_call One success cannot mask another failed matching call Parser unit keep Protects the mixed-result boundary absent from single-receipt mutations.
tests/test_codex_desktop_evidence.py::test_final_answer_alone_cannot_prove_tool_use Exact completion token without MCP evidence fails Parser unit keep Directly protects against treating an answer as execution evidence.
tests/test_integration_contract.py::test_codex_desktop_macos_ci_is_blocking_and_installs_a_pinned_app Required pinned macOS lane selects both journeys Collection/workflow contract keep Extends the existing selection contract rather than adding parallel scaffolding.
tests/test_integration_contract.py::test_codex_desktop_windows_ci_is_blocking_and_installs_a_pinned_app Required pinned Windows lane selects both journeys Collection/workflow contract keep Protects the separate Windows marker/job boundary.
tests/test_integration_contract.py::test_latest_harness_runs_the_newest_signed_chatgpt_app macOS harness-latest selects both journeys Collection/workflow contract keep Protects latest-build selection separately from pinned required jobs.

Existing baseline inference journeys remain unchanged and retain their diagnostic/model-picker checks. Timeout rollout capture is present in both new journeys. No additional live failure matrix is needed; parser failures belong in the offline cases.

To verify native behavior, run both required desktop jobs and macOS harness-latest in authorized CI. Confirm both cases are selected per lane, then inspect agent-sessions.json for the exact token, managed model/provider, and successful query receipts sharing the completed turn ID.

Scope

All changes support desktop local-skill/MCP execution coverage or its documentation. No unrelated changes were found. UC skill download and connector sign-in remain explicitly outside scope. The supplied base/head objects were unavailable locally, so their relationship to the checksum-verified snapshot was not independently verified.


Automated review by Polly · workflow run

@andy-xu-db

Copy link
Copy Markdown
Collaborator Author

/ug-review

deadline = time.monotonic() + timeout
path = app.session.home / ".codex/config.toml"
while True:
mode = tomllib.loads(path.read_text(encoding="utf-8")).get("windows", {}).get("sandbox")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
what happens if the app rewrites config.toml while we poll? a partial read raises TOMLDecodeError (missing file raises FileNotFoundError) and ends the 180s wait right away. can we treat both as "not ready yet" and keep polling?

Suggestion:

Suggested change
mode = tomllib.loads(path.read_text(encoding="utf-8")).get("windows", {}).get("sandbox")
try:
config = tomllib.loads(path.read_text(encoding="utf-8"))
except (FileNotFoundError, tomllib.TOMLDecodeError):
config = {}
mode = config.get("windows", {}).get("sandbox")

@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

None identified.

Security vulnerabilities

None identified. Registration forwards authentication environment names rather than credential values. No demonstrated approval-boundary bypass was found.

Non-blocking notes

None.

Approach

The approach is sound: reuse the existing MCP config writer, share desktop orchestration, and reserve native journeys for behavior that component tests cannot establish. No concrete simplification clearly preserves the same coverage with less setup.

Summary

Diff size (computed): 12 files; +675 / -26 lines (701 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 11 673 24 99.4%
Documentation 0 0 0 0.0%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 1 2 2 0.6%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

Adds local authored-skill and MCP execution coverage to Codex Desktop on macOS and Windows, with the registration and submission fixes needed to complete those journeys. The diff also includes shared automation helpers, regression coverage, diagnostics, and coverage documentation. No dependencies or generated assets change.

The size is justified by the two native platform boundaries and their evidence-validation logic. No clearly redundant scenario or removable scaffolding was established.

Tests

Test-by-test assessment

Assessment is static. Tests were not executed, and the reported CI results and native artifacts were not independently verified. Both trusted review policies were read in full. The referenced helper catalog was unavailable; existing helpers and nearest scenarios were inspected instead.

The native receipts correlate request arguments, successful responses, and the completed model turn. They do not independently verify gateway-side usage records or attribution; this PR does not claim that coverage. No concrete missing scenario was established after checking existing coverage.

Test / case Behavior protected Layer Needed? Action / rationale
tests/integration/test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_completes_a_gateway_turn Rendered draft, explicit Send, native submission, and completed inference Native E2E keep Preserves baseline inference independently of skill/MCP setup and retains failed-click diagnostics.
tests/integration/test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_skill_calls_mcp Skill discovery, hidden reference inputs, approval, successful MCP receipts, and final token in one turn Native E2E keep Uniquely exercises macOS accessibility, desktop skill loading, and native MCP startup.
tests/integration/test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_completes_a_gateway_turn Explicit submission and completed inference with the existing model-picker assertions Native E2E keep Preserves baseline coverage across the distinct Windows UI Automation boundary.
tests/integration/test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_skill_calls_mcp Visible sandbox setup, readiness recovery, skill discovery, approval, and correlated MCP completion Native E2E keep Component fixtures cannot establish native sandbox persistence or app-server restart behavior.
tests/test_codex_desktop_evidence.py::test_windows_readiness_recheck_is_visible_and_bounded[ready] Already-ready app requires no retry Component keep Detects unnecessary recovery actions.
tests/test_codex_desktop_evidence.py::test_windows_readiness_recheck_is_visible_and_bounded[recovers] One visible recheck clears the readiness error Component keep Protects the recovery branch without requiring a live restart.
tests/test_codex_desktop_evidence.py::test_windows_readiness_recheck_is_visible_and_bounded[fails] Persistent readiness error fails within the bound Component keep Prevents continuing after unsuccessful recovery.
tests/test_codex_desktop_evidence.py::test_windows_setup_requires_the_native_elevated_mode[None, unelevated] Missing or incorrect saved sandbox mode fails Component/filesystem keep Both cases reject inadequate setup evidence.
tests/test_codex_desktop_evidence.py::test_windows_setup_requires_the_native_elevated_mode[elevated] Persisted elevated mode establishes setup completion Component/filesystem keep Protects the successful filesystem oracle; native setup remains separately exercised.
tests/test_codex_desktop_evidence.py::test_gui_approves_only_the_requested_search[None] — both role pairs Expected approval permits Allow once Component keep Exercises the success branch for both platform role contracts.
tests/test_codex_desktop_evidence.py::test_gui_approves_only_the_requested_search[server, tool] — both role pairs Wrong server or tool cannot receive approval Component keep Protects approval identity before clicking.
tests/test_codex_desktop_evidence.py::test_gui_approves_only_the_requested_search[query, echoed_query] — both role pairs Wrong query or an earlier transcript echo cannot authorize the call Component keep Protects exact-query validation and its positional boundary.
tests/test_codex_desktop_evidence.py::test_gui_approves_only_the_requested_search[ambiguous] — both role pairs Multiple Allow once controls fail without clicking Component keep Covers control ambiguity independently of argument validation.
tests/test_codex_desktop_evidence.py::test_submission_requires_the_native_user_prompt — four parameterized cases Only the exact native user message establishes submission Component keep Rejects assistant echoes, another prompt, and the wrong event envelope at the smallest reliable layer.
tests/test_codex_desktop_evidence.py::test_skill_values_are_only_in_the_reference_file Query and token remain outside the main skill instructions Component/filesystem keep Detects accidental disclosure that successful native execution alone would miss.
tests/test_codex_desktop_evidence.py::test_native_mcp_receipt_proves_execution_including_code_mode Receipt-only execution evidence works and extra successful calls are allowed Component keep Protects code-mode compatibility without brittle call counts.
tests/test_codex_desktop_evidence.py::test_wrong_origin_model_turn_or_answer_fails — origin/provider mutations Other clients or providers cannot impersonate the desktop session Component keep Protects desktop-session identity.
tests/test_codex_desktop_evidence.py::test_wrong_origin_model_turn_or_answer_fails — model/context-turn mutations Completion joins to the expected model context Component keep Protects composition with the existing completed-model parser.
tests/test_codex_desktop_evidence.py::test_wrong_origin_model_turn_or_answer_fails — receipt-turn/event mutations Receipt is completed within the answer’s turn Component keep Prevents borrowing incomplete or unrelated-turn evidence.
tests/test_codex_desktop_evidence.py::test_wrong_origin_model_turn_or_answer_fails — completion/answer mutations Only the exact completed answer establishes success Component keep Rejects intermediate messages and nonexact tokens.
tests/test_codex_desktop_evidence.py::test_every_matching_mcp_receipt_must_succeed — type/server/tool/query mutations Required evidence identifies the exact MCP operation and hidden query Component keep Prevents unrelated tool receipts from satisfying the execution contract.
tests/test_codex_desktop_evidence.py::test_every_matching_mcp_receipt_must_succeed — unfinished/failed/error mutations Unsuccessful matching calls fail validation Component keep Protects native call-status boundaries independently of the final answer.
tests/test_codex_desktop_evidence.py::test_every_matching_mcp_receipt_must_succeed — missing/empty/blank result mutations Absent response content cannot prove execution Component keep Exercises the response-presence boundary.
tests/test_codex_desktop_evidence.py::test_every_matching_mcp_receipt_must_succeed — isError mutation MCP protocol errors fail despite text content Component keep Covers a distinct error signal from native call status.
tests/test_codex_desktop_evidence.py::test_success_cannot_hide_another_failed_matching_call Every matching call must succeed Component keep Specifically catches validation that stops after finding one successful receipt.
tests/test_codex_desktop_evidence.py::test_final_answer_alone_cannot_prove_tool_use Correct token without an MCP receipt fails Component keep Protects against accepting final-answer evidence alone.
tests/test_codex_desktop_evidence.py::records Shared valid session, context, receipt, and completion records Fixture keep Minimal baseline; individual tests retain their mutations and assertions.
tests/test_integration_contract.py::test_codex_desktop_macos_ci_is_blocking_and_installs_a_pinned_app Required macOS selection includes both journeys Static contract keep Checks workflow selection without claiming live execution.
tests/test_integration_contract.py::test_codex_desktop_windows_ci_is_blocking_and_installs_a_pinned_app Required Windows selection includes both journeys Static contract keep Protects the separate Windows workflow declaration.
tests/test_integration_contract.py::test_latest_harness_runs_the_newest_signed_chatgpt_app Latest-harness selection includes the macOS skill journey Static contract keep Covers a separate workflow while preserving existing signature checks.
tests/test_mcp.py::TestAddCodexMcpServer::test_registration_forwards_auth_names_without_persisting_credentials Installed-CLI registration forwards auth names, excludes values, replaces stale transport, and preserves unrelated config Component/filesystem keep Existing writer tests do not cover this changed dispatch path.
tests/integration/utils/codex_desktop.py:16 — readiness and sandbox helpers Bounded visible recovery and app-persisted setup evidence Test support keep Shared platform orchestration is exercised by focused branch tests and the Windows journey.
tests/integration/utils/codex_desktop.py:55 — approval and submission helpers Validated Allow once and native user-message submission Test support keep Centralizes shared orchestration while retaining negative component coverage.
tests/integration/utils/codex_desktop.py:110 — skill construction Separately stored hidden inputs Test support keep Avoids duplicating scenario setup across platforms.
tests/integration/utils/codex_desktop.py:129 — turn/session assertions Same-session, same-turn model completion and successful MCP evidence Test support keep Reuses existing model parsing and adds scenario-specific receipt validation.
tests/integration/utils/macos_app.py:89 — type_line Optional Return allows draft inspection before Send Native input primitive keep Preserves default behavior and supports the changed native submission sequence.
tests/integration/utils/windows_app.py:59 — diagnostics Redacted MCP/setup evidence survives failures Artifact support keep Adds diagnosis without treating logs alone as execution proof.
tests/integration/utils/windows_app.py:119 — type_line Optional Enter allows explicit Send Native input primitive keep Preserves existing callers and supports draft verification.
tests/integration/utils/windows_uia.ps1:13 — UTF-8 output and optional Enter Correct UI-label decoding and controlled submission Native transport/input primitive keep These platform boundaries are exercised through the Windows journeys.
tests/README.md — changed coverage entries and desktop documentation Documents registration coverage, native evidence, and exclusions Documentation keep Keeps the suite-level coverage map aligned with the added behavior.
tests/integration/README.md — changed desktop journey documentation Defines setup, approval, receipt requirements, prerequisites, and exclusions Documentation keep Provides the detailed execution contract and run guidance.

No tests are removed. Existing inference assertions remain, including Windows model-picker and macOS failed-click coverage.

For manual confirmation on each supported desktop, run the documented journey and verify that the draft is submitted visibly, only the expected search receives Allow once, and the hidden token appears after completion. On Windows, also verify visible sandbox setup and persisted elevated mode.

Scope

All changes support the stated desktop skill/MCP outcome or the registration, submission, and diagnostics needed to exercise it reliably. No unrelated changes or unresolved scope questions were found.


Automated review by Polly · workflow run

@andy-xu-db
andy-xu-db force-pushed the andy/codex-desktop-mcp-skills-e2e branch from e615203 to 3b12e72 Compare November 20, 2026 00:01
@andy-xu-db
andy-xu-db changed the base branch from main to andy-xu_data/stack/andy/codex-mcp-auth-env November 20, 2026 00:02
Add native macOS and Windows skill journeys with hidden reference inputs, verified tool approval and successful same-turn MCP receipts. Preserve the validated desktop setup and explicit Send fixes.
@andy-xu-db
andy-xu-db force-pushed the andy/codex-desktop-mcp-skills-e2e branch from 3b12e72 to adefaaf Compare November 20, 2026 04:58
@andy-xu-db
andy-xu-db changed the base branch from andy-xu_data/stack/andy/codex-mcp-auth-env to main November 20, 2026 04:58
@andy-xu-db

Copy link
Copy Markdown
Collaborator Author

/ug-review

Comment on lines +151 to +152
"test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_completes_a_gateway_turn",
"test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_skill_calls_mcp",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · high
integration README says these need browser OAuth and the CI service principal can't do it. won't this make the blocking macOS/Windows/latest jobs fail every run? can we gate or mark them non-blocking?

assert _collect_integration_cases(_ci_marker_expression(job, "codex")) == {
"test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_completes_a_gateway_turn"
"test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_completes_a_gateway_turn",
"test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_skill_calls_mcp",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · high
same here: the Windows desktop job is blocking, but this test needs an interactive databricks auth login. can we keep it out of the required job until the runner can actually log in?

@github-actions github-actions Bot removed the ready-to-approve Ready for maintainer approval label Nov 20, 2026
@andy-xu-db
andy-xu-db marked this pull request as draft November 20, 2026 05:04
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

  • Required CI cannot complete the new login. tests/integration/test_ug_codex_desktop_macos.py:125, tests/integration/test_ug_codex_desktop_windows.py:116
    can we provision browser authentication or move these journeys out of required unattended jobs? they discard the runner bearer, then require interactive OAuth.

    Both cases inherit the existing desktop job selectors. Those jobs supply service-principal credentials, not a browser user, and feed the required aggregate gate. Login therefore times out before desktop/MCP validation. The new selection assertions in tests/test_integration_contract.py:152 and tests/test_integration_contract.py:171 enforce this prerequisite mismatch.

Security vulnerabilities

  • Approval validation can click an unrelated dialog. tests/integration/utils/codex_desktop.py:97
    can we bind validation to the active approval dialog? transcript text matching the expected title and query currently lets an unrelated dialog reach Allow once.

    An offline probe reproduced the click with an expected title/query earlier in the flattened tree, followed by a different server/tool approval. This weakens the test harness’s consent boundary; it is not evidence of a production authorization bypass or actual execution of an unrelated tool.

Non-blocking notes

  • tests/integration/utils/codex_desktop.py:114, tests/integration/utils/codex_desktop.py:156
    can we correlate the submitted prompt with the answering turn? an earlier matching prompt and a later unrelated turn currently pass both checks.

    The single submission and fresh hidden query/token reduce the practical risk. Extend tests/test_codex_desktop_evidence.py::test_wrong_origin_model_turn_or_answer_fails with that cross-turn case.

  • tests/test_codex_desktop_evidence.py:113
    can we extend test_gui_approves_only_the_requested_search with an expected transcript title/query followed by an unrelated live approval?

Approach

The native journeys establish boundaries mocks cannot: desktop input, skill loading, approval, and native MCP receipts. A concrete simplification is placing receipt parsing beside the existing helpers in tests/integration/utils/agents/codex.py, rather than maintaining a separate evidence-parser location.

Summary

Diff size (computed): 10 files; +732 / -23 lines (755 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 10 732 23 100.0%
Documentation 0 0 0 0.0%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 0 0 0 0.0%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

The PR adds desktop skill/MCP journeys, OAuth setup and cleanup, native evidence validation, platform input support, component checks, and coverage documentation. No production implementation, dependency, or generated-file changes were found.

The breadth supports the intended cross-platform outcome. Shared journey helpers avoid duplicating most logic; consolidating native receipt parsing with the existing Codex evidence helpers would reduce maintenance overhead.

Tests

Test-by-test assessment

The reported 190 passing checks were not independently executed or collected at the supplied head. Native macOS/Windows execution and current-head OAuth/keyring cleanup remain unverified. Offline function probes reproduced the approval and prompt-correlation gaps described above.

Existing Codex interaction helpers and neighboring evidence tests were inspected. The referenced tests/_helpers/README.md is absent from the trusted checkout. No test cases were removed.

Test / case Behavior protected Layer Needed? Action / rationale
tests/integration/test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_completes_a_gateway_turn Bearer inference, rendered draft, native submission, completed answer macOS E2E keep Preserves the inference-only journey; actual desktop submission requires native execution.
tests/integration/test_ug_codex_desktop_macos.py::test_ug_codex_desktop_macos_skill_calls_mcp OAuth/profile setup, skill reference use, approval, MCP receipt and answer macOS E2E keep Distinct desktop/process boundary; resolve required-job authentication before gating on it.
tests/integration/test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_completes_a_gateway_turn Bearer inference, model selection, rendered draft and submission Windows E2E keep Retains Windows-specific UI and inference coverage.
tests/integration/test_ug_codex_desktop_windows.py::test_ug_codex_desktop_windows_skill_calls_mcp Skill/MCP journey plus elevated sandbox setup and readiness retry Windows E2E keep Windows setup and UIA behavior cannot be established by the macOS case.
tests/test_codex_desktop_evidence.py::test_cli_login_removes_bypasses_and_cleans_up_after_successful_login — success, login failure, identity failure Override removal, selected profile, failure propagation and logout registration Component keep Distinct orchestration paths; mocked commands do not verify real CLI/keyring behavior.
tests/test_codex_desktop_evidence.py::test_windows_readiness_recheck_is_visible_and_bounded — ready, recovers, fails No unnecessary click, one visible retry, persistent-error rejection Component keep Small state matrix belongs below E2E.
tests/test_codex_desktop_evidence.py::test_windows_setup_requires_the_native_elevated_mode — absent, unelevated, elevated Only the saved elevated setting satisfies setup Component/filesystem keep Checks meaningful native-setting boundaries without adding live setup permutations.
tests/test_codex_desktop_evidence.py::test_gui_approves_only_the_requested_search — both role pairs; valid, server, tool, query, echoed-query and ambiguous cases Approval identity/query checks and ambiguity rejection Component keep Add the transcript-title plus unrelated-dialog regression; current mutations do not cover it.
tests/test_codex_desktop_evidence.py::test_submission_requires_the_native_user_prompt — matching user input, assistant echo, different prompt, wrong event kind Native user submission rather than echoed text Component/parser keep Distinct false-positive guards; these checks do not establish same-turn completion.
tests/test_codex_desktop_evidence.py::test_skill_values_are_only_in_the_reference_file Hidden query/token reside in the reference, not skill instructions Component/filesystem keep Protects the hidden-input invariant underlying the skill evidence.
tests/test_codex_desktop_evidence.py::test_native_mcp_receipt_proves_execution_including_code_mode Successful native receipt acceptance and additional successful calls Component/parser keep Positive control for receipt-based validation; synthetic data does not prove live code-mode execution.
tests/test_codex_desktop_evidence.py::test_wrong_origin_model_turn_or_answer_fails — origin, provider, model, context turn, receipt turn, receipt event, completion event and answer mutations Attribution, completed-event and turn/answer validation Component/parser keep Each mutation protects a distinct evidence dimension; add prompt-to-answer turn correlation here.
tests/test_codex_desktop_evidence.py::test_every_matching_mcp_receipt_must_succeed — type, server, tool and query mutations Rejects unrelated or incorrectly addressed tool evidence Component/parser keep Necessary matching boundaries at the smallest reliable layer.
tests/test_codex_desktop_evidence.py::test_every_matching_mcp_receipt_must_succeed — pending/failed status, error, missing result, error result, empty/blank content Rejects unsuccessful or empty native results Component/parser keep Checks result semantics rather than tool-name telemetry alone.
tests/test_codex_desktop_evidence.py::test_success_cannot_hide_another_failed_matching_call Every matching receipt must succeed Component/parser keep Multi-call failure is not covered by single-receipt mutations.
tests/test_codex_desktop_evidence.py::test_final_answer_alone_cannot_prove_tool_use A completion token without a receipt is insufficient Component/parser keep Explicit negative control for the central MCP evidence claim.
tests/test_codex_desktop_evidence.py::records Shared synthetic native completion and receipt input Fixture keep Reuses a compact baseline with mutations in tests; add submitted-prompt evidence for correlation coverage.
tests/test_integration_contract.py::test_codex_desktop_macos_ci_is_blocking_and_installs_a_pinned_app Required macOS job, app pin/integrity and selected cases Static contract keep Align the new selection assertion with the authentication/provisioning decision.
tests/test_integration_contract.py::test_codex_desktop_windows_ci_is_blocking_and_installs_a_pinned_app Required Windows job, app pin/integrity and selected cases Static contract keep Same prerequisite mismatch, with distinct Windows installation checks.
tests/test_integration_contract.py::test_latest_harness_runs_the_newest_signed_chatgpt_app Latest signed-app installation and desktop journey selection Static contract keep Retain signature coverage; selected OAuth cases also need an authenticated operator or provisioned lane.
tests/integration/utils/codex_desktop.py::login_databricks_cli Public OAuth setup, override removal, profile verification and logout Process support keep Shared platform setup avoids duplication; real OAuth and cleanup remain unverified.
tests/integration/utils/codex_desktop.py::finish_windows_readiness_check, tests/integration/utils/codex_desktop.py::wait_for_windows_sandbox Bounded visible retry and elevated-mode gate UI/filesystem support keep Isolates Windows-specific setup mechanics.
tests/integration/utils/codex_desktop.py::wait_for_skill_answer, tests/integration/utils/codex_desktop.py::wait_for_submitted_prompt Visible answer, guarded approval and native submission UI/evidence support keep Bind approval to its dialog and correlate submission with final completion.
tests/integration/utils/codex_desktop.py::write_search_skill, tests/integration/utils/codex_desktop.py::assert_skill_mcp_turn, tests/integration/utils/codex_desktop.py::assert_desktop_skill_mcp_session Hidden reference creation, single-session attribution and successful same-turn receipts Filesystem/parser support keep Shared across platforms; existing Codex evidence helpers offer a concrete home for receipt parsing.
tests/integration/utils/macos_app.py::MacApp.type_line Optional Return after typing macOS input support keep Enables draft verification while preserving default submission behavior.
tests/integration/utils/windows_app.py::WindowsApp.type_line, tests/integration/utils/windows_app.py::WindowsApp.__exit__ Optional Enter and setup/MCP diagnostics Windows input/process support keep Supports visible submission and preserves relevant failure evidence.
tests/integration/utils/windows_uia.ps1 — UTF-8 output and optional enter handling Correct label decoding and native typing without immediate submission PowerShell/UIA support keep Companion behavior required by the Python input helper.
tests/README.md — coverage inventory and desktop evidence descriptions Discoverability and coverage boundaries Documentation keep Distinguish implemented assertions from verified native passes and required-CI readiness.
tests/integration/README.md — OAuth prerequisites and skill/MCP journey documentation Reproduction requirements and evidence expectations Documentation keep Browser authentication is a material prerequisite; align required-job claims with actual provisioning.

Native receipts would establish local tool-call/result evidence, not independently verify gateway HTTP status/headers or downstream usage attribution. Those remain unverified.

To verify the fixes, extend the existing component scenarios with the two synthetic triggers above. Then run each desktop journey on its target OS with browser OAuth available, checking the actual approval’s server/tool/query, completed-turn evidence, and profile logout/keyring cleanup.

Scope

All changed files support desktop skill/MCP execution or its submission, setup, diagnostics and documentation. No clearly unrelated changes were found. The blocking issue is CI readiness for the intended journey, not an unrelated outcome.


Automated review by Polly · workflow run

@andy-xu-db
andy-xu-db marked this pull request as ready for review November 20, 2026 14:51
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review skipped

adefaaf already has a review, so there is nothing new to look at.

Push a commit before requesting another review.

@github-actions github-actions Bot added the ready-to-approve Ready for maintainer approval label Nov 20, 2026
@andy-xu-db

Copy link
Copy Markdown
Collaborator Author

/ug-review

"--skip-upgrade",
timeout=240,
)
session.run("mcp", "add", "--agents", "codex", "--names", MCP_SERVICE, timeout=240)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintainability · low
can we move the profile/configure/mcp-add setup and the completed-turn polling into codex_desktop.py? the macOS and Windows skill tests duplicate them and could drift apart.

@github-actions github-actions Bot removed the ready-to-approve Ready for maintainer approval label Nov 20, 2026
@andy-xu-db

Copy link
Copy Markdown
Collaborator Author

/ug-review

Comment on lines +116 to +121
session = live_session
profile = configure_databricks_token_profile(session, workspace)
marker = fresh_id()
query = f"Databricks Unity Catalog documentation {fresh_id()}"
write_search_skill(session.home, query, marker)
use_managed_config_fixture(session, "managed_workspace_default")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintainability · low
can we move the profile/skill/configure/mcp add setup, the prompt string, and the 60s completed-turn wait into utils/codex_desktop.py? they're copied verbatim in the macOS test and can drift apart.

@github-actions github-actions Bot added the ready-to-approve Ready for maintainer approval label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-approve Ready for maintainer approval

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants