Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1273,6 +1273,7 @@ integration utilities; only CUJ-specific evidence correlation stays in a test fi
| `test_ug_configure_codex_custom_azure_openai_mps` | Select custom Azure OpenAI MPS through the configure picker, then launch Codex without a provider override | Saved provider; completed TUI file task on the selected GPT deployment; normal exit; no routing |
| `test_ug_configure_claude_bedrock_mps` | Select the restricted Bedrock MPS in the real configure picker; launch Claude without an explicit model | Saved provider in status; Sonnet 4.6 is selected from the Sonnet + Haiku allow-list and completes a TUI file task; normal exit |
| `test_ug_configure_claude_bedrock_mantle_mps` | Select the Mantle MPS in the real configure picker; launch Claude on Opus 5.5 without a provider override | Saved provider and Claude provider header; completed TUI file task on the selected model; normal exit; no routing |
| `test_ug_codex_desktop_macos_skill_calls_mcp`, `test_ug_codex_desktop_windows_skill_calls_mcp` | Reuse the existing CI credential in an isolated CLI token profile, configure Codex with --use-pat, add web-search MCP, and invoke a local skill | Verified CLI identity before configure/add; hidden reference token on screen and in a completed managed-model desktop turn; successful native MCP receipts with the hidden query in that same turn. Uses the saved-token path with inherited bearer overrides removed; browser OAuth and token refresh are not covered |
| `test_ug_configure_codex_databricks` | Configure Databricks Hosted; execute the generated auth helper; open Codex TUI and read a file | Generated helper invokes `ug` with clean token stdout; completed assistant answer contains the file value; normal exit and reopen |
| `test_ug_configure_codex_openai_mps` | Select OpenAI MPS in the real configure picker; launch Codex | Saved provider in status; completed TUI file task; normal exit |
| `test_ug_configure_codex_bedrock_runtime_mps` | Select the Bedrock Runtime MPS in the real configure picker; launch Codex on its GPT-5.6 inference profile without a provider override | Saved provider in status; completed TUI file task on the selected model; normal exit; no routing |
Expand Down Expand Up @@ -1452,6 +1453,27 @@ and managed-settings coverage remain deferred; a green installation check alone
does not establish a successful live task.
Required macOS and Windows jobs open the pinned ChatGPT desktop app after managed `ug configure`, send a
prompt through the GUI, and assert the exact answer, model, and completed turn from the app's rollout.
Both also invoke a user-authored `$CODEX_HOME/skills` skill whose reference file supplies a hidden
query and completion token, after registering `system.ai.web_search` with `ug mcp add`.
The token must render in the GUI and complete on the managed model; native successful MCP receipts
must carry the query in that same Codex Desktop/Databricks turn. The pinned app versions are
macOS `26.1103.31037` and Windows `26.1103.7773.0`; harness-latest also selects the macOS journey.
`test_codex_desktop_evidence.py` rejects false positives from failed/missing calls, unrelated
turns/queries/servers, wrong origins/models and echoed or incomplete answers. Skill/MCP evidence
selects the single root rollout and retains any child rollouts in the artifacts. Missing or
ambiguous roots fail, and successful child evidence cannot satisfy a failed or incomplete root.
These are offline parser checks; live desktop coverage requires the native CI jobs.
Both inference and skill journeys wait for the typed draft, click Send, and require the submitted user prompt
in the rollout before waiting for an answer. Windows first completes its visible sandbox setup.
The app's saved elevated sandbox mode proves completion even after its short-lived banner disappears;
the tests only read that native setting. Missing or partial TOML is polled until the setup
deadline; component checks cover recovery and deadline failure. Windows artifacts retain early MCP/setup diagnostics.
If the app's setup restart leaves a sandbox-readiness error, the journey uses the visible
Try again control once before sending the prompt; a persistent error fails. No task is resubmitted.
Offline submission checks reject assistant echoes, unrelated prompts and non-message events.
The native MCP approval is answered with Allow once only when the dialog names web_search on
the expected server with the exact hidden query. Offline checks reject other tools, servers,
queries and ambiguous approval controls.
The `All integration tests` check requires every selected integration job to pass, including
both managed-config lanes for full/live runs; full coverage does not depend on a label or
a manual request.
Expand Down Expand Up @@ -1599,7 +1621,7 @@ verify that neither a sudo capability probe nor a prompting worker is started.
| Follow-up turns and conversation resume | Not covered; reopen proves startup, not conversation resume |
| Claude/Codex interactive smart routing | First-prompt routing through `first_prompt_and_subagent_no_orch_v0` is covered by the `managed_fixture` banner journeys and CUJ4's preset journeys. CUJ4's managed journeys verify that `smart_routing.enabled` alone leaves Claude's first prompt unrouted and turns on Smart Router Orchestrator for Claude, and that Codex's managed default stays unrouted and unorchestrated; the Claude managed-default skill-toggle cases and a Codex launch-only opt-out case cover those paths; `--enable-smart-routing` has component coverage only. The live subagent-only journeys spawn native children and invoke the installed Smart Router skill to verify routing at launch, off, and off -> on in separate Claude/Codex sessions. The scheduled harness-latest workflow repeats CUJ4, routing hooks, skill toggles, and banners with both latest agents. Plugin-refresh survival and native daemon/background dispatch remain deferred. |
| Full allow/deny tool-permission matrix | Not covered; onboarding/trust uses actual TUI choices |
| Desktop Codex app, Isaac itself, auto-upgrades | Not covered by command forwarding or pinned-version tests. Codex desktop's shared-config catalog pointer (repeat configure and a user-owned pointer) is covered by `integration/test_ug_configure_codex_desktop.py`; GUI inference on macOS and Windows by `integration/test_ug_codex_desktop_macos.py` and `integration/test_ug_codex_desktop_windows.py`; desktop tracing and Linux GUI are not |
| Desktop Codex app, Isaac itself, auto-upgrades | Not covered by command forwarding or pinned-version tests. Codex desktop's shared-config catalog pointer (repeat configure and a user-owned pointer) is covered by `integration/test_ug_configure_codex_desktop.py`; GUI inference, local skill execution and MCP web-search invocation on macOS and Windows by `integration/test_ug_codex_desktop_macos.py` and `integration/test_ug_codex_desktop_windows.py`; UC skill downloads through the desktop, connector browser sign-in, desktop tracing and Linux GUI are not |
| Native macOS/Windows live TUI, managed settings, resize/signals | Windows fresh-install, CLI, local helpers, and the Claude headless gateway journey are advisory; Codex Windows CI is blocked on npm proxy access. Live PTY/TUI, managed settings, and signal behavior still need separate platform implementation and coverage |
| Other agents | Live scope is Claude Code and Codex, plus a non-blocking OpenCode headless case |
| OpenCode `--model` / `-m` | Unit/component tests cover raw CLI forwarding, configured model selection, launch arguments, rejection of unknown models before starting the native process, and the existing failure when no default model is available; no dedicated live integration journey |
Expand Down
46 changes: 43 additions & 3 deletions tests/integration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,46 @@ accessibility tree captured before the fatal error in command evidence.
Automation: each screen gets 60 seconds and the answer four minutes, with no retries, and it saves
a screenshot of the app window as evidence. It also requires the app's model picker to list only
the managed model, under its readable name (`5.6 Sol`) rather than the raw id.
Both inference cases wait for the rendered draft, click Send, and require the user prompt in
the native rollout before the answer wait. Failed answer waits also retain rollout evidence.
Each module also runs `test_ug_codex_desktop_*_skill_calls_mcp`. It reuses the same
CI OAuth bearer as the inference cases, saving it in an isolated CLI token profile through
`databricks configure --host <workspace> --profile codex-desktop-ci` with redacted stdin.
`databricks current-user me` verifies that profile before configure/add. The test removes
bearer and bearer-command overrides and uses the default CLI config location. UG selects
`--profile codex-desktop-ci --use-pat`, its existing saved-token mode also used by the Cowork
connector tests, so the MCP proxy reads the credential from disk without forwarded auth
environment variables. No new PAT is minted; the credential remains the runner's short-lived
OAuth token. The profile is removed with the disposable home and is never uploaded.
Browser sign-in, OAuth-cache lookup and token refresh are not covered. The token must remain
valid for the journey; expiry fails the test. The inference-only cases keep their existing
bearer setup.
After managed configure,
`ug mcp add --agents codex --names system.ai.web_search` registers the real search service.
A user-authored skill in the session's `$CODEX_HOME/skills` points to a reference file containing a fresh query
and completion token. Windows completes sandbox setup through the visible Continue button.
It waits through a missing or partially written config within the same setup deadline, then
reads the elevated mode saved by the app after success, without writing native settings or
depending on the transient success banner, and retains early MCP/setup diagnostics in evidence.
After setup's app-server restart, a visible readiness error gets one Try again recheck before
Send. A persistent error fails; this does not resubmit a task or change sandbox permissions.
Both platforms wait for the typed draft, click Send, and require the user prompt in the native
rollout before the answer wait. The GUI prompt names only the skill. Success requires the token on screen,
a completed Codex Desktop/Databricks turn on the managed model, and native `McpToolCall`
receipts in that same turn with the reference query, successful status and nonempty MCP content.
The skill/MCP check selects the single root rollout while retaining child rollouts as evidence.
Missing or ambiguous roots fail; a child's successful call or answer cannot satisfy the root's checks.
Every matching call is checked; additional successful calls are allowed, including calls made
through code mode. Native approval dialogs must name web_search on the expected server and show
the hidden query before the test clicks Allow once; unexpected or ambiguous approvals fail.
These journeys use the same pinned apps and managed workspace as inference;
the workspace identity must have access to `system.ai.web_search`. They cover local skill
execution and MCP invocation, not UC skill downloads, connector browser sign-in or search ranking.
`../test_codex_desktop_evidence.py` checks hidden reference inputs and rejects missing/failed
receipts, unrelated servers/queries/turns, wrong origins/models and incomplete answers offline.
It also rejects false submission evidence and checks profile setup failure propagation,
removal of inherited credentials, and redaction of token stdin/stdout/stderr from evidence.
These component checks do not establish live authentication or MCP execution.
`scripts/run_integration.py` ignores both modules unless `--desktop-app` is passed, which only the
desktop CI jobs do, so a local run never drives a developer's installed app.

Expand Down Expand Up @@ -1147,8 +1187,8 @@ test_ug_codex_v2_routing.py # exact native v2 assignments and route
test_ug_configure_claude_lifecycle.py # repeat setup, revert, rejected credentials
test_ug_configure_claude_desktop.py # Desktop lifecycle, system.ai/relay inference, installed Cowork MCP; no GUI
test_ug_configure_codex_desktop.py # stubbed managed Codex desktop catalog pointer, no GUI
test_ug_codex_desktop_macos.py # ChatGPT desktop app on macOS: GUI prompt, exact answer, model, completed turn
test_ug_codex_desktop_windows.py # the same journey on Windows through UI Automation
test_ug_codex_desktop_macos.py # macOS GUI: inference plus a local skill invoking real MCP web_search
test_ug_codex_desktop_windows.py # the same journeys on Windows through UI Automation
test_ug_configure_claude_workspace_switch.py # real skills MCP cleanup across two workspaces
test_ug_configure_codex_lifecycle.py # repeat setup, revert, rejected credentials
test_ug_configure_scope.py # files written by default, --user, UG_SKIP_OS_SETTINGS, --global; launch after each
Expand Down Expand Up @@ -1646,7 +1686,7 @@ OpenAI's versioned appcast zip, pinned by version and SHA256 in `integration.yml
Accessibility to the runner's shell, so no permission prompt is needed. It uploads
`integration-codex-desktop-macos` evidence. To update the app, take a newer `<enclosure>` from
`https://persistent.oaistatic.com/codex-app-prod/appcast.xml` and its SHA256. The
harness-latest workflow runs the same journey against the newest appcast build, so it shows
harness-latest workflow runs the same journeys against the newest appcast build, so it shows
when a bump is due.
The required **Codex desktop inference · Windows** job installs the ChatGPT MSIX that OpenAI keeps
at `codex-app-prod/releases/<version>/ChatGPT-x64.msix`, pinned by version and SHA256 in
Expand Down
75 changes: 72 additions & 3 deletions tests/integration/test_ug_codex_desktop_macos.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,15 @@
from pathlib import Path

import pytest
from utils.codex_desktop import (
MCP_SERVICE,
SKILL_NAME,
assert_desktop_skill_mcp_session,
configure_databricks_token_profile,
wait_for_skill_answer,
wait_for_submitted_prompt,
write_search_skill,
)
from utils.evidence import (
agent_sessions,
assert_completed_task_model,
Expand All @@ -28,7 +37,7 @@ def test_ug_codex_desktop_macos_completes_a_gateway_turn(live_session, workspace
Expected: the app shows the answer, and its own Codex Desktop rollout, using ug's Databricks
provider, records exactly that answer in a completed turn whose model is the managed system.ai
GPT default. Onboarding is skipped through its visible buttons with mouse clicks and the prompt
is typed with real keystrokes; nothing is seeded. GUI rendering of other screens, MCP, tracing
is typed with real keystrokes, then submitted with Send; nothing is seeded. GUI rendering of other screens, MCP, tracing
and revert while the app runs are not covered. A subsequent click on an absent label fails
with the native accessibility tree before the fatal error, captured in command evidence.
"""
Expand All @@ -46,8 +55,15 @@ def test_ug_codex_desktop_macos_completes_a_gateway_turn(live_session, workspace
app.click("AXButton", "Go to ChatGPT")
# Click the composer so the keystrokes cannot land in another control.
app.click("AXTextArea", None, timeout=120)
app.type_line(f"Reply with exactly {marker} and nothing else.")
app.wait_for("AXStaticText", marker, 300)
prompt = f"Reply with exactly {marker} and nothing else."
app.type_line(prompt, submit=False)
app.wait_for("AXTextArea", prompt, 60)
app.click("AXButton", "Send")
try:
wait_for_submitted_prompt(session, prompt)
app.wait_for("AXStaticText", marker, 300)
finally:
session.record("agent-sessions.json", agent_sessions(session, "codex"))
# The answer can render before the rollout records the completed turn; keep the app open.
deadline = time.monotonic() + 60
while not completed_task_models(session, "codex", marker):
Expand Down Expand Up @@ -91,3 +107,56 @@ def test_ug_codex_desktop_macos_completes_a_gateway_turn(live_session, workspace
assert tree_rows, lines
assert all(len(row.split("\t")) == 4 for row in tree_rows), tree_rows
assert any(row.split("\t")[0].startswith("AX") for row in tree_rows), tree_rows


def test_ug_codex_desktop_macos_skill_calls_mcp(live_session, workspace):
"""Scenario: use an authenticated profile, add web_search, and invoke a skill in the desktop GUI.

Expected: the existing CI credential is saved through the CLI and its identity verified before
configure/add. The app and MCP use that token profile with no inherited bearer override.
Browser OAuth and refresh are not covered. The skill reference supplies the hidden query and token.
The app displays that token and its Databricks-backed completed turn records successful
native MCP web_search receipts with the exact query on the managed model. User onboarding
and Send use visible controls; the rollout must record the submitted prompt before the
answer wait. The expected search is approved with Allow once; no approval/config state is
seeded. UC skill downloads, connector
browser sign-in and built-in web search are outside this journey.
"""
session = live_session
profile = configure_databricks_token_profile(session, workspace)
marker = fresh_id()
query = f"Databricks Unity Catalog documentation {fresh_id()}"
write_search_skill(session.home, query, marker)
use_managed_config_fixture(session, "managed_workspace_default")
session.run(
"configure",
"--agents",
"codex",
"--profile",
profile,
"--use-pat",
"--skip-upgrade",
timeout=240,
)
session.run("mcp", "add", "--agents", "codex", "--names", MCP_SERVICE, timeout=240)

with MacApp(session, CHATGPT_APP, "chatgpt") as app:
app.click("AXRadioButton", "Engineering", timeout=180)
app.click("AXButton", "Continue")
app.click("AXButton", "Skip")
app.click("AXButton", "Go to ChatGPT")
try:
app.click("AXTextArea", None, timeout=120)
prompt = f"Use ${SKILL_NAME}. Follow its instructions and reply with only its completion token."
app.type_line(prompt, submit=False)
app.wait_for("AXTextArea", prompt, 60)
app.click("AXButton", "Send")
wait_for_submitted_prompt(session, prompt)
wait_for_skill_answer(app, query, marker, "AXStaticText", "AXButton", 300)
finally:
session.record("agent-sessions.json", agent_sessions(session, "codex"))
deadline = time.monotonic() + 60
while not completed_task_models(session, "codex", marker):
assert time.monotonic() < deadline, "The app never recorded the completed skill turn"
time.sleep(1)
assert_desktop_skill_mcp_session(session, query, marker, MANAGED_CODEX_MODEL)
Loading
Loading