Skip to content

test(ci): record native Cowork control wait timing - #1802

Open
lilly-luo wants to merge 3 commits into
mainfrom
isaac/lilly.luo/cowork-wait-evidence-20261120
Open

lilly-luo wants to merge 3 commits into
mainfrom
isaac/lilly.luo/cowork-wait-evidence-20261120

Conversation

@lilly-luo

@lilly-luo lilly-luo commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What did you change, and why?

The Windows Hosted Cowork file task failed waiting for its folder-permission dialog in CI 44751086897. Both failure snapshots already contain the exact expected label, enabled and onscreen, so they cannot distinguish late dialog arrival from a slow or inconsistent UI Automation lookup.

Record requested labels, scan count, configured wait budget, elapsed time at timeout and snapshot completion time in the existing result.json. Capture wait-failed.json from the actual subtree searched, including native folder dialogs. If capture fails, record its exception class and retain the original control timeout. Document interpretation and add a Windows script-validation self-check. Wait budgets, selectors, consent clicks and completed-task assertions remain unchanged.

How do you know it works?

Confirmed the downloaded failure artifact contains the exact enabled, onscreen permission label. git diff --check, YAML parsing of the Windows validation step and commit/push secret scans passed.

The final-head PowerShell self-check passed in the Windows Hosted and Anthropic Relay validation steps (Hosted job, Relay job). It executes the real wait function with external UI Automation calls substituted, checking a ready control, JSON-serialized timing fields, searched-root forwarding and snapshot failure preserving the original timeout.

PowerShell/UI Automation is unavailable on this Linux Arclet. Actual native control-timeout artifact delivery remains unverified; passing component checks do not establish the underlying UI flake is fixed. tests/README.md is unchanged.

Checklist before requesting review

  • Comment /ug-review and address the comments. The connector footer prevented exact command matching, so the supported ug-review label triggered the completed review. Recommendation: APPROVE. The snapshot-error thread is fixed, replied to and resolved; Polly's failure-capture and serialization suggestions are addressed.
  • Ordinary PR CI runs native integration; /integration-test will not be commented, as instructed.

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

/ug-review


This comment was generated with GitHub MCP.

@lilly-luo lilly-luo added the ug-review Run the automated UG review label Nov 20, 2026
Comment thread scripts/cowork-inference/test-task.ps1 Resolved
@github-actions

Copy link
Copy Markdown
Contributor

UG review recommendation

RECOMMENDATION: APPROVE

Reasons:

  • The only finding is medium severity and affects error reporting in a test/inference script (scripts/cowork-inference/test-task.ps1), not production behavior.
  • If Save-UI throws on a closed dialog or permission root, or on the full desktop tree, report.error gets a less specific message. That is a diagnostics-quality problem, not data loss, leakage, or a broken workflow.
  • The suggested try/catch around Save-UI with a separate snapshotError field is a reasonable, low-risk follow-up but does not need to block the merge.

Findings considered: 1 (severities: medium: 1); see the inline review comments.

Advisory only: this recommendation never approves or merges the pull request. The ready-to-approve label tracks it.

@github-actions github-actions Bot added the ready-to-approve Ready for maintainer approval label Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

None identified.

Security vulnerabilities

None identified. Diagnostic output retains existing redaction.

Non-blocking notes

  • scripts/cowork-inference/test-task.ps1:58: can we record snapshot errors separately so a dialog closing during the final sleep cannot replace the control timeout?
  • scripts/cowork-inference/test-task.ps1:52: can we add focused PowerShell cases for timeout-field serialization, searched-root forwarding, and snapshot failure?

Approach

The approach is sound: reuse Save-UI and the existing result report without changing polling behavior. Guarding snapshot capture would preserve the timeout diagnosis when a short-lived dialog disappears.

Summary

Diff size (computed): 2 files; +17 / -1 lines (18 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 0 0 0 0.0%
Documentation 1 6 0 33.3%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 1 11 1 66.7%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

Adds control-wait timing and scan evidence, captures the searched subtree, and documents how to interpret a later snapshot. The implementation and documentation are proportional to that outcome; no concrete size reduction is warranted.

Tests

Test-by-test assessment

No test cases were added, modified, or removed. Existing coverage does not establish delivery of the new timeout diagnostics.

Test / case Behavior protected Layer Needed? Action / rationale
scripts/cowork-inference/test-task.ps1 — existing Windows task scenarios Native consent, control interaction, and completed-task outcomes Native E2E keep Protects real UI boundaries; successful runs do not exercise timeout diagnostics.
scripts/cowork-inference/test-diagnostics.ps1 Diagnostic text redaction PowerShell component keep Failure snapshots continue using the existing redaction helper.
scripts/cowork-inference/test-task.ps1::Wait-Control — missing focused cases Timeout JSON fields, searched-root forwarding, and snapshot-error handling PowerShell component uncertain No targeted assertions found; isolated failure-path cases are smaller than another live E2E.
scripts/cowork-inference/README.md:22 — diagnostic guidance Correct interpretation of post-timeout snapshot evidence Documentation keep Correctly distinguishes snapshot visibility from availability during polling.

PowerShell syntax parsing and a synthetic ordered-dictionary JSON round-trip passed. Existing test suites and Windows UI Automation were not run; actual failure-artifact delivery remains unverified. No API or usage-attribution behavior changes.

To verify on Windows, run the existing Cowork job. On a control timeout, inspect result.json and wait-failed.json for the requested labels, budget, scan count, timing order, and searched dialog subtree.

Scope

Both changed files support the stated diagnostic outcome. No unrelated changes were found. Wait budgets, selectors, consent clicks, and completed-task assertions remain unchanged.


Automated review by Polly · workflow run

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Both Polly follow-ups are addressed: snapshot errors are recorded separately while retaining the original timeout (55a1319), and the focused PowerShell self-check validates searched-root forwarding, a ready control, missing control evidence, and failed capture. d42946f additionally checks the actual JSON round-trip so field serialization is covered. The self-check runs in the native Windows validation step. PowerShell is unavailable locally; CI remains the execution evidence.


This comment was generated with GitHub MCP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants