Skip to content

test(ci): retain immediate VS Code click failure evidence - #1804

Closed
lilly-luo wants to merge 4 commits into
mainfrom
isaac/lilly.luo/vscode-hover-readiness-20261120
Closed

lilly-luo wants to merge 4 commits into
mainfrom
isaac/lilly.luo/vscode-hover-readiness-20261120

Conversation

@lilly-luo

@lilly-luo lilly-luo commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What did you change, and why?

The VS Code Claude-to-GPT journey failed opening the model picker in CI 44751086897. Playwright found the button visible, enabled and stable, but a Use configured approval settings hover intercepted pointer events. Existing screenshots are taken only after the whole journey fails, when that overlay may already be gone; clicks consumed by the picker's screen-change recovery lose their original error details.

Keep the original two-second click timeout and picker recovery, and capture each failed click immediately: screenshot, action, elapsed time, native error, target bounds, visible hover text/bounds and recognized screen. JSON goes through the existing redacted evidence writer. Read hover data before target lookup so it survives a disappeared target. Failed metadata lookup retains core evidence with an exception class; failed evidence writing records only its error class and preserves the original click exception. Failure screenshot and target lookup each have a 500ms limit; ordinary screenshots retain their default timeout. Document that these post-timeout observations are not atomic.

How do you know it works?

/usr/bin/python3.12 -m unittest tests.test_vscode_harness -v: four real-helper component checks passed, covering successful short clicks without failure evidence, retained geometry/error/screenshot data, a disappeared target keeping core/hover evidence, and evidence-write failure preserving the original click exception. Tests verify the capture limits and the original single click. Browser/file-writing boundaries are substituted; these checks do not prove native tooltip geometry or live artifact delivery.

Python compilation, git diff --check and commit/push secret scans passed. Ruff/full pytest are unavailable locally; PR CI covers those and native VS Code. tests/README.md is unchanged. This adds diagnostics, not a claim to fix the underlying tooltip flake. The original short deadline is required so the picker can resample a changed screen within its existing step budget.

Checklist before requesting review

  • Comment /ug-review and address the comments. Supported review labels trigger the workflow because the connector appends a footer to comments. The final diagnostic-scope UG review recommended APPROVE; both final-scope threads are addressed and resolved. Secondary Polly review remains pending.
  • Ordinary PR CI runs integration; /integration-test will not be commented, as instructed.

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

/ug-review


This comment was generated with GitHub MCP.

@lilly-luo lilly-luo added the ug-review Run the automated UG review label Nov 20, 2026
Comment thread tests/integration/utils/vscode.py Resolved
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

UG review recommendation

RECOMMENDATION: APPROVE

Reasons:

  • Finding 1 (medium, performance) is about the capture timeout in test utility code. Up to about 4s spent per failed click eats into the retry budget, but this is a tuning issue in integration test helpers, not a correctness or data-loss problem.
  • Finding 2 (low, bug) notes that hover evidence is lost when bounding_box raises on a missing target. This makes diagnostics less complete in an edge case but does not break existing workflows, so it can be a follow-up.
  • Neither finding involves security, data loss, configuration ownership, or a reproducible production bug. Both are confined to test tooling and can be addressed as optional improvements.

Findings considered: 2 (severities: low: 1, medium: 1); see the inline review comments.

Advisory only: this recommendation never approves or merges the pull request. The ready-to-approve label tracks it.

@github-actions github-actions Bot added the ready-to-approve Ready for maintainer approval label Nov 20, 2026
@lilly-luo lilly-luo changed the title test(ci): wait for VS Code hover tooltips before clicking test(ci): use the step budget for VS Code click readiness Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

None found.

Security vulnerabilities

None found.

Non-blocking notes

  • tests/test_vscode_harness.py:27: can we assert pointer movement precedes polling? The count stub dismisses itself, so waiting before moving would still pass.
  • tests/integration/utils/vscode.py:250: can we bound hover polling by the remaining caller budget? Each click now gets another 60s, extending the existing soft picker cap.

Approach

The shared helper is the right location, and reusing _wait preserves polling and process-exit handling. A remaining-deadline bound would limit added latency without forcing clicks or hiding persistent overlays.

Summary

Diff size (computed): 3 files; +57 / -0 lines (57 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 3 57 0 100.0%
Documentation 0 0 0 0.0%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 0 0 0 0.0%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

The change adds hover-dismissal readiness before shared VS Code clicks, focused component coverage, and documentation of its limits. The implementation is proportional to the reported interception failure. No concrete duplication warrants removal.

Tests

Test-by-test assessment
Test / case Behavior protected Layer Needed? Action / rationale
tests/test_vscode_harness.py::TestHoverReadiness.test_click_waits_for_hover_to_close Polls until hover absence, then clicks once with the original timeout Mocked component keep Distinct regression case. Strengthen movement-before-polling assertions as noted above; mocked counts do not prove browser dismissal.
tests/test_vscode_harness.py::TestHoverReadiness.test_click_without_hover Click proceeds immediately when no hover exists Mocked component keep Distinct normal path; complements delayed dismissal and timeout coverage.
tests/test_vscode_harness.py::TestHoverReadiness.test_persistent_hover_fails_without_clicking Persistent hover raises and prevents the click Mocked component keep Covers the material failure path without needing VS Code.
tests/test_vscode_harness.py:12 — shared setup Exercises the real helper without launching the application Component support keep Small setup substitutes external boundaries; no additional fixture framework is needed.
tests/integration/utils/vscode.py:248 — _click Applies readiness consistently before shared clicks Integration support keep Reuses existing polling rather than adding retries or bypassing pointer interception.
tests/integration/README.md:823 — readiness documentation Describes the wait and distinguishes component checks from native evidence Documentation keep Relevant harness documentation; accurately limits what offline checks establish.

All three added cases passed in offline execution. Removing the helper change caused the delayed-dismissal and persistent-hover cases to fail.

Existing VS Code journeys already cover model selection, completed tasks, and correlated inference requests/responses; another live journey is unnecessary. Actual tooltip-selector matching, browser visibility semantics, and dismissal timing remain unverified. Live integration and lint were not run.

To verify the native behavior, use the pinned VS Code build and existing Claude-to-GPT journey: reproduce the obstructing tooltip, confirm it disappears before the click, and confirm model switching and the subsequent task complete.

Scope

All changes support the hover-interception fix or its verification and documentation. No unrelated changes were found.


Automated review by Polly · workflow run

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Polly's notes concern the superseded hover wrapper. 20ccca2 removes that polling loop and its mock tests; pointer movement remains immediately before the original single Playwright click. Native actionability uses the existing step budget, with no second wait layered on top. The README records the remaining limit: an unavailable click can take up to the step budget before the existing picker screen-change recovery runs. This keeps the fix to one native timeout argument rather than adding caller-deadline plumbing or replaying an action. Native CI remains pending.


This comment was generated with GitHub MCP.

@lilly-luo lilly-luo removed automerge ready-to-approve Ready for maintainer approval labels Nov 20, 2026
@lilly-luo lilly-luo changed the title test(ci): use the step budget for VS Code click readiness test(ci): retain immediate VS Code click failure evidence Nov 20, 2026
@lilly-luo lilly-luo added the code-review Run the automated UG review label Nov 20, 2026
Comment thread tests/integration/utils/vscode.py Resolved
Comment thread tests/integration/utils/vscode.py Resolved
@github-actions github-actions Bot added the ready-to-approve Ready for maintainer approval label Nov 20, 2026
@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Native evidence is now captured in artifact14642086286: the model-button center (1266.56,813) is inside the neighboring hover bounds (1207..1437.61,811..838), with the original2s error and immediate screenshot retained. Three of four Linux VS Code cases passed; the fourth reproduced this exact pre-existing interception.

Pinned upstream source also confirms the primary model-name button is focusable and handles Enter/Space. The smaller native-keyboard fix is being validated in #1805. Holding this diagnostic PR without automerge and canceling its redundant CI to reduce provider pressure; if the small fix passes native validation, this PR can be closed as superseded while retaining its evidence and branch for audit.


This comment was generated with GitHub MCP.

@lilly-luo

Copy link
Copy Markdown
Collaborator Author

Closing as superseded by the smaller real fix #1805. Its exact final head d6468c4 passed all4 native VS Code cases on both Linux and Windows, including readable Claude/GPT model selection and completed file-task answers. The generic capture code is therefore not needed to solve this flake.

Diagnostic evidence remains preserved in artifact14642086286, the discussion and this pushed branch. It established that the adjacent approval tooltip covers the model button's center; the native keyboard fix avoids that pointer overlap while retaining the original2s recovery. This PR will remain unmerged.


This comment was generated with GitHub MCP.

@lilly-luo lilly-luo closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted ai-factory code-review Run the automated UG review omni-automation ready-to-approve Ready for maintainer approval ug-review Run the automated UG review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants