Skip to content

Navigator passes a sign-in split over three pages; add regression scenario - #294

Merged
DavertMik merged 1 commit into
mainfrom
staged-login-regression
Nov 6, 2026
Merged

DavertMik merged 1 commit into
mainfrom
staged-login-regression

Conversation

@DavertMik

Copy link
Copy Markdown
Contributor

Summary

Adds a regression scenario for a sign-in split over three pages (email, password, one-time code), and fixes the two Navigator gaps it exposed.

Fixture and scenario

  • /signin, /signin/password and /signin/otp each ask for one value. Each page redirects to /signin if the earlier stage wasn't done, and only the code stage opens /account.
  • seeds/knowledge/staged-login.md holds the credentials and no instructions, so the Navigator has to work out the steps from each page.
  • regression:staged-login runs explorbot navigate /account. It passes only on exit 0 with no Captain handover, because a Captain rescue would hide a Navigator failure.
  • It is included in regression:all and the workflow's scenario list, with no-AI checks added to regression:smoke.

Navigator fixes (src/ai/navigator.ts)

  1. A page change drops the rest of the batch. The model proposes up to 5 alternatives per batch. After one moved the page, the others still ran against a page they weren't written for, using up the 10-iteration budget and adding a 5s URL wait each. This also covers a block that moves the page and then fails on a later line.
  2. Every retry includes the current page. Retries had no page content beyond the one-time full HTML send, and no ARIA diff appeared on these retries, so on later pages the model guessed locators such as input[name="otp"] and I.click('Continue'). Retries now include a <page> block with toAiContext().

Neither fix adds instruction text to a prompt. The second does change what the model reads, by re-sending the existing <page> context on each retry, so it deserves a look in review. It is ARIA only, which keeps to the Navigator's HTML access rule.

Test results

Real-AI runs of the Navigator on the staged sign-in, all on openai/gpt-oss-20b:nitro:

Code Reached /account through the Navigator alone
Fix 1, first version, without fix 2 4 of 12
Fixes 1 and 2 as committed 9 of 10

The one failure on the final code was an empty model response on the first call, after which the Navigator stops. That is unrelated to staged pages and is not addressed here.

These are small samples from a nondeterministic model, so read them as a strong shift rather than a precise rate. The scenario uses the default retries like its siblings, which can hide an occasional failure in CI.

  • regression:smoke passes (39 checks).
  • tests/integration/ passes (174 pass, 1 skipped).
  • All Navigator unit tests pass. Two new tests failed before their fixes; the first version of fix 1 also got a test that failed before it.
  • tests/unit/ has 8 failures (Provider, ApibotConfigParser). The same 8 tests fail on a clean checkout without this change.

Not run

No CI regression run was started. This PR touches src/ai/, so a run will wait at the regression environment gate for approval. It would be the first evidence for the Navigator changes in the other scenarios, none of which were run locally.

🤖 Generated with Claude Code

…nario

Add a staged sign-in to the Trackly fixture (email, password and one-time
code on separate pages) and a regression:staged-login scenario that runs
`explorbot navigate /account` and fails if Captain has to take over.

The scenario exposed two Navigator gaps on multi-page flows:

- After a step moved the page, the rest of the batch still ran against a
  page it was not written for, using up the attempt budget. A page change
  now drops the rest of the batch.
- Retries carried no page content beyond a single HTML send, so on later
  pages the model guessed locators. Every retry now includes the current
  page's ARIA context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Explorbot Self-Regression

Commit 9f46043 · run

Scenario Result Attempts Time Previous median Change
basic (native) PASS 1/3 4m 58s 5m 42s -13%
experience: control OK — failed as expected 1/1 54s 27s +100% slower
experience: seeded PASS 1/3 53s 43s +23% slower
slow (native) PASS 1/3 1m 37s 1m 59s -18%
gated (native) PASS 1/3 1m 59s 2m 19s -14%
staged-login (native) PASS 1/3 1m 0s — —

Time is the passing attempt (the last attempt if none passed). Change compares it with the median passing time of the last 4 runs; ±20% or more is flagged.

Attempt details

  • basic (native) attempt 1 — PASS (4m 58s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=6, wellFormed=true, keywords=7/3); scenarios: PASS (tests=8/5, features=4/3); tests passed: PASS (5 passed, 0 failed (reporter: 5 passed, 0 failed))
  • experience: control attempt 1 — PASS (54s): control: OK — failed as expected (0 passed, 0 failed)
  • experience: seeded attempt 1 — PASS (53s): seeded: PASS (1 passed, 0 failed)
  • slow (native) attempt 1 — PASS (1m 37s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=2, wellFormed=true, keywords=7/3); scenarios: PASS (tests=7/5, features=3/3); tests passed: PASS (1 passed, 0 failed (reporter: 1 passed, 0 failed))
  • gated (native) attempt 1 — PASS (1m 59s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=4, wellFormed=true, keywords=7/3); scenarios: PASS (tests=6/5, features=4/3); tests passed: PASS (1 passed, 0 failed (reporter: 1 passed, 0 failed))
  • staged-login (native) attempt 1 — PASS (1m 0s): reached target: PASS; navigator alone: PASS (captain handovers: 0)

Session analysis — basic (native):

Session Analysis

Creating an issue and opening its detail page work end to end. Keyword search shows the empty state for a non-matching term, but matching results and empty-title rejection are not reliably verified.

Coverage

  • Pages: /issues, /issues?new=1, /issues/6
  • Features: issue creation from toolbar, new-issue form title validation, issue detail navigation, keyword search, empty search results

What works

  • Issue creation - ET-1 Create a new issue from the toolbar and verify it appears in the issues list
  • Issue detail navigation - ET-3 Open an issue detail page from the list via its title link and verify the correct issue is shown
  • Empty search results - ET-5 Search issues with a keyword that matches nothing and verify the empty result state

Execution Issues

  • ET-1 Create a new issue from the toolbar and verify it appears in the issues list - Pre-create count never recorded; increase not verified.
  • ET-2 Submit the new issue form without a title and verify the issue is rejected - Passed despite failed steps; submit click absent from log.
  • ET-4 Search issues by a keyword from an existing issue title and verify only matching issues are listed - Passed with no assertion steps in log.

Session analysis — slow (native):

Session Analysis

Issue creation from the Issues toolbar works end to end: a new issue saves with the entered title and appears in the table with status open and priority normal. Only this single creation path was exercised, so editing, validation, and other issue actions are untested.

Coverage

  • Pages: /issues, /issues?new=1
  • Features: Issue creation drawer (toolbar New Issue), issues table listing

What works

  • Issue creation — ET-1 Create a new issue from the toolbar and verify it appears in the issues table

Session analysis — gated (native):

Session Analysis

Issue creation from the workspace toolbar appears to persist the new issue to the issues table. The run did not confirm that the form closes or that the issue shows without a page reload.

Coverage

  • Pages: /workspace, /issues, /issues?new=1
  • Features: Toolbar "New Issue" drawer, issue creation form (title, description, priority), issues table listing

Execution Issues

  • ET-1 Create a new issue from the toolbar and verify it appears in the issues table - passed without checking form close or no-reload visibility

@DavertMik
DavertMik merged commit 5b243f9 into main Nov 6, 2026
3 checks passed
@DavertMik
DavertMik deleted the staged-login-regression branch November 6, 2026 12:23

This branch was successfully deployed

1 active deployment
regression — 02d8bed9 Deployed Nov 5, 2026 by DavertMik via regression #203
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant