Repository navigation
Navigator passes a sign-in split over three pages; add regression scenario - #294
Conversation
…nario Add a staged sign-in to the Trackly fixture (email, password and one-time code on separate pages) and a regression:staged-login scenario that runs `explorbot navigate /account` and fails if Captain has to take over. The scenario exposed two Navigator gaps on multi-page flows: - After a step moved the page, the rest of the batch still ran against a page it was not written for, using up the attempt budget. A page change now drops the rest of the batch. - Retries carried no page content beyond a single HTML send, so on later pages the model guessed locators. Every retry now includes the current page's ARIA context. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explorbot Self-RegressionCommit
Time is the passing attempt (the last attempt if none passed). Change compares it with the median passing time of the last 4 runs; ±20% or more is flagged. Attempt details
Session analysis — basic (native): Session AnalysisCreating an issue and opening its detail page work end to end. Keyword search shows the empty state for a non-matching term, but matching results and empty-title rejection are not reliably verified. Coverage
What works
Execution Issues
Session analysis — slow (native): Session AnalysisIssue creation from the Issues toolbar works end to end: a new issue saves with the entered title and appears in the table with status open and priority normal. Only this single creation path was exercised, so editing, validation, and other issue actions are untested. Coverage
What works
Session analysis — gated (native): Session AnalysisIssue creation from the workspace toolbar appears to persist the new issue to the issues table. The run did not confirm that the form closes or that the issue shows without a page reload. Coverage
Execution Issues
|
Summary
Adds a regression scenario for a sign-in split over three pages (email, password, one-time code), and fixes the two Navigator gaps it exposed.
Fixture and scenario
/signin,/signin/passwordand/signin/otpeach ask for one value. Each page redirects to/signinif the earlier stage wasn't done, and only the code stage opens/account.seeds/knowledge/staged-login.mdholds the credentials and no instructions, so the Navigator has to work out the steps from each page.regression:staged-loginrunsexplorbot navigate /account. It passes only on exit 0 with no Captain handover, because a Captain rescue would hide a Navigator failure.regression:alland the workflow's scenario list, with no-AI checks added toregression:smoke.Navigator fixes (
src/ai/navigator.ts)input[name="otp"]andI.click('Continue'). Retries now include a<page>block withtoAiContext().Neither fix adds instruction text to a prompt. The second does change what the model reads, by re-sending the existing
<page>context on each retry, so it deserves a look in review. It is ARIA only, which keeps to the Navigator's HTML access rule.Test results
Real-AI runs of the Navigator on the staged sign-in, all on
openai/gpt-oss-20b:nitro:/accountthrough the Navigator aloneThe one failure on the final code was an empty model response on the first call, after which the Navigator stops. That is unrelated to staged pages and is not addressed here.
These are small samples from a nondeterministic model, so read them as a strong shift rather than a precise rate. The scenario uses the default retries like its siblings, which can hide an occasional failure in CI.
regression:smokepasses (39 checks).tests/integration/passes (174 pass, 1 skipped).tests/unit/has 8 failures (Provider, ApibotConfigParser). The same 8 tests fail on a clean checkout without this change.Not run
No CI regression run was started. This PR touches
src/ai/, so a run will wait at theregressionenvironment gate for approval. It would be the first evidence for the Navigator changes in the other scenarios, none of which were run locally.🤖 Generated with Claude Code