[DavertMik]

Problem

A plan for a pet store had 25 tests: 9 for sign-in and registration, none for the cart or checkout. Planner guidance centres on create, update and delete, and its grounding rules stopped it planning anything research had not seen. "Add to cart" only appears after clicking a product, so the store's main flow was never planned.

Change

Prompt only, in src/ai/planner.ts:

• New <main_purpose> block: decide what the website is for and plan that outcome end to end first, unless a focus was requested.
• Hints for five common flows: buying or reserving, managing records, communicating, multi-step applications, reporting.
• Sign-in, FAQ and support pages come after the main flow, unless they are the main flow.
• Unseen steps of the main flow are described by goal. For those steps this overrides <capability_grounding> and the rules requiring evident prerequisites or observed details.
• Removed "100% sure" from the task list.

Testing

Real plan run against the store on merged main, with saved research hidden. Research had no mention of Add to cart.

Plan	Tests	Cart	Reaches checkout	Sign-in/register
Before	25	0	0	9
After	7	4	1	1

This compares one planning round with a plan built over several rounds, and the new tests were generated, not executed.

bun test tests/unit tests/integration: 1923 pass, 8 fail. The same 8 failed on a clean main locally before the latest merge (7 Provider, 1 ApibotConfigParser).

Known limits

• The checkout test only verifies the checkout step opens. It does not complete the order, although the hint asks for the confirmation.
• Only the buying flow was checked on a real site. The other hints are untested.
• Plans can now include completing a purchase or booking, so runs against production need test data.

The regression workflow needs a human approval because this touches src/ai/.

🤖 Generated with Claude Code

[github-actions]

Explorbot Self-Regression

Commit 20d22c7 · run

Scenario	Result	Time	Previous median	Change
basic (native)	PASS	5m 11s	5m 3s	+3%
experience: control	OK — failed as expected	24s	28s	-14%
experience: seeded	PASS	44s	44s	0%
slow (native)	PASS	2m 46s	1m 49s	+52% slower
gated (native)	PASS	2m 26s	2m 5s	+17%
staged-login (native)	PASS	12s	1m 0s	-80% faster
gated-no-auth (native)	PASS	21s	16s	+31% slower

Change compares it with the median passing time of the last 5 runs; ±20% or more is flagged.

Run details

• basic (native) — PASS (5m 11s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=7, wellFormed=true, keywords=7/3); scenarios: PASS (tests=8/5, features=4/3); tests passed: PASS (5 passed, 0 failed (reporter: 5 passed, 0 failed))
• experience: control — PASS (24s): control: OK — failed as expected (0 passed, 0 failed)
• experience: seeded — PASS (44s): seeded: PASS (1 passed, 0 failed)
• slow (native) — PASS (2m 46s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=2, wellFormed=true, keywords=7/3); scenarios: PASS (tests=7/5, features=3/3); tests passed: PASS (1 passed, 0 failed (reporter: 1 passed, 0 failed))
• gated (native) — PASS (2m 26s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=4, wellFormed=true, keywords=7/3); scenarios: PASS (tests=8/5, features=5/3); tests passed: PASS (1 passed, 0 failed (reporter: 1 passed, 0 failed))
• staged-login (native) — PASS (12s): reached target: PASS; navigator alone: PASS (captain handovers: 0)
• gated-no-auth (native) — PASS (21s): stopped: PASS (exited on its own before the 180s timeout); captain handovers: PASS (1/1)

---

Session analysis — basic (native):

Session Analysis

The keyword search on the Issues page filters the table correctly. Issue creation, status, label, and combined filters were exercised, but their passing results rest on weak or contradictory evidence, so their behavior is unconfirmed.

Coverage

• Pages: /issues, /issues?new=1
• Features: issue creation from the toolbar, keyword search, status filter, label filter, combined keyword and status filter

What works

• Keyword search - ET-2 Filter the issues list by a keyword from an existing issue title and verify only matching issues remain

Execution Issues

• ET-1 Create a new issue from the toolbar and verify it appears in the issues list - Passed though notes say new issue not visible; title/status entry unlogged.
• ET-3 Filter the issues list by a status value and verify only issues with that status remain - First dropdown click failed; Closed-status check claimed but not logged.
• ET-4 Filter the issues list by a label and verify only issues with that label remain - First label dropdown click failed; recovered via fallback selector.
• ET-5 Combine a keyword search with a status filter and verify the results match both criteria - No negative assertion; "only one row" claim unlogged.

---

Session analysis — slow (native):

Session Analysis

Creating an issue from the toolbar appears to save it and list it in the issues table with the entered title, status, and priority. Whether the issue detail page shows the entered status is unconfirmed, and the run reports a status mismatch there.

Coverage

• Pages: /issues, /issues?new=1, tester.ts required fields for open ai models #6
• Features: Issue creation from toolbar, issues list, issue detail view, issue edit form

Execution Issues

• ET-1 Create a new issue from the toolbar and verify it appears in the issues list - passed with no expectations checked; detail-page status mismatch unresolved.

---

Session analysis — gated (native):

Session Analysis

Issue creation from the Issues list appears to work: a submitted title showed up in the list as a linked issue. The run did not confirm a clean single submission, because the Create Issue click needed retries.

Coverage

• Pages: /workspace, /issues, /issues?new=1
• Features: New Issue drawer, title, description, and priority fields, Create Issue submission, issues list verification

What works

None confirmed. The one test passed, but its submit step failed and was retried, so it is listed under Execution Issues.

Execution Issues

• ET-1 Create a new issue with a valid title and verify it appears in the issues list - first Create Issue click failed on a malformed selector; retried with other selectors.
• ET-1 Create a new issue with a valid title and verify it appears in the issues list - Create Issue clicked multiple times; duplicate submission not ruled out.