Skip to content

docs(help): add platform checks and audit to the dogfood workflow - #4600

Open
thymikee wants to merge 1 commit into
feat/static-accessibility-audit-v2from
docs/dogfood-platform-checks
Open

thymikee wants to merge 1 commit into
feat/static-accessibility-audit-v2from
docs/dogfood-platform-checks

Conversation

@thymikee

@thymikee thymikee commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Summary

help dogfood now has a Platform checks: block after Goal/Loop: on the first screen and after a screen changes meaningfully, run audit first and then snapshot -i (audit performs its own capture, so it goes first), act on the snapshot refs or the pinned refs audit prints, and report its findings as accessibility findings; on iOS test Back and the edge-back swipe where a screen was pushed; on Android test system Back and applicable sheet dismissal; probe a dismissible sheet's unchanged swipe-down (expect dismiss), edited dismissal (expect confirm or save), and explicit Cancel; check destructive confirmation or undo, swipe-action alternatives, and each field's return key against its task; exercise each control once with reversible test data. One line carries the exceptions: autosave, explicit discard semantics, mandatory flows, nondismissible presentations. It links the Apple Human Interface Guidelines and Material accessibility guidance and does not restate what audit already checks (target size, names).

The new command is discoverable outside that topic: one line each in skills/agent-device/SKILL.md and skills/dogfood/SKILL.md, and the recommended agent rule in website/docs/docs/agent-setup.md now mentions audit. Both routing lines send an agent to agent-device help dogfood first, so a normal app-driving task still starts immediately and only QA or accessibility work takes the audit pass. The dogfood skill line carries the trigger (first screen, and after a screen changes meaningfully) and applies only when the target supports snapshots, so snapshotless targets (Vega) skip it.

Stacked on #4592.

Evidence

  • agent-device help dogfood renders the block on lines 18-25, inside the first 30 lines.
  • Bench (--case dogfood-mode, Sonnet, 3 runs): 3/3 with the block, 9/9 checks each, and every planned pass now emits audit before snapshot -i. With the earlier draft that only said "run audit", Sonnet stopped planning snapshot -i and the case failed usesSnapshotI (2/3), which is why the line names both commands in order. The new runsPlatformAudit expectation is falsified by the base text: base runs produce plans with no audit command.
  • One sampling run of the same command scored 2/3 because that plan chose logs-only evidence and missed the pre-existing capturesStrongIssueEvidence check; the runs before and after it score 3/3.

Test plan

  • pnpm format
  • pnpm build
  • npx vitest run over src/commands/schema/cli-help-{first-screen,workflows}.test.ts, scripts/__tests__/help-conformance-{bench,topic-coverage,expectation-falsification,error-recovery-coverage,find-verification,sample-outputs}.test.ts, scripts/__tests__/agent-setup-startup-contract.test.ts, scripts/__tests__/simulator-skills-contract.test.ts, scripts/__tests__/npm-skills-exclusion.test.ts (155 tests)
  • node scripts/help-conformance-bench.mjs --case dogfood-mode --runner claude:sonnet --repeat 3 → 3/3, 9/9 checks each
  • CI on a4ba7fcba4: all gates green (20 pass, 2 skipping). The first Coverage run failed only on test/integration/interaction-contract/coordinate.contract.test.ts timing out at 5s under runner load; it passes locally on this head and a rerun is green. This PR touches no test file or device command.

🤖 Generated with Apex

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.33 MB 6.33 MB +1.2 kB
Package (unpacked) 6.33 MB 6.33 MB +1.2 kB
Package (download) 1.92 MB 1.92 MB +458 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 39.7 ms 28.7 ms -11.0 ms
CLI --help 67.8 ms 54.8 ms -12.9 ms

@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

Not in the reviewers' prompt (6 files)
  • scripts/help-conformance-cases.mjs: no reviewer was selected for this change
  • skills/agent-device/SKILL.md: no reviewer was selected for this change
  • skills/dogfood/SKILL.md: no reviewer was selected for this change
  • src/commands/schema/cli-help-first-screen.test.ts: no reviewer was selected for this change
  • src/commands/schema/cli-help-workflows.ts: no reviewer was selected for this change
  • website/docs/docs/agent-setup.md: no reviewer was selected for this change

Reviewed with this repository's own Sentinel configuration (a30accf38081).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of aadf30b4 · 16.11.2026 16:56 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

Not in the reviewers' prompt (6 files)
  • scripts/help-conformance-cases.mjs: no reviewer was selected for this change
  • skills/agent-device/SKILL.md: no reviewer was selected for this change
  • skills/dogfood/SKILL.md: no reviewer was selected for this change
  • src/commands/schema/cli-help-first-screen.test.ts: no reviewer was selected for this change
  • src/commands/schema/cli-help-workflows.ts: no reviewer was selected for this change
  • website/docs/docs/agent-setup.md: no reviewer was selected for this change

Reviewed with this repository's own Sentinel configuration (b3c1a3c8abd9).

Review of 59b2272d · 16.11.2026 16:23 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

Not in the reviewers' prompt (6 files)
  • scripts/help-conformance-cases.mjs: no reviewer was selected for this change
  • skills/agent-device/SKILL.md: no reviewer was selected for this change
  • skills/dogfood/SKILL.md: no reviewer was selected for this change
  • src/commands/schema/cli-help-first-screen.test.ts: no reviewer was selected for this change
  • src/commands/schema/cli-help-workflows.ts: no reviewer was selected for this change
  • website/docs/docs/agent-setup.md: no reviewer was selected for this change

Reviewed with this repository's own Sentinel configuration (244d1ff68344).

Review of 35230e29 · 16.11.2026 16:03 UTC

Apex Sentinel · ✅ Approve

No open findings.

Not in the reviewers' prompt (6 files)
  • scripts/help-conformance-cases.mjs: no reviewer was selected for this change
  • skills/agent-device/SKILL.md: no reviewer was selected for this change
  • skills/dogfood/SKILL.md: no reviewer was selected for this change
  • src/commands/schema/cli-help-first-screen.test.ts: no reviewer was selected for this change
  • src/commands/schema/cli-help-workflows.ts: no reviewer was selected for this change
  • website/docs/docs/agent-setup.md: no reviewer was selected for this change

Reviewed with this repository's own Sentinel configuration (244d1ff68344).

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files

You’re at about 96% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread skills/agent-device/SKILL.md Resolved
Comment thread skills/dogfood/SKILL.md Resolved
Comment thread scripts/help-conformance-cases.mjs Resolved
Comment thread skills/agent-device/SKILL.md Resolved
@thymikee
thymikee force-pushed the docs/dogfood-platform-checks branch from 35230e2 to 59b2272 Compare November 16, 2026 16:22
@thymikee
thymikee force-pushed the docs/dogfood-platform-checks branch from 59b2272 to aadf30b Compare November 16, 2026 16:55
@thymikee

Copy link
Copy Markdown
Member Author

I found nothing that blocks this change at 59b2272. All 17 checks are green and there are no conflicts. The delta is read against the base head 2e90b57, since the audit code lives only in open PR #4592.

Not blocking, take or leave: the audit line in skills/agent-device/SKILL.md:18 has no support or trigger condition, and Vega has no snapshot (commands.md:340), so you could prefix it with "When the platform supports snapshot, and after a screen change,". The sentence in skills/dogfood/SKILL.md:27 holds two instructions, so you could split it, and do the same for the first line of Platform checks in the help. The comment at scripts/help-conformance-cases.mjs:149 says audit on the initial screen, but the regex matches an audit line anywhere in the plan, so "plans an audit" would be accurate.

The open review thread on the missing Vega condition (#4600 (comment)) still apply, as do the three lower-priority threads on the same notes. The thread at #4600 (comment) does not apply: audit keeps the ref frame and issues only a partial publication, so snapshot refs stay valid after it. You can resolve that one.

This can merge after base #4592 lands.

@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at aadf30b: I found no problems in the code, and all 17 checks are green. There are no conflicts. Not blocking, and you can take or leave these: the audit line at https://github.com/callstack/agent-device/blob/aadf30b/skills/agent-device/SKILL.md#L18 (and the matching clause in agent-setup.md) tells every app-driving task to run audit, while the sibling lines say a normal task should start immediately, and the issue scopes the audit nudge to help dogfood, so you could drop it or point it at help dogfood; the matcher at https://github.com/callstack/agent-device/blob/aadf30b/scripts/help-conformance-cases.mjs#L151 only matches agent-device audit at the start of a line, so a numbered or indented plan would miss it, and /\bagent-device\s+audit\b/i would match the neighbouring cases; and the Platform checks block at https://github.com/callstack/agent-device/blob/aadf30b/src/commands/schema/cli-help-workflows.ts#L222 goes a bit past the audit, snapshot and iOS back checks the issue asks for, and the external URLs may not load offline. This PR is stacked on #4592, so merge it after #4592 lands.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 16, 2026
@thymikee
thymikee added this pull request to stack #4598 November 16, 2026 20:56
@thymikee
thymikee force-pushed the docs/dogfood-platform-checks branch from aadf30b to a4ba7fc Compare November 16, 2026 21:19
@thymikee

Copy link
Copy Markdown
Member Author

Took the first two, kept the third pair.

  • skills/agent-device/SKILL.md and the matching clause in agent-setup.md now route instead of instructing: "For exploratory QA or accessibility checks, read agent-device help dogfood; its platform checks run agent-device audit." A normal app-driving task still starts immediately, and the pinned-ref detail stays with the dogfood surfaces that act on those refs. The dogfood skill line is unchanged.
  • The matcher is now /\bagent-device\s+audit\b/i, so a numbered or indented plan line matches. The comment says presence, not order, and names the tolerance.

Kept as is: the Platform checks: block scope is the approved design for this feature (Feature D of the mobile a11y design), so the sheet, destructive-action, and return-key probes stay together rather than split across PRs. The two links are optional further reading at the end of the block, not a step any command depends on, so an offline run loses nothing.

Bench after both changes: --case dogfood-mode --runner claude:sonnet --repeat 3 → 3/3, 9/9 each. One earlier run of the same command scored 2/3, where a plan chose logs-only evidence and missed capturesStrongIssueEvidence; that check is untouched by these edits and is plan sampling, not a regression. Help and first-screen tests, the skill and agent-setup contracts, and the affected gate are green; details in the updated Test plan.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed a4ba7fc: still ready. The rewritten commit keeps the change the earlier review of aadf30b approved, all 22 checks pass, and there are no conflicts or open threads. It is still stacked on #4592, so merge it after #4592 lands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant