Repository navigation
Description
Investigate the Android smoke:automation-system Back-to-Settings transition without changing the shutdown scope in #4063. Follow-up to umbrella #4045.
Observed evidence
- Failed Android job, PR head
df42fdb25c49206a67cd8340b85042fb0ae2926f, actual CI merge1e4261finto65ec68be9f2b8ed8ea5deef479db334c1b73f2fb, package0.21.36-dev. - Failure artifact contains step history, daemon/request logs, coverage report and
failed-step-88-snapshot.json/PNG. live-automation-scenario.tscaptures the Automation baseline withsnapshot -i, calls bareback, then immediately callsdiff snapshot -i. The assertion at line 201 expects removed “Open automation alert” and added “Open automation lab”. Only afterwards does the test wait for Settings.- Back and diff both returned success. The diff reported zero additions/removals and 21 unchanged Automation nodes. The failure snapshot subsequently showed Settings and “Open automation lab”. Snapshot/Back/diff durations were 765/556/734 ms. These facts do not distinguish delayed navigation from stale capture or another cause.
- Current Android job passed at head
ccdc6d48d35681d335432e400ba0ffb1ed472f22, actual merge2504514. Its artifact recordssmoke:automation-systemcompleted. The scenario source and package version are identical between the two CI merges. This is a later successful run, not a controlled reproduction or proof of root cause.
Required behavior
Establish whether the failure belongs to navigation readiness, snapshot freshness, or the test's baseline/readiness contract. Preserve the meaningful removed/added assertions and snapshot baseline. Do not add sleeps, widen budgets, weaken assertions, or attribute the failure to contention without evidence. Do not change shutdown behavior to hide this failure.
If readiness is the missing precondition, use the existing readiness owner. Verify its baseline behavior on the frozen failing CI tree before changing the test. A settled Back may publish a new baseline, so an added settle flag followed by a separate diff is not automatically equivalent.
Observable completion
- Retain the failed and successful run identities and artifacts.
- In a disposable checkout, record navigation dispatch, immediate capture/diff, readiness, cache/freshness metadata and baseline publication in one bounded discriminating experiment.
- State the demonstrated cause or explicitly retain a diagnostic-only result if it cannot be reproduced.
- Fix the demonstrated boundary and add a credible regression/control that fails before the fix.
- Preserve the Settings navigation and removed/added checks. Run focused proof and serialized exact-head affected validation; record any live evidence at its named SHA.
Dependencies and limits
No implementation dependency on #4063 is established. Diagnosis comes before an implementation decision. The current green Android job does not establish that the failure is fixed, and no base-versus-head comparison establishes that #4063 caused it. Keep this issue in triage until the owning boundary is demonstrated.
Activity
apex-sentinel-staging commentedon Oct 30, 2026
Apex Sentinel · Issue review
Suggested label:
bugThe issue documents a concrete failed Android CI job with logs/artifacts and describes a real test/automation inconsistency: a back transition followed by immediate diff reported zero changes while the later snapshot showed Settings. It gives enough evidence for a maintainer to investigate and reproduce.
Required sections
thymikee commentedon Nov 2, 2026
Two more occurrences of the same signature on 2026-11-02, both on PR branches that don't touch Android. In each, the live Android emulator smoke's snapshot diff right after an action shows no change, as if the capture ran before the screen updated.
expected removed snapshot line containing Open automation alert. Summary: additions 0, removals 0, unchanged 21.expected added snapshot line containing Open automation lab. Summary: additions 0, removals 1, unchanged 20 (assertDiffLine,live-assertions.ts:63).Both passed on rerun, and main stays green. With this issue's Back-to-Settings case, that makes three occurrences of one class: the post-action snapshot diff is captured before the transition settles.
thymikee commentedon Nov 2, 2026
One more occurrence on 2026-11-02: #4160 (iOS and docs only) at af69d17, run 41460910773, job 125329396005. Same
expected added snapshot line containing Open automation lab(assertDiffLine,live-assertions.ts:63), this time with additions 0, removals 9, unchanged 12. That's the fourth time today, the second on this PR, and main stays green.test(android): pin that a bare back keeps arming freshness (#4105)
thymikee commentedon Nov 16, 2026
Measurement: how long the Android helper tree lags the screen after navigation (evidence for the ~2 s lag premise behind the freshness window).
ReactNative_API_35, Android 15), local host, load average 29–45 during the probe.press), helper 0.21.38-dev, persistent helper session.com.android.settings, then plainpress 'text="Battery"'(no--settle). As soon as that command returned, runsnapshot --jsononce and check for the Battery page (Battery percentage). Stop at the first capture that shows it. Each step is a separate CLI process, so the probe cannot see below about 400 ms (process startup).Results:
elapsedMsfor that capture, sorted (ms): 30, 83, 97, 102, 115, 118, 172, 181, 183, 231, 291, 314, 510, 557, 581.cacheInvalidated: true,rootPresent: true,helperTruncated: false, and no missing window roots. Freshness recovery retries: 0/15.Related, same device and build (#4605 benchmark, interleaved base 244d1ff vs branch, N=16 per arm): a navigating
press --settlefollowed immediately bysnapshotorpress, plus an in-place toggle followed bysnapshot. 96 follow-up commands, 0 captures showing the previous screen, 0 freshness retries.This does not reproduce the CI signature (bare
back, thendiff snapshot -i). It also cannot measure lag below the about 400 ms CLI floor.thymikee commentedon Nov 16, 2026
Another occurrence on 2026-11-16: #4605 at ae68544, run 43919820425, job 133483871649. Same failure,
expected added snapshot line containing Open automation lab(live-automation-scenario.ts:202). Diff summary: additions 0, removals 19, unchanged 2.From the artifact:
backtook 138 ms. Thediff snapshot -istarted 210 ms later and took 213 ms, so no freshness retry ran (the first retry waits 250 ms).[button] "Navigate up". The baseline had 21.So the back's freshness window judged this mid-transition tree trustworthy.
androidFreshnessReasontreats a sharp drop as stale only when no node carries a label, value, identifier, or hittable flag. The leftover labeled, hittableNavigate upsatisfies that, so no retry ran. On the same 21 → 2 drop without that label, the classifier returnssharp-drop.perf(android): skip the freshness window after a settle on a proved-f…