Skip to content

Diagnose Android Back-to-Settings diff capturing the previous screen #4105

Open

Description

@thymikee

Investigate the Android smoke:automation-system Back-to-Settings transition without changing the shutdown scope in #4063. Follow-up to umbrella #4045.

Observed evidence

  • Failed Android job, PR head df42fdb25c49206a67cd8340b85042fb0ae2926f, actual CI merge 1e4261f into 65ec68be9f2b8ed8ea5deef479db334c1b73f2fb, package 0.21.36-dev.
  • Failure artifact contains step history, daemon/request logs, coverage report and failed-step-88-snapshot.json/PNG.
  • live-automation-scenario.ts captures the Automation baseline with snapshot -i, calls bare back, then immediately calls diff snapshot -i. The assertion at line 201 expects removed “Open automation alert” and added “Open automation lab”. Only afterwards does the test wait for Settings.
  • Back and diff both returned success. The diff reported zero additions/removals and 21 unchanged Automation nodes. The failure snapshot subsequently showed Settings and “Open automation lab”. Snapshot/Back/diff durations were 765/556/734 ms. These facts do not distinguish delayed navigation from stale capture or another cause.
  • Current Android job passed at head ccdc6d48d35681d335432e400ba0ffb1ed472f22, actual merge 2504514. Its artifact records smoke:automation-system completed. The scenario source and package version are identical between the two CI merges. This is a later successful run, not a controlled reproduction or proof of root cause.

Required behavior

Establish whether the failure belongs to navigation readiness, snapshot freshness, or the test's baseline/readiness contract. Preserve the meaningful removed/added assertions and snapshot baseline. Do not add sleeps, widen budgets, weaken assertions, or attribute the failure to contention without evidence. Do not change shutdown behavior to hide this failure.

If readiness is the missing precondition, use the existing readiness owner. Verify its baseline behavior on the frozen failing CI tree before changing the test. A settled Back may publish a new baseline, so an added settle flag followed by a separate diff is not automatically equivalent.

Observable completion

  • Retain the failed and successful run identities and artifacts.
  • In a disposable checkout, record navigation dispatch, immediate capture/diff, readiness, cache/freshness metadata and baseline publication in one bounded discriminating experiment.
  • State the demonstrated cause or explicitly retain a diagnostic-only result if it cannot be reproduced.
  • Fix the demonstrated boundary and add a credible regression/control that fails before the fix.
  • Preserve the Settings navigation and removed/added checks. Run focused proof and serialized exact-head affected validation; record any live evidence at its named SHA.

Dependencies and limits

No implementation dependency on #4063 is established. Diagnosis comes before an implementation decision. The current green Android job does not establish that the failure is fixed, and no base-versus-head comparison establishes that #4063 caused it. Keep this issue in triage until the owning boundary is demonstrated.

Activity

  1. apex-sentinel-staging commented on Oct 30, 2026

    @apex-sentinel-staging

    Apex Sentinel · Issue review

    Suggested label: bug

    The issue documents a concrete failed Android CI job with logs/artifacts and describes a real test/automation inconsistency: a back transition followed by immediate diff reported zero changes while the later snapshot showed Settings. It gives enough evidence for a maintainer to investigate and reproduce.

    Required sections

    • Steps to reproduce (missing)
    • Expected and actual behavior (missing)
    • Version or environment (missing)
  2. thymikee commented on Nov 2, 2026

    @thymikee
    MemberAuthor

    Two more occurrences of the same signature on 2026-11-02, both on PR branches that don't touch Android. In each, the live Android emulator smoke's snapshot diff right after an action shows no change, as if the capture ran before the screen updated.

    Both passed on rerun, and main stays green. With this issue's Back-to-Settings case, that makes three occurrences of one class: the post-action snapshot diff is captured before the transition settles.

  3. added
    bugSomething isn't working
    and removed on Nov 2, 2026
  4. thymikee commented on Nov 2, 2026

    @thymikee
    MemberAuthor

    One more occurrence on 2026-11-02: #4160 (iOS and docs only) at af69d17, run 41460910773, job 125329396005. Same expected added snapshot line containing Open automation lab (assertDiffLine, live-assertions.ts:63), this time with additions 0, removals 9, unchanged 12. That's the fourth time today, the second on this PR, and main stays green.

  5. thymikee commented on Nov 16, 2026

    @thymikee
    MemberAuthor

    Measurement: how long the Android helper tree lags the screen after navigation (evidence for the ~2 s lag premise behind the freshness window).

    • Device: Android emulator API 35 (ReactNative_API_35, Android 15), local host, load average 29–45 during the probe.
    • Build: agent-device at ac43329 (perf(android): skip the freshness window after a settle on a proved-fresh tree #4605 branch; the capture path matches main for a plain press), helper 0.21.38-dev, persistent helper session.
    • Method, N=15: relaunch com.android.settings, then plain press 'text="Battery"' (no --settle). As soon as that command returned, run snapshot --json once and check for the Battery page (Battery percentage). Stop at the first capture that shows it. Each step is a separate CLI process, so the probe cannot see below about 400 ms (process startup).

    Results:

    • The first capture after the press showed the new screen in 15/15 runs. No run needed a second capture.
    • When that first capture completed, measured from the press returning, sorted (ms): 427, 681, 687, 698, 699, 717, 723, 806, 940, 966, 1060, 1219, 1237, 1404, 1511 (p50 806, max 1511).
    • Helper elapsedMs for that capture, sorted (ms): 30, 83, 97, 102, 115, 118, 172, 181, 183, 231, 291, 314, 510, 557, 581.
    • Every capture reported cacheInvalidated: true, rootPresent: true, helperTruncated: false, and no missing window roots. Freshness recovery retries: 0/15.

    Related, same device and build (#4605 benchmark, interleaved base 244d1ff vs branch, N=16 per arm): a navigating press --settle followed immediately by snapshot or press, plus an in-place toggle followed by snapshot. 96 follow-up commands, 0 captures showing the previous screen, 0 freshness retries.

    This does not reproduce the CI signature (bare back, then diff snapshot -i). It also cannot measure lag below the about 400 ms CLI floor.

  6. thymikee commented on Nov 16, 2026

    @thymikee
    MemberAuthor

    Another occurrence on 2026-11-16: #4605 at ae68544, run 43919820425, job 133483871649. Same failure, expected added snapshot line containing Open automation lab (live-automation-scenario.ts:202). Diff summary: additions 0, removals 19, unchanged 2.

    From the artifact:

    • The bare back took 138 ms. The diff snapshot -i started 210 ms later and took 213 ms, so no freshness retry ran (the first retry waits 250 ms).
    • The captured tree had 2 nodes: the scroll area and [button] "Navigate up". The baseline had 21.
    • The only warning was the recent-drop warning, which is published only when the capture has no freshness annotation.

    So the back's freshness window judged this mid-transition tree trustworthy. androidFreshnessReason treats a sharp drop as stale only when no node carries a label, value, identifier, or hittable flag. The leftover labeled, hittable Navigate up satisfies that, so no retry ran. On the same 21 → 2 drop without that label, the classifier returns sharp-drop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions