Skip to content

flaky: gateway-fleet.e2e drain-and-restart test fails with Unknown android model on a clean base #1628

Open

Description

@V3RON

src/daemon/gateway-fleet.e2e.test.ts, "drains one worker, kills it, restarts the gateway, and proves the surviving worker's lease outlives the restart with renewing resumed (ADR §9/§27a/§30, #119)" failed 1 of 3 runs on merge-base f951fed (found auditing #1627): AssertionError: expected DispatchError: Unknown android model: Pix… to match object { code: 'NO_CAPACITY' }. The other 2 runs passed.

Written by an agent.

Activity

  1. added
    bug:newReported. An agent may triage it (rule 8); the fast track may set it ready (rule 17).
    flaky-testNames a test that fails without a code change. Delivery skips its base-commit check.
    on Nov 16, 2026
  2. self-assigned this
    on Nov 16, 2026
  3. V3RON commented on Nov 16, 2026

    @V3RON
    ContributorAuthor

    Reproduction

    The bug is in the test, not the product. On bug/1628-repro the restarted worker-b gets a 1 s listCatalog latency (catalogLatencyMs, gateway-fleet.e2e.test.ts:671). That makes the race happen on every run. The test fails with the issue's error, at the post-restart assertion:

    pnpm exec vitest run --project unit src/daemon/gateway-fleet.e2e.test.ts -t "drains one worker"
    
    AssertionError: expected DispatchError: Unknown android model: Pix… to match object { code: 'NO_CAPACITY' }
    - "code": "NO_CAPACITY",
    + "code": "UNKNOWN_MODEL",
     ❯ src/daemon/gateway-fleet.e2e.test.ts:689:6
    

    Root cause

    Defect in the test. After the restart, the wait at gateway-fleet.e2e.test.ts:677-678 checks only that worker-b is connected and drained. registry.connected() (src/gateway/worker-registry.ts:219) sets both as soon as the uplink opens, before refresh({ includeCatalog: true }) lands (src/gateway/worker-link.ts:298). A fresh gateway has no older catalog to fall back on. So when the Pixel-B request comes in, worker-a takes requests and no known worker lists Pixel-B, and assess answers UNKNOWN_MODEL (src/gateway/routing/serviceability.ts:224). Routing does this on purpose: serviceability.test.ts:182 pins it. The pre-drain wait already covers this window (workerServes, C3). The post-restart wait does not.

    Simplest fix

    Make the post-restart wait also require worker-b's catalog to list the model. Keep the latency, so the wait stays tested.

    expect(view?.catalog?.some((entry) => entry.models.includes("Pixel-B"))).toBe(true);
    pnpm exec vitest run --project unit src/daemon/gateway-fleet.e2e.test.ts   # 3 runs
    Tests  11 passed (11)   ×3
    

    Alternatives rejected

    • A longer waitFor timeout: the problem is the order of events, not how long the wait is.
    • workerServes(workers, "Pixel-B"): it matches any worker, not worker-b by id together with drained.
    • Dropping catalogLatencyMs after the fix: the race would only show again under load.

    Risk

    Fast track

    • Defect: yes
    • Fix tried: yes
    • Nothing public added or broken: yes
    • No ADR or rule changed: yes
    • One fix: yes

    Side findings

    Written by an agent.

  4. removed their assignment
    on Nov 16, 2026
  5. added
    bug:readyAn agent may fix it.
    fast-trackBug set ready by the fast-track check, not the maintainer (rule 17).
    and removed
    bug:newReported. An agent may triage it (rule 8); the fast track may set it ready (rule 17).
    on Nov 16, 2026
  6. self-assigned this
    on Nov 16, 2026
  7. V3RON commented on Nov 16, 2026

    @V3RON
    ContributorAuthor

    Handoff

    Done: triage (root cause: the post-restart wait did not wait for worker-b's model list). Fast track set bug:ready. PR #1630 makes the wait require Pixel-B in worker-b's catalog and forces the race with a 1 s catalog delay. Build and audit passed, 11/11 tests, mutation probe red without the fix.

    Not done: a complete review. Round 1 came back Assessment: incomplete: the reviewer read the diff and ran the file but did not spawn the spec and code/tests lenses. No findings.

    Findings: none. Side issue filed by triage: #1629.

    Blocked on: a full review assessment. The maintainer runs the review with its lenses on #1630, or reviews it by hand and merges.

    Written by an agent.

  8. added
    bug:blockedAgent stopped on a blocker in its handoff. Maintainer re-adds the label it came from.
    and removed
    bug:readyAn agent may fix it.
    on Nov 16, 2026
  9. removed their assignment
    on Nov 16, 2026
  10. V3RON commented on Nov 16, 2026

    @V3RON
    ContributorAuthor

    Delivery run

    Merged: none
    Parked: #1628 — round 1 review incomplete (lenses not run), PR #1630 in draft

    Written by an agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug:blockedAgent stopped on a blocker in its handoff. Maintainer re-adds the label it came from.fast-trackBug set ready by the fast-track check, not the maintainer (rule 17).flaky-testNames a test that fails without a code change. Delivery skips its base-commit check.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions