You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
flaky: gateway-fleet.e2e drain-and-restart test fails with Unknown android model on a clean base #1628
src/daemon/gateway-fleet.e2e.test.ts, "drains one worker, kills it, restarts the gateway, and proves the surviving worker's lease outlives the restart with renewing resumed (ADR §9/§27a/§30, #119)" failed 1 of 3 runs on merge-base f951fed (found auditing #1627): AssertionError: expected DispatchError: Unknown android model: Pix… to match object { code: 'NO_CAPACITY' }. The other 2 runs passed.
The bug is in the test, not the product. On bug/1628-repro the restarted worker-b gets a 1 s listCatalog latency (catalogLatencyMs, gateway-fleet.e2e.test.ts:671). That makes the race happen on every run. The test fails with the issue's error, at the post-restart assertion:
pnpm exec vitest run --project unit src/daemon/gateway-fleet.e2e.test.ts -t "drains one worker"
AssertionError: expected DispatchError: Unknown android model: Pix… to match object { code: 'NO_CAPACITY' }
- "code": "NO_CAPACITY",
+ "code": "UNKNOWN_MODEL",
❯ src/daemon/gateway-fleet.e2e.test.ts:689:6
Root cause
Defect in the test. After the restart, the wait at gateway-fleet.e2e.test.ts:677-678 checks only that worker-b is connected and drained. registry.connected() (src/gateway/worker-registry.ts:219) sets both as soon as the uplink opens, before refresh({ includeCatalog: true }) lands (src/gateway/worker-link.ts:298). A fresh gateway has no older catalog to fall back on. So when the Pixel-B request comes in, worker-a takes requests and no known worker lists Pixel-B, and assess answers UNKNOWN_MODEL (src/gateway/routing/serviceability.ts:224). Routing does this on purpose: serviceability.test.ts:182 pins it. The pre-drain wait already covers this window (workerServes, C3). The post-restart wait does not.
Simplest fix
Make the post-restart wait also require worker-b's catalog to list the model. Keep the latency, so the wait stays tested.
Done: triage (root cause: the post-restart wait did not wait for worker-b's model list). Fast track set bug:ready. PR #1630 makes the wait require Pixel-B in worker-b's catalog and forces the race with a 1 s catalog delay. Build and audit passed, 11/11 tests, mutation probe red without the fix.
Not done: a complete review. Round 1 came back Assessment: incomplete: the reviewer read the diff and ran the file but did not spawn the spec and code/tests lenses. No findings.
Findings: none. Side issue filed by triage: #1629.
Blocked on: a full review assessment. The maintainer runs the review with its lenses on #1630, or reviews it by hand and merges.
Activity
V3RON commentedon Nov 16, 2026
Reproduction
The bug is in the test, not the product. On
bug/1628-reprothe restarted worker-b gets a 1 slistCataloglatency (catalogLatencyMs,gateway-fleet.e2e.test.ts:671). That makes the race happen on every run. The test fails with the issue's error, at the post-restart assertion:Root cause
Defect in the test. After the restart, the wait at
gateway-fleet.e2e.test.ts:677-678checks only that worker-b isconnectedanddrained.registry.connected()(src/gateway/worker-registry.ts:219) sets both as soon as the uplink opens, beforerefresh({ includeCatalog: true })lands (src/gateway/worker-link.ts:298). A fresh gateway has no older catalog to fall back on. So when thePixel-Brequest comes in, worker-a takes requests and no known worker listsPixel-B, andassessanswersUNKNOWN_MODEL(src/gateway/routing/serviceability.ts:224). Routing does this on purpose:serviceability.test.ts:182pins it. The pre-drain wait already covers this window (workerServes, C3). The post-restart wait does not.Simplest fix
Make the post-restart wait also require worker-b's catalog to list the model. Keep the latency, so the wait stays tested.
Alternatives rejected
waitFortimeout: the problem is the order of events, not how long the wait is.workerServes(workers, "Pixel-B"): it matches any worker, not worker-b by id together withdrained.catalogLatencyMsafter the fix: the race would only show again under load.Risk
NO_CAPACITYexpectation still holds, because worker-b is drained.Fast track
Side findings
Written by an agent.
V3RON commentedon Nov 16, 2026
Handoff
Done: triage (root cause: the post-restart wait did not wait for worker-b's model list). Fast track set
bug:ready. PR #1630 makes the wait requirePixel-Bin worker-b's catalog and forces the race with a 1 s catalog delay. Build and audit passed, 11/11 tests, mutation probe red without the fix.Not done: a complete review. Round 1 came back
Assessment: incomplete: the reviewer read the diff and ran the file but did not spawn the spec and code/tests lenses. No findings.Findings: none. Side issue filed by triage: #1629.
Blocked on: a full review assessment. The maintainer runs the review with its lenses on #1630, or reviews it by hand and merges.
Written by an agent.
V3RON commentedon Nov 16, 2026
Delivery run
Merged: none
Parked: #1628 — round 1 review incomplete (lenses not run), PR #1630 in draft
Written by an agent.