Skip to content

fix(server): retry a failed docs index and log what /ready is waiting on - #9082

Merged
bokelley merged 2 commits into
mainfrom
fix/ready-docs-index-retry
Nov 18, 2026
Merged

bokelley merged 2 commits into
mainfrom
fix/ready-docs-index-retry

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Production deploys have failed since #9072. This PR makes the next attempt either recover or say why it's stuck.

What's failing

Two deploys failed the same way: c925886c82 (#9072) and 954eb9db41 (#9006).

/ready needs the database, the training-agent reporting probe, and Addie web chat. Web chat needs Addie's knowledge index: the docs index plus the external-repo index. In production the external repos are read from the image (SKIP_REPO_SYNC=true), so that part always completes. That leaves the docs index as the only part of the knowledge index that can fail, and it had two gaps:

  1. No retry. If initializeDocsIndex() threw, knowledge search stayed unready "until a later caller retries". Nothing calls it again after boot, so a single failure left the machine unready permanently. That matches what Fly saw.
  2. No reason logged. The failure was logged as { error }, but this logger only serializes errors under err, so the exception never showed up.

The 3.3-rc docs snapshot (#9071 and #9072) took the docs index from 5 versions to 6. I measured indexing locally under the web process's 2 GB heap limit:

Docs versions Docs indexed Peak heap Retained heap Time
5 (b5ac0e1bf7) 6,393 488 MB 426 MB 12 s
6 (now) 8,528 703 MB 588 MB 19 s

That growth is the most likely trigger. I can't prove it: the stuck machine's logs were deleted at rollback, and the forensics log tail was captured afterwards.

Changes

  • Automatic retry (knowledge-search.ts): an incomplete initialization retries on its own after 5s, 15s and 30s, then every 60s until it succeeds. The failure is now logged as { err }, with memory usage, so the exception appears in the logs. getKnowledgeReadiness() reports which part of the index is missing, and stopKnowledgeSearchRetry() cancels a pending retry.
  • /ready names its blocker (http.ts): when the probe fails, it logs a warning listing which checks block it (database, reporting, chat). The warning includes the database or reporting error, and for chat, webChatReadiness() (tool registration, client, and which parts of the knowledge index are ready). It logs when the set of blockers changes, and at most once a minute while it stays the same. It logs once more when the probe recovers. Fly doesn't keep probe response bodies, so this log is the only record. The /ready response body is unchanged.

Expected outcome: if the stuck machine's docs index is throwing, the retry should let the next deploy succeed. If the process is crashing instead, the deploy will still fail, but the logs will now show the blocker and the reason.

Tests

  • New (knowledge-search-initialization.test.ts): a docs index that fails twice and then succeeds becomes ready without any other caller. The test checks the backoff timing and that nothing more is scheduled once ready.
  • Existing: the knowledge suites, plus the suites that mock knowledge search or wire up /ready, pass locally (123 tests across six files). The typecheck is clean.
  • Hook bypass: committed with --no-verify, which @bokelley approved. The pre-commit hook kept failing on timeouts at host load 60–80, in suites this change doesn't touch: hosted compliance, governance agent credentials, the fixed-trace CLIs, and exact-inference. Then it hit its 25-minute ceiling. Pushed with --no-verify too, so CI's full suite gates this PR.

No changeset: this is a server change only.

🤖 Generated with Claude Code

Production deploys have failed since #9072: in each bluegreen rollout one
green machine never passed /ready within Fly's 10-minute wait_timeout and
Fly rolled back (deploys of c925886 and 954eb9d). /ready waits on web
chat, which waits on Addie's knowledge index, and that left two gaps:

- If initializeDocsIndex() threw, knowledge search stayed unready until "a
  later caller" retried, and after boot nothing calls it again, so one
  failed attempt stranded the machine. It now retries on its own with
  backoff (5s, 15s, 30s, then every 60s) until it succeeds.
- The failure was logged as { error }, which the logger does not serialize,
  so the reason never appeared. It is now logged as { err } with memory
  usage.

/ready now logs which checks block it (database, reporting, chat) with the
database and reporting errors and, for chat, which part of its knowledge
index is missing; once per change and at most every 60s while unchanged.
Fly does not keep probe bodies, so this is how the next stuck rollout
names its blocker.

The 3.3-rc docs snapshot (#9071, #9072) took the docs index from 5 to 6
versions: locally, under the web process's 2 GB heap limit, peak heap
during indexing went from 488 MB to 703 MB and indexing time from 12s to
19s. That is the likely trigger, but the stuck machine's own logs were
lost at rollback, so this change makes the next attempt recover or report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Nov 18, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve.

Server-only resilience/diagnostics PR for the Addie docs index: adds automatic backoff retry (5s/15s/30s/60s, unref'd timer, stop fn) for a failed index, corrects { err } logging so exceptions serialize, and adds a throttled /ready blocker log. Reviewer verified operational readiness (retry preserves an already-ready index, blocker set matches the ready condition, error vars in scope, new test covers backoff timing and ready-cancels-retry). No schema/docs/changeset/wire surface touched — server-only, so no changeset is correct.

No blocking or Medium findings. high_risk is true only via (modified) matches on server/src/addie/** and server/src/routes/**, with no medium-or-higher findings on those files, so no escalation trigger fires (rows 3–5 require deletions or a medium finding). gated_paths is false. No author-team gate. Falls through to row 9.

(One Low noted but not blocking: scheduleRetry increments retryAttempt even when a timer is already pending, which can inflate the diagnostic counter — style-level only.)

The /ready diagnostics change added webChatReadiness() to
server/src/routes/addie-chat.ts, so its source hash in the generated
inventory moved. Regenerated with npm run build:addie-tools; npm run
test:addie-tools passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve.

Subsequent pass on an already-approved PR. The delta since the prior approve is a single generated file (server/src/addie/generated/tool-surface-inventory.generated.json) — one regenerated SHA-256 hash for server/src/routes/addie-chat.ts, consistent with the already-reviewed webChatReadiness export. No reviewable code in the delta and no new findings.

Checks:

  • Findings list is empty — no critical/high (row 1 n/a), no medium (rows 4, 5, 8 n/a).
  • gated_paths: false (row 2 n/a).
  • high_risk is true but all reasons are (modified) with no medium findings and no (deleted) entries (rows 3, 5 n/a).
  • Prior decision was approve, not escalate (row 6 n/a).
  • No no-auto-approve team match (row 7 n/a).
  • No protocol surface touched (static/schemas/source/**, docs/reference, dist artifacts) — no changeset/schema/oneOf/immutability concerns.

Falls through to row 9. The prior clean approval stands.

@bokelley
bokelley merged commit f6ada5f into main Nov 18, 2026
31 checks passed
@bokelley
bokelley deleted the fix/ready-docs-index-retry branch November 18, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant