Skip to content

Board and Fleet stop running out of memory on a busy board - #6

Merged
marvin-bitterlich merged 2 commits into
mainfrom
fleet-daemon-memory
Nov 6, 2026
Merged

marvin-bitterlich merged 2 commits into
mainfrom
fleet-daemon-memory

Conversation

@marvin-bitterlich

Copy link
Copy Markdown
Contributor

Why?

The Fleet daemon for the Platform board runs out of memory every 6 to 15 minutes. Node stops it at the 4 GB heap limit and the supervisor starts it again, 50 times since 4 November. Each restart drops in-flight observation and spends minutes on a full Board read before Fleet does useful work again.

Heap snapshots of a copy of that board, taken a few minutes apart, show the cause. The daemon kept about 460 complete Board index reads alive, each with its own document list, identity map, diagnostics, settled catalogue and modification times. The index rebuilds after each change in the board folder, and Fleet changes files there every few seconds (for example fleet/health.json), so the reads piled up quickly.

How?

Each index read keeps the file services so that it can load archived records later. It took them by capturing the whole fiber context. That context also holds the stack frame of the buildSnapshot call, and the frame keeps the call's arguments, which include the previous read. So every read kept the read before it, and the reads formed a chain back to the first one.

flowchart LR
  C[read 3] -- archived --> CC[fiber context] -- stack frame --> B[read 2] -- archived --> BC[fiber context] -- stack frame --> A[read 1]
Loading

The read now keeps only FileSystem and Path, picked from the context in the same way watchIndex already does. The Board server and the Fleet daemon share this index, so both stop growing.

A new index story drops a read, rebuilds the index after a change, forces garbage collection and checks that the dropped read is gone. It fails without the fix.

On the same copy of the Platform board, with providers and GitHub cut off, the heap after a forced collection:

Minutes after start 1 2 4 7.5 12
Before 653 MB 1,244 MB 2,014 MB 2,928 MB stopped, still rising
After 557 MB 561 MB 559 MB 559 MB 511 MB

Callouts

  • The leak path goes through Effect.fn: its lazy stack closure shares a V8 context with the call's arguments, so any long-lived value that holds a fiber context keeps the arguments of every Effect.fn call on its stack. This change removes the one place in the index that did that. Other places that keep a whole context for a long time can leak in the same way.

Each index read kept the whole fiber context to load archived records
later. That context holds the buildSnapshot stack frame, whose arguments
include the previous read, so every read kept every earlier one alive and
the Fleet daemon ran out of heap within minutes. Keep only FileSystem and
Path instead.
@marvin-bitterlich
marvin-bitterlich added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit acba5e3 Nov 6, 2026
16 checks passed
@marvin-bitterlich
marvin-bitterlich deleted the fleet-daemon-memory branch November 6, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant