Repository navigation
Board and Fleet stop running out of memory on a busy board - #6
Merged
Merged
Conversation
Each index read kept the whole fiber context to load archived records later. That context holds the buildSnapshot stack frame, whose arguments include the previous read, so every read kept every earlier one alive and the Fleet daemon ran out of heap within minutes. Keep only FileSystem and Path instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
The Fleet daemon for the Platform board runs out of memory every 6 to 15 minutes. Node stops it at the 4 GB heap limit and the supervisor starts it again, 50 times since 4 November. Each restart drops in-flight observation and spends minutes on a full Board read before Fleet does useful work again.
Heap snapshots of a copy of that board, taken a few minutes apart, show the cause. The daemon kept about 460 complete Board index reads alive, each with its own document list, identity map, diagnostics, settled catalogue and modification times. The index rebuilds after each change in the board folder, and Fleet changes files there every few seconds (for example
fleet/health.json), so the reads piled up quickly.How?
Each index read keeps the file services so that it can load archived records later. It took them by capturing the whole fiber context. That context also holds the stack frame of the
buildSnapshotcall, and the frame keeps the call's arguments, which include the previous read. So every read kept the read before it, and the reads formed a chain back to the first one.The read now keeps only
FileSystemandPath, picked from the context in the same waywatchIndexalready does. The Board server and the Fleet daemon share this index, so both stop growing.A new index story drops a read, rebuilds the index after a change, forces garbage collection and checks that the dropped read is gone. It fails without the fix.
On the same copy of the Platform board, with providers and GitHub cut off, the heap after a forced collection:
Callouts
Effect.fn: its lazy stack closure shares a V8 context with the call'sarguments, so any long-lived value that holds a fiber context keeps the arguments of everyEffect.fncall on its stack. This change removes the one place in the index that did that. Other places that keep a whole context for a long time can leak in the same way.