Skip to content

Done Fleet work sleeps until its brief or decisions change - #8

Merged
marvin-bitterlich merged 2 commits into
mainfrom
settled-completions
Nov 6, 2026
Merged

marvin-bitterlich merged 2 commits into
mainfrom
settled-completions

Conversation

@marvin-bitterlich

@marvin-bitterlich marvin-bitterlich commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Why?

The Fleet daemon kept a CPU core busy even when no work moved. After #7, most of the remaining cost came from done and released records. Each one stayed in the scheduler's active set while it had a Board completion action, so every scheduling write for any work woke all of them (up to about 1,700 on the live Board). Each wake applied completion again and ran closeFinishedDecisions, which scanned every Board document to find the decisions of that one work item.

The goal: a done record does no work until something it depends on changes.

How?

Completion now records what it last saw. After completeBoardWork applies completion and closes finished decisions, it saves boardSettledRevision on the record: a hash of the completion action and the work's Board revision. The Board revision covers the brief's full source and the source of each Fleet decision published for that work. It saves the marker only when completion applies to the record (it has a brief path and a worker receipt) and neither completion nor decision closure failed, so refused or failing completions keep retrying as before. Saving the marker alone does not move the record's updatedAt. The field is per cycle, so a new cycle starts without it.

The scheduler skips settled records. A new Fleet operation, settledCompletions, reads the work revisions from the cached Board index in the current read pass and returns the records whose marker still matches. activeRecords drops those records when completion is their only reason to be active. The scheduler's each-loop reader uses it. The manual cycle that stories drive keeps the full active set, because only the event-driven scheduler is woken again by Board events.

A Board edit still wakes the scheduler, because Board watcher events invalidate the scheduler inputs. When a brief or one of its decisions changes, the revision moves, the record becomes active once, completion is applied again (or the reopened brief is set back to done), and the new marker puts the record back to rest.

closeFinishedDecisions({ workId }) now reads a per-snapshot index (workIndex) that maps each work id to its published decisions and holds the snapshot's problems. It no longer reads every document for one work item. The whole-Board sweep without workId is unchanged.

The idleHost story kit now wires Board events to the scheduler the same way the daemon does (wakeSchedulerOnInputs, extracted from hostScheduler). Before, the kit missed those events, so its admission story passed only when the Board index caught the write in time.

The table compares main with this branch on the same copy of the live Board and database (1,339 done and 340 released records), with workers and GitHub unreachable. Each run had 240 s of warm-up, so the one-time marker burst ended before the 180 s measurement window.

Daemon main this branch
CPU 24.9% 18.4%
Main-thread busy 15.7% 11.8%
closeFinishedDecisions and publishedDecision, share of busy time 21% under 0.5%

What remains is mostly record listing and decoding on each scheduling write. The Board server stayed at about 4% CPU in both runs.

Decisions

  • closeFinishedDecisions trigger for done work. For a settled done or released record, decision closure runs again only when the brief's source or the source of one of that work's published decision files changes. A change elsewhere on the Board, or a scheduling write, no longer runs it. The whole-Board sweep keeps its current triggers.
  • New column boardSettledRevision. Migration 38_board_settled_revision adds it to the records and history tables. It is additive and nullable.

Callouts

  • One-time reconcile burst after deploy. No record has a marker yet, so when the new daemon starts, each done and released record whose completion applies reconciles once to record its first marker (up to about 1,700 on the live Board). After that, those records stay asleep. Expect a short period of higher CPU and Board reads right after the redeploy.

Done and released records stayed in the scheduler's active set while
they had a Board completion action, so every scheduling write woke each
of them and closeFinishedDecisions scanned the whole Board per record.

Completion now stores the Board revision it last saw for the work (the
brief's source and its published decisions). The scheduler skips records
whose stored revision matches the current Board, and decision closure for
one work reads a per-snapshot index instead of every document.
The marker is now saved only when completion applies to the record and
neither completion nor decision closure failed, so refused and
inapplicable completions keep retrying as before and old deliveries stay
untouched. Saving the marker alone no longer moves updatedAt. The manual
test cycle keeps the full active set, since only the event-driven
scheduler can rely on Board events to wake settled records.
@marvin-bitterlich
marvin-bitterlich added this pull request to the merge queue Nov 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 6, 2026
@marvin-bitterlich
marvin-bitterlich added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit dbbb52d Nov 6, 2026
16 checks passed
@marvin-bitterlich
marvin-bitterlich deleted the settled-completions branch November 6, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant