Skip to content

Re-seed or refuse a start below descriptor log coverage - #323

Open
timmclaughlin wants to merge 1 commit into
ClickHouse:mainfrom
timmclaughlin:fix/bootstrap-handoff-start-lsn
Open

timmclaughlin wants to merge 1 commit into
ClickHouse:mainfrom
timmclaughlin:fix/bootstrap-handoff-start-lsn

Conversation

@timmclaughlin

@timmclaughlin timmclaughlin commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This complements #284. That PR keeps the bootstrap handoff until streaming saves its manifest. This PR handles a start below what the descriptor log covers, which #284 does not change.

How we hit it: an object-store bootstrap landed at 5AA/FFDFFFF8, then crashed before the first manifest was written. The restart resumed at the source head and seeded the descriptor log there (valid_from = 5FA/6B000000). The operator then restarted with --start-lsn 5AA/FF000000. That passed every existing check, yet every record in 5AA/FF..5FA/68 was skipped as first record skipped … reason="not_covered", with no error.

#284 stops the first restart from jumping to the head. This PR is still needed for any deployment that already has a log seeded too high, and for any manual --start-lsn below the seed.

Root cause

  • check_start_lsn only compares the start with floor_at_write and head. A log that has never run GC has a floor of 0, so a start below the seed passes.
  • Below the seed, lookup_spanned returns NotCovered for every seeded relation, so their records skip without decoding.

Fix

  • New DescriptorLog::seed_valid_from: returns the lowest valid_from among the seed batch's surviving entries. It returns None when the seed was empty or GC has dropped every seeded entry; floor_at_write bounds history in that case.
  • open_db_desc_log, when the start is below that seed:
    • manifest present: refuse with an error naming both positions and pointing to --ignore-cursor or re-bootstrap.
    • no manifest: no run ever made progress durable against this log, so discard it and re-seed at the start. This is the operator's manual recovery, done automatically.
  • discard_desc_log: now shared with --ignore-cursor. NotFound is ignored; any other error now fails instead of being dropped.
  • Docs: architecture/recovery.md and docs/operations.md describe both behaviours.

Testing

  • desc_log::seed_valid_from_tracks_surviving_seed_entries: reopening the log, a GC that keeps one seeded relation, and a GC that drops the last one.
  • source_db unit tests:
    • A start at or above the seed keeps the log.
    • A start below it with no manifest re-seeds; the test uses the production positions.
    • A start below it with a manifest is refused.
  • Merges cleanly with Fix bootstrap restart gaps #284.
  • On macOS, cargo fmt --check and cargo clippy --workspace --all-targets -D warnings pass. cargo test --lib fails only the 6 tests that also fail on main on macOS (syncfs and other Linux-only paths). The bin unit tests fail only housekeeping::queued_segments_publish_highest_end, which also fails on main on macOS.
  • Not run: a full daemon restart drill. The _ch tests need Linux and ClickHouse.

🤖 Generated with Claude Code

A descriptor log seeded at one restart covers only from its seed's
valid_from. Starting lower (an operator --start-lsn after a crash
between bootstrap and the first manifest) makes every seeded relation's
record read as NotCovered and skip undecoded. With no manifest behind
the log nothing durable depends on it, so discard and re-seed at start;
with one, fail and point at --ignore-cursor or re-bootstrap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@timmclaughlin timmclaughlin changed the title Resume a landed bootstrap at its end, not the source head Re-seed or refuse a start below descriptor log coverage Nov 17, 2026
@timmclaughlin
timmclaughlin force-pushed the fix/bootstrap-handoff-start-lsn branch from caa55f5 to ac2f2ef Compare November 17, 2026 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant