Skip to content

[server] Recover KV tables after complete local disk loss - #4567

Open
swuferhong wants to merge 1 commit into
apache:mainfrom
swuferhong:kv-empty-disk-recovery
Open

swuferhong wants to merge 1 commit into
apache:mainfrom
swuferhong:kv-empty-disk-recovery

Conversation

@swuferhong

Copy link
Copy Markdown
Contributor

Restore empty replica log offsets and writer state from the durable snapshot and remote WAL before rebuilding KV state. Reject missing recovery ranges, clean up failed KV initialization, and resume tiering across a WAL gap covered by the recovered snapshot.

Purpose

Linked issue: close #4566

After complete local disk loss, KV recovery can fail because an empty local log starts at offset 0 while the retained snapshot references a higher offset. This change restores the log boundary before replaying KV state.

  • Initialize empty KV replica logs from the snapshot and committed remote WAL, including the high watermark and leader end-offset snapshot.
  • Restore writer state when recovering to the remote WAL end to preserve duplicate detection for surviving writes.
  • Validate recovery coverage and reject missing ranges or truncated remote log files.
  • Clean up partially initialized KV tablets before retrying recovery.
  • Allow log tiering to resume when the recovered snapshot is ahead of the old remote WAL, while preserving the missing WAL range explicitly.

Non-empty local logs are preserved. Recovery may lose acknowledged writes beyond the recoverable boundary; it does not guarantee that every pre-failure changelog offset remains readable.

Brief change log

Tests

API and Format

Documentation

Restore empty replica log offsets and writer state from the durable snapshot
and remote WAL before rebuilding KV state. Reject missing recovery ranges,
clean up failed KV initialization, and resume tiering across a WAL gap covered
by the recovered snapshot.

Add coverage for all-replica disk loss, acknowledged tail loss, resumed writes
and replication, writer deduplication, expired or truncated remote logs, and
interrupted recovery.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 309/309
AI-Contributed/UT: 433/433
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] KV recovery fails after all TabletServer local data is lost

1 participant