Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ jobs:
- name: Install PostgreSQL ${{ matrix.pg-major }} (PGDG)
run: |
sudo systemctl stop postgresql 2>/dev/null || true
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
curl -fsSL --retry 5 --retry-all-errors https://www.postgresql.org/media/keys/ACCC4CF8.asc \
| sudo gpg --dearmor -o /usr/share/keyrings/postgres-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/postgres-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main ${{ matrix.pg-major }}" \
| sudo tee /etc/apt/sources.list.d/pgdg.list
Expand All @@ -163,7 +163,7 @@ jobs:
- name: Install ClickHouse (server + client)
run: |
sudo apt-get install -y apt-transport-https ca-certificates
curl -fsSL https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key \
curl -fsSL --retry 5 --retry-all-errors https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key \
| sudo gpg --dearmor -o /usr/share/keyrings/clickhouse-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb stable main" \
| sudo tee /etc/apt/sources.list.d/clickhouse.list
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential curl gnupg lsb-release
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
curl -fsSL --retry 5 --retry-all-errors https://www.postgresql.org/media/keys/ACCC4CF8.asc \
| sudo gpg --dearmor -o /usr/share/keyrings/postgres-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/postgres-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main ${{ matrix.pg-major }}" \
| sudo tee /etc/apt/sources.list.d/pgdg.list
Expand Down
25 changes: 18 additions & 7 deletions docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ insert batch
- restart can resend acknowledged-nearby rows, generated table engine deduplicates them during merge or `FINAL`
- `tee` tables using engines other than `ReplacingMergeTree` keep duplicate rows after retries
- destination queries without `FINAL` can observe multiple row versions until background merge
- initial-load rows use backup start as `_lsn` and can show stale or incorrect
values until backup-window replay writes newer row versions, see
[large values](#large-values)
- bounded ClickHouse retry exhaustion stops daemon, supervisor restart continues from persisted floor

## Initial loads
Expand All @@ -94,8 +97,12 @@ insert batch
- restored TOAST page images in backup-window WAL mirror every chunk they
carry, repairing pages the backup copied mid-write. They date from the page's
own version, so walked chunks never outrank them
- an external value the mirror cannot reassemble stops the load rather than
substituting one; rerun against a fresher backup, or load the table with
- greenfield bootstrap reads deferred external values at heap page LSN or
backup start, whichever is later. It skips a backup row whose value cannot be
reassembled only if backup-window replay deleted or updated that row's heap
slot on a route that keeps deletes. Skipped rows increment
`walshadow_bootstrap_replay_covered_rows_total`. Other failures stop loading,
as do all such failures during per-table loads. Use a newer backup or
`initial_load = 'copy'`
- a multixact xmax the backup's `pg_multixact` cannot bound stops the load, with
the same remedies
Expand Down Expand Up @@ -137,11 +144,15 @@ Default `clickhouse` value mode uses persistent TOAST chunk mirrors. Missing
required mirror tables stop replication, see
[large toasted values](destination-tables.md#large-toasted-values)

Reused TOAST value IDs can leave ambiguous generations in backup chunk mirrors
when hint bits do not prove older chunks dead. Baseline rows and later
unchanged-pointer updates both resolve from those mirrors. Chunk lookup orders
by `(ver, blkno, offnum)` for determinism, not generation correctness: a newer
generation at a lower TID can lose when versions tie
PostgreSQL reuses a TOAST value ID only after all chunks of its previous value
are gone. During backup, chunk mirrors can contain both old and new values
under that ID. Initial-load rows read values at heap page LSN or backup start,
so chunks from restored page images can cause them to read an old value.
Backup-window replay writes correct values at commit LSN. If reassembly fails,
greenfield bootstrap skips a backup row only when replay deleted or updated its
slot.
After replay, `FINAL` returns correct values; reads without it can still return
old values until merges finish

### Shadow value mode

Expand Down
18 changes: 8 additions & 10 deletions plans/bootstrap.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@ rows and deferred TOAST. Scratch writes alone cannot acknowledge deferred tuples

## Completion

Extend [pending-row settlement test](../tests/bootstrap_pending_settle_ch.rs)
beyond direct-bootstrap INSERT commit and rollback. Cover UPDATE and DELETE,
multixact updaters, subtransactions, deferred external values, and transactions
begun inside backup window. Repeat for object-store bootstrap and per-table
backup loads with explicit retained-history assumptions
[Pending-row tests](../tests/bootstrap_pending_shapes_ch.rs) cover updates,
deletes, multixact updaters, subtransactions, and external values during direct
bootstrap. Add transactions begun during backup. Repeat for object-store
bootstrap and per-table backup loads, stating which WAL history must remain
available

Restart before settlement and after promotion but before ledger persistence.
Interrupt pending-table cleanup and staged publication. Cover outcomes arriving
Expand All @@ -46,11 +46,9 @@ convergence boundaries

Backup-based initial loads issue no source SQL scans for user rows. Explicit
`initial_load = "copy"` still scans selected table and remains `init`'s default.
Baseline external values in backup modes resolve from walked chunk mirrors,
so reused TOAST generations need a physical proof of age: backup transaction
logs and tuple-location order do not
establish it, and no source read compensates. Keep WAL replay ordered behind
required chunk history
Backup rows read external values from chunk mirrors populated during page walk
and window replay. [Bootstrap ordering](bootstrap_ordering.md) covers read
bounds and replacement evidence. Keep WAL replay behind required chunk writes

To remove all source SQL, also build descriptors from copied catalogs, convert
types while preserving source OIDs without reading source catalogs, and replace
Expand Down
91 changes: 83 additions & 8 deletions src/backfill/bootstrap_marker.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,18 @@ pub struct BootstrapMarker {
pub attempts: u32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub backup_name: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub handoff: Option<Handoff>,
}

/// Keep bootstrap position until streaming saves its manifest and descriptor log
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct Handoff {
pub end_lsn: u64,
pub timeline: u32,
/// Earliest record among transactions open at window seal
#[serde(default, skip_serializing_if = "Option::is_none")]
pub open_floor: Option<u64>,
}

#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
Expand Down Expand Up @@ -109,10 +121,21 @@ impl BootstrapMarker {
.context("persist bootstrap marker")
}

/// End the attempt. Pending rows already in ClickHouse must be recorded
/// first, so a restart can still publish them
pub async fn complete(dir: &Path, _: PendingRecorded) -> Result<()> {
ExtractedCheckpoint::clear(dir).await?;
/// Record pending rows first so restart can publish them.
/// Keep marker until streaming saves its resume state
pub async fn hand_off(dir: &Path, handoff: Handoff, _: PendingRecorded) -> Result<()> {
let marker = Self::read(dir)?.context("bootstrap marker vanished before handoff")?;
Self {
handoff: Some(handoff),
..marker
}
.write(dir)
.await?;
ExtractedCheckpoint::clear(dir).await
}

/// Clear marker after streaming saves its manifest and descriptor log
pub async fn complete(dir: &Path) -> Result<()> {
Self::clear(dir).await
}

Expand All @@ -129,6 +152,7 @@ impl BootstrapMarker {
Self {
attempts: self.attempts + 1,
backup_name: self.backup_name.clone(),
handoff: None,
}
}

Expand All @@ -152,8 +176,12 @@ impl BootstrapMarker {
}
}

pub fn handed_off(dir: &Path) -> Result<Option<Handoff>> {
Ok(BootstrapMarker::read(dir)?.and_then(|m| m.handoff))
}

pub fn pending_attempt(dir: &Path, mode: BootstrapMode) -> Result<Option<BootstrapMarker>> {
let Some(marker) = BootstrapMarker::read(dir)? else {
let Some(marker) = BootstrapMarker::read(dir)?.filter(|m| m.handoff.is_none()) else {
return Ok(None);
};
anyhow::ensure!(
Expand Down Expand Up @@ -271,6 +299,7 @@ async fn first_attempt(data_dir: &Path, pin: Option<String>) -> Result<Bootstrap
let marker = BootstrapMarker {
attempts: 1,
backup_name: pin,
handoff: None,
};
marker.write(data_dir).await?;
Ok(marker)
Expand Down Expand Up @@ -309,6 +338,7 @@ mod tests {
BootstrapMarker {
attempts,
backup_name: Some("base_original".into()),
handoff: None,
}
}

Expand All @@ -325,6 +355,44 @@ mod tests {
assert_eq!(BootstrapMarker::read(dir).unwrap(), None);
}

#[tokio::test]
async fn handoff_outlives_the_attempt_until_streaming_completes() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path();
pinned(2).write(dir).await.unwrap();
ExtractedCheckpoint {
backup_name: "base_original".into(),
start_lsn: 0x100,
end_lsn: 0x200,
timeline: 1,
deferred_spools: Vec::new(),
handback_spools: Vec::new(),
}
.write(dir)
.await
.unwrap();
let handoff = Handoff {
end_lsn: 0x200,
timeline: 1,
open_floor: Some(0x180),
};
BootstrapMarker::hand_off(dir, handoff, PendingRecorded::nothing_pending())
.await
.unwrap();
assert_eq!(ExtractedCheckpoint::read(dir).unwrap(), None);
assert_eq!(handed_off(dir).unwrap(), Some(handoff));
assert_eq!(BootstrapMarker::read(dir).unwrap().unwrap().attempts, 2);
for mode in [
BootstrapMode::Off,
BootstrapMode::Direct,
BootstrapMode::ObjectStore,
] {
assert_eq!(pending_attempt(dir, mode).unwrap(), None);
}
BootstrapMarker::complete(dir).await.unwrap();
assert_eq!(handed_off(dir).unwrap(), None);
}

#[test]
fn no_marker_is_not_a_pending_attempt() {
let tmp = tempfile::tempdir().unwrap();
Expand Down Expand Up @@ -378,6 +446,7 @@ mod tests {
BootstrapMarker {
attempts: 1,
backup_name,
handoff: None,
}
.write(tmp.path())
.await
Expand Down Expand Up @@ -517,6 +586,7 @@ mod tests {
let unpinned = BootstrapMarker {
attempts: 1,
backup_name: None,
handoff: None,
};
assert!(
resolve_backup(&storage, configured, Some(&unpinned))
Expand Down Expand Up @@ -591,15 +661,20 @@ mod tests {
assert!(version.exists());
assert!(data_dir.join(EXTRACTED_FILENAME).exists());

BootstrapMarker::complete(&data_dir, PendingRecorded::nothing_pending())
let handoff = Handoff {
end_lsn: 0x200,
timeline: 1,
open_floor: None,
};
BootstrapMarker::hand_off(&data_dir, handoff, PendingRecorded::nothing_pending())
.await
.unwrap();
assert!(!data_dir.join(EXTRACTED_FILENAME).exists());
assert_eq!(BootstrapMarker::read(&data_dir).unwrap(), None);
assert_eq!(handed_off(&data_dir).unwrap(), Some(handoff));

second.write(&data_dir).await.unwrap();
std::fs::create_dir(data_dir.join(EXTRACTED_FILENAME)).unwrap();
let err = BootstrapMarker::complete(&data_dir, PendingRecorded::nothing_pending())
let err = BootstrapMarker::hand_off(&data_dir, handoff, PendingRecorded::nothing_pending())
.await
.unwrap_err();
assert_eq!(err.to_string(), "clear extracted checkpoint");
Expand Down
41 changes: 33 additions & 8 deletions src/bin/stream/bootstrap.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1137,16 +1137,18 @@ pub(crate) async fn run_bootstrap(
.context(format!("bootstrap: chmod 0700 {dir}"))?;
}

BootstrapMarker::complete(&shadow_data_dir, recorded).await?;
let handoff = bootstrap_marker::Handoff {
end_lsn: outcome.end.end_lsn,
timeline: outcome.start.timeline,
open_floor,
};
BootstrapMarker::hand_off(&shadow_data_dir, handoff, recorded).await?;
#[cfg(feature = "test-support")]
walshadow::crash_test::hit_now(walshadow::crash_test::Point::BootstrapHandoff);

timing.finish();
Ok((
BootstrapHandoff {
end_lsn: outcome.end.end_lsn,
timeline: outcome.start.timeline,
open_floor,
shadow: running_shadow,
},
BootstrapHandoff::new(handoff, running_shadow),
BootstrapMetrics {
progress,
oracle: oracle_stats,
Expand Down Expand Up @@ -1185,6 +1187,15 @@ pub(crate) struct BootstrapHandoff {
}

impl BootstrapHandoff {
pub(crate) fn new(handoff: bootstrap_marker::Handoff, shadow: Option<OwnedShadow>) -> Self {
Self {
end_lsn: handoff.end_lsn,
timeline: handoff.timeline,
open_floor: handoff.open_floor,
shadow,
}
}

/// Source or archive must retain crossing transaction records
pub(crate) fn resume_lsn(&self) -> u64 {
self.open_floor.unwrap_or(self.end_lsn).min(self.end_lsn)
Expand Down Expand Up @@ -1256,6 +1267,8 @@ pub(crate) async fn bootstrap_build_mapping(
pub(crate) enum ShadowStart {
Bootstrap(PathBuf),
Rebootstrap(PathBuf, BootstrapMarker),
/// Load finished, streaming never persisted its resume state
Handoff(PathBuf, bootstrap_marker::Handoff),
Resume(PathBuf),
}

Expand All @@ -1266,7 +1279,10 @@ impl ShadowStart {

pub(crate) fn data_dir(&self) -> &Path {
match self {
Self::Bootstrap(d) | Self::Rebootstrap(d, _) | Self::Resume(d) => d,
Self::Bootstrap(d)
| Self::Rebootstrap(d, _)
| Self::Handoff(d, _)
| Self::Resume(d) => d,
}
}
}
Expand All @@ -1285,6 +1301,15 @@ pub(crate) fn resolve_shadow_start(args: &Args, mode: BootstrapMode) -> Result<S
other.display(),
);
}
if let Some(handoff) = bootstrap_marker::handed_off(dir)? {
tracing::info!(
target: "walshadow::bootstrap",
data_dir = %dir.display(),
end_lsn = %format_pg_lsn(handoff.end_lsn),
"resuming streaming after bootstrap",
);
return Ok(ShadowStart::Handoff(dir.clone(), handoff));
}
if let Some(marker) = bootstrap_marker::pending_attempt(dir, mode)? {
return Ok(ShadowStart::Rebootstrap(dir.clone(), marker));
}
Expand Down
11 changes: 9 additions & 2 deletions src/bin/stream/session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -389,7 +389,7 @@ async fn session_body(
let bootstrap_plan = resolve_bootstrap(args, ch_config.as_ref())?;
let shadow_start = resolve_shadow_start(args, bootstrap_plan.mode)?;
if ch_config.as_ref().is_some_and(|c| c.toast.mode.is_shadow())
&& let ShadowStart::Resume(dir) = &shadow_start
&& let ShadowStart::Resume(dir) | ShadowStart::Handoff(dir, _) = &shadow_start
{
walshadow::filter::shadow_relations::ShadowRelations::load(dir).await?;
}
Expand Down Expand Up @@ -451,6 +451,8 @@ async fn session_body(
.context("bootstrap")?;
bootstrap_metrics = Some(stage);
Some(handoff)
} else if let ShadowStart::Handoff(_, handoff) = &shadow_start {
Some(BootstrapHandoff::new(*handoff, None))
} else {
None
};
Expand Down Expand Up @@ -1110,7 +1112,8 @@ async fn session_body(
wal_seg_size: WAL_SEG_SIZE as u32,
},
lineage: &lineage,
manifest_present: manifest_at_boot.is_some(),
// A crash after saving manifest can leave descriptor logs missing
manifest_present: manifest_at_boot.is_some() && bootstrap_end_lsn.is_none(),
start_lsn_override,
raw_start,
aligned,
Expand All @@ -1119,6 +1122,10 @@ async fn session_body(
);
}
let desc_log = desc_logs[primary_index].clone();
// Manifest and descriptor logs now preserve resume position
if bootstrap_end_lsn.is_some() {
walshadow::bootstrap_marker::BootstrapMarker::complete(shadow_start.data_dir()).await?;
}
let all_desc_logs = walshadow::desc_log::DescriptorLogs::new(desc_logs.clone());

let stash_live = xact_buffer.lock().await.stash_live();
Expand Down
Loading
Loading