diff --git a/architecture/README.md b/architecture/README.md
index b11a113d..e3f420d4 100644
--- a/architecture/README.md
+++ b/architecture/README.md
@@ -37,25 +37,38 @@ planning committed transactions
## Commit pipeline
-
+
`BufferingDecoderSink` and `ReorderSink` share one record-queue worker
-`ReorderSink` plans each commit and places its rows onto the batcher in order
-Batcher seals per-table batches. A separate conversion pool resolves PostgreSQL
-types before inserters send complete Native blocks. Conversion concurrency
-matches oracle capacity; `[ch].inserter_pool_size` sizes the inserter pool
+`ReorderSink` plans each commit and places it onto the batcher in order.
+Planning routes, validates and detoasts rows but leaves them WAL bytes; each
+outbox flush hands the rows of many commits to one encode job on the runtime,
+which writes per-table column chunks. Batcher awaits jobs in order, merges
+chunks into Native blocks and groups blocks into per-table INSERTs. A
+separate conversion pool resolves PostgreSQL types, then serializes each block
+once and keeps prepared bytes until its INSERT is durable. Compression follows
+`[ch] compression`; inserters rebuild blocks only when a connection rejects
+prepared bytes. Conversion concurrency matches oracle capacity;
+`[ch].inserter_pool_size` sizes inserter pool
Each inserter owns a ClickHouse connection and takes any ready batch
Sequence numbers identify work slices, not necessarily whole transactions
Only a commit's final slice publishes its LSN, after all earlier work finishes
Bounded queues and a shared payload budget limit work in flight; transaction
-and plan data can spill to disk
-
-Queue worker runs on a dedicated thread and receives batches with heap decode
-started ahead of dispatch. It checks decoded results against descriptor history
-and decodes again if needed. Commits drain only their own transaction trees.
-Consecutive small commits can share row chunks while retaining separate
-acknowledgements
+and plan data can spill to disk. Each tail's output pool holds batches from
+batcher to acknowledgement and reserves seal workspace apart from them, so
+conversion finishes however much output waits on inserters
+
+Queue worker runs on a dedicated thread. A dispatcher scans each batch ahead of
+it to start TOAST store reads for values updated rows keep. Ordinary heap
+records stay WAL bytes under their record-time descriptor until commit;
+planning reads tuples in place, resolved external values patching them.
+Records resolved from stash at commit carry the same bytes under their resolved
+descriptor. Compressed inline values, numeric, JSONB, and fast defaults can
+need row scratch. Commits drain only their own
+transaction trees. Consecutive small commits can share column blocks while
+retaining separate acknowledgements. COPY and backup loads carry tuple bytes or
+COPY fields and encode their own column chunks
## Design boundaries
diff --git a/architecture/overview.svg b/architecture/overview.svg
index f7ac034d..23790359 100644
--- a/architecture/overview.svg
+++ b/architecture/overview.svg
@@ -68,7 +68,7 @@
lookup
ReorderSink → commit pipeline
- Inline rows → batcher → conversion pool → inserters
+ Column chunks → batcher → conversion / sealing → inserters
SchemaEvent / TRUNCATE use an ordered DDL barrier
diff --git a/architecture/values.md b/architecture/values.md
index 3eef993c..d8dba34e 100644
--- a/architecture/values.md
+++ b/architecture/values.md
@@ -33,6 +33,18 @@ acknowledgement alone is insufficient because restart could still reread a
pre-drop row. Rewrite barriers handle changed physical generations without
unconditionally wiping history needed by earlier reads
+## Value forms
+
+Values stay in forms PostgreSQL produced until they become ClickHouse Native
+column bytes: stored datums read in place from WAL, page images and TOAST
+chunks, or `typoutput` text from SQL-read fast defaults and COPY casts. COPY's
+binary send forms convert to stored datums on arrival where a local codec reads
+the type; types only PostgreSQL converts keep their send bytes for its
+`typreceive`, falling back to text casts without binary I/O. Each destination
+column compiles one conversion from its source type, so there is no decoded
+intermediate value. Detoasted values hold their stored bytes, so resident
+accounting counts what was fetched rather than a rendering
+
## Convert unfamiliar types
After decompression, values outside local codec set are sent to PostgreSQL
@@ -41,9 +53,13 @@ columns for destination insertion. Batch conversion avoids a database round trip
per row and keeps type semantics in PostgreSQL
Conversion workers run between batcher and inserters, so PostgreSQL conversion
-overlaps ClickHouse insertion. Batches carrying only local columns pass through
-without a PostgreSQL request. Inserters combine local and converted columns into
-complete Native blocks and retain them across insert retries
+overlaps ClickHouse insertion. Blocks carrying only local columns pass through
+without a PostgreSQL request. Conversion workers combine local and converted
+columns into complete Native blocks, serialize and compress each once, and
+release column buffers. Inserters send prepared bytes directly. A connection
+rejecting prepared bytes, such as one using different compression, gets one
+block decoded and rebuilt at a time. Retries resend every block of an INSERT;
+prepared blocks remain owned until every required destination acknowledges
Conversion failure stops batch with column and row context. Module pins output
settings to make conversion reproducible. Greenfield bootstrap uses a temporary
diff --git a/architecture/values.svg b/architecture/values.svg
index 1525b98b..5604d733 100644
--- a/architecture/values.svg
+++ b/architecture/values.svg
@@ -1,6 +1,6 @@
TOAST resolution and PostgreSQL type conversion
- Transaction heaps and TOAST chunks enter planning and resolution. ToastResolver reads or writes versioned ClickHouse mirrors for values outside the current transaction. InsertBatcher stores locally encoded column slabs or Oracle cells. A separate conversion pool sends ENCODE_NATIVE requests to shadow and validates returned Native columns. Inserters combine them with local columns and send complete ClickHouse INSERTs.
+ Transaction WAL and TOAST chunks enter planning and resolution. ToastResolver reads or writes versioned ClickHouse mirrors for values outside current transaction. Commit planning encodes column chunks; InsertBatcher merges them across commits. Conversion workers request and validate oracle columns, combine them with local columns, and seal prepared Native blocks. Inserters retain those blocks across retries and send complete ClickHouse INSERTs.
@@ -15,11 +15,11 @@
XactBuffer
- Heap tuples + TOAST chunks
+ WAL + cells + TOAST chunks
In-memory or spilled
ReorderSink / ToastResolver
- Plan + resolve external values
+ Resolve TOAST; encode chunks
Persist TOAST before dispatch
pg_toast_<relid>
@@ -29,10 +29,10 @@
put/fetch
- Rows placed inline, in commit order
+ Column chunks, in commit order
- InsertBatcher / TableEncoder
- ColumnEncoding selects local slabs or Oracle cells; both remain in one sealed InsertBatch
+ InsertBatcher
+ Merge column chunks across commits; encode load rows; batch local columns and Oracle cells
@@ -48,20 +48,21 @@
walshadow worker
Bridge: ENCODE_NATIVE
- One request per batch
+ One request per block
Return partial Native block
- Native response checks
- Validate row count, names,
- column types and block shape
+ Validate + seal
+ Check shape; combine columns
+ Serialize + compress once
Native columns
Inserter
- Combine Native columns
- Retain block across retries
-
+ Send prepared blocks
+ Retain until acknowledged
+ Rebuild only if incompatible
+
diff --git a/architecture/workers.svg b/architecture/workers.svg
index 661d3819..45831e19 100644
--- a/architecture/workers.svg
+++ b/architecture/workers.svg
@@ -1,6 +1,6 @@
Commit pipeline, queues and acknowledgements
- ReorderSink plans commits and places rows inline in commit order. One batcher seals per-table batches. Conversion workers resolve PostgreSQL types before inserters send complete Native blocks to ClickHouse. Bounded queues separate stages. Register, Placed and Acked events track work slices; only contiguous completed work advances emitter_ack_lsn.
+ ReorderSink plans commits into encoded column chunks in commit order. One batcher merges chunks into per-table batches and encodes load rows. Conversion workers resolve PostgreSQL types and seal prepared Native blocks before inserters send them to ClickHouse. Bounded queues separate stages. Register, Placed and Acked events track work slices; only contiguous completed work advances emitter_ack_lsn.
@@ -15,11 +15,11 @@
ReorderSink · record-queue worker
- Plan commits and place rows inline, in commit order
+ Plan column chunks; dispatch in commit order
Assign sequence numbers to work slices
- Rows + FlushAll · bounded FIFO queue
+ Chunks / load rows / FlushAll · bounded FIFO
@@ -36,7 +36,7 @@
Conversion pool · W tasks
- Convert PostgreSQL types; pass local columns through
+ Resolve oracle columns; serialize and compress blocks
resolver[1]
@@ -52,7 +52,7 @@
Inserter pool · N tasks
- Build Native blocks; retain each block across insert retries
+ Send prepared blocks; retain across INSERT retries
inserter[1]
diff --git a/bench/README.md b/bench/README.md
index c6d94f45..7b1b6237 100644
--- a/bench/README.md
+++ b/bench/README.md
@@ -32,19 +32,21 @@ cargo bench --bench spool_read -- --plan-only --planner --mapped --rows 1000 --p
cargo bench --bench spool_read -- --plan-only --planner --mapped --rows 200000 --payload-bytes 256 --relations 8 --control-every 10000
```
-Use `--planner` to run `Planner::plan_batch` before replay and place replayed rows
-in batches. Use `--mapped` to assign frozen routes. By default, heaps have no
-mapping: planning counts and discards them without writing them to spool.
+Use `--planner` to run `Planner::plan_batch` over drained slices built before
+timing starts. Rows enter as WAL insert records and encode from tuple views;
+add `--cells` to plan decoded rows through the cell adapter instead. Use
+`--mapped` to assign frozen routes over an int4 and a text column. By default,
+heaps have no mapping: planning counts and discards them without encoding.
Use `--relations` to cycle through relation descriptors, one per row.
Use `--control-every` to insert unrelated schema controls and clear cached routes.
Use `--plan-only` to skip deferred spool measurements. Plans spill to disk when
-they exceed default 1 MiB memory limit.
+encoded chunks exceed default 1 MiB memory limit.
-Read `plan_total` for combined write and replay time per sample, including row
-placement with `--planner`. For plans in memory, byte counts estimate memory use;
-for spilled plans, they report encoded file size. Route caching measures descriptor
-identity checks and reuse of cached references. Measurements exclude mapping
-resolution, WAL draining, TOAST I/O, control execution, and PostgreSQL/ClickHouse I/O.
+Read `plan_total` for combined write and replay time per sample. For plans in
+memory, byte counts report allocated chunk bytes; for spilled plans, encoded
+file size. Route caching measures descriptor identity checks and reuse of cached
+references. Measurements exclude mapping resolution, WAL draining, TOAST I/O,
+control execution, and PostgreSQL/ClickHouse I/O.
## pgbench workload
diff --git a/benches/bootstrap_pump.rs b/benches/bootstrap_pump.rs
index bed8ae44..c7925e4d 100644
--- a/benches/bootstrap_pump.rs
+++ b/benches/bootstrap_pump.rs
@@ -8,6 +8,11 @@
//! concurrent tar parts so shared-sink contention shows up.
//! - `drain`: synthetic `BackfillTuple`s → `pipeline::bootstrap::drain` →
//! metrics-only tail. Covers routing plus the `BatcherMsg` hop.
+//! - `load`: `pump` feeding `drain` feeding the real batcher, sealed batches
+//! dropped instead of inserted. Ends at the same `InsertBatch` blocks
+//! whichever stage encodes rows, so it compares across encoder designs.
+//!
+//! `--text N` appends a text column of N bytes to every tuple.
//!
//! ```text
//! cargo bench --bench bootstrap_pump -- \
@@ -41,12 +46,13 @@ use walshadow::backfill::backup_source::{
BackupSink, BackupSource, EndInfo, PumpStats, PumpTarget, StartInfo, pump_tar_to_sink,
};
use walshadow::backfill_bootstrap::{BootstrapConfig, spawn_greenfield_bootstrap};
-use walshadow::backup_page_walk::{BackfillTuple, CatalogMap, PAGE_BYTES};
-use walshadow::heap_decoder::ColumnValue;
+use walshadow::backup_page_walk::{BackfillTuple, CatalogMap, PAGE_BYTES, TupleData};
+use walshadow::datum::build;
+use walshadow::decode::datum::DatumRef;
use walshadow::mapping::{ColumnMapping, TableMapping, TableTarget};
use walshadow::pipeline::bootstrap;
use walshadow::pos::{EmitterAck, Monotone};
-use walshadow::schema::{INT4OID, RelAttr, RelDescriptor, RelName, ReplIdent};
+use walshadow::schema::{INT4OID, RelAttr, RelDescriptor, RelName, ReplIdent, TEXTOID};
use walshadow::toast::ToastResolver;
const DB_NODE: u32 = 5;
@@ -68,6 +74,8 @@ struct Shape {
tuples_per_page: usize,
/// int4 columns per tuple
columns: usize,
+ /// Trailing text column bytes, none at 0
+ text: usize,
parallelism: usize,
}
@@ -79,6 +87,7 @@ impl Default for Shape {
pages_per_segment: 256,
tuples_per_page: 100,
columns: 8,
+ text: 0,
parallelism: 4,
}
}
@@ -98,7 +107,35 @@ impl Shape {
}
}
-fn descriptor(filenode: u32, columns: usize) -> Arc {
+fn descriptor(filenode: u32, shape: Shape) -> Arc {
+ let mut attributes: Vec = (0..shape.columns)
+ .map(|i| RelAttr {
+ attnum: i as i16 + 1,
+ name: format!("c{i}"),
+ type_oid: INT4OID,
+ typmod: -1,
+ not_null: false,
+ dropped: false,
+ type_name: "int4".into(),
+ type_byval: true,
+ type_len: 4,
+ type_align: 'i',
+ type_storage: 'p',
+ missing_default: None,
+ })
+ .collect();
+ if shape.text > 0 {
+ attributes.push(RelAttr {
+ attnum: shape.columns as i16 + 1,
+ name: "body".into(),
+ type_oid: TEXTOID,
+ type_name: "text".into(),
+ type_byval: false,
+ type_len: -1,
+ type_storage: 'x',
+ ..attributes[0].clone()
+ });
+ }
RelDescriptor {
rfn: RelFileNode {
spc_node: 1663,
@@ -113,62 +150,80 @@ fn descriptor(filenode: u32, columns: usize) -> Arc {
persistence: 'p',
ancestors: Vec::new(),
replident: ReplIdent::Default { pk_attnums: None },
- attributes: (0..columns)
- .map(|i| RelAttr {
- attnum: i as i16 + 1,
- name: format!("c{i}"),
- type_oid: INT4OID,
- typmod: -1,
- not_null: false,
- dropped: false,
- type_name: "int4".into(),
- type_byval: true,
- type_len: 4,
- type_align: 'i',
- type_storage: 'p',
- missing_default: None,
- })
- .collect(),
+ attributes,
}
.into()
}
-fn mapping(filenode: u32, columns: usize) -> TableMapping {
+fn mapping(filenode: u32, shape: Shape) -> TableMapping {
+ let mut columns: Vec = (0..shape.columns)
+ .map(|i| ColumnMapping {
+ src_attnum: i as i16 + 1,
+ target_name: format!("c{i}"),
+ target_type: "Int32".into(),
+ type_pinned: false,
+ })
+ .collect();
+ if shape.text > 0 {
+ columns.push(ColumnMapping {
+ src_attnum: shape.columns as i16 + 1,
+ target_name: "body".into(),
+ target_type: "String".into(),
+ type_pinned: false,
+ });
+ }
TableMapping {
target: TableTarget::new("bench", &format!("t{filenode}")),
- columns: (0..columns)
- .map(|i| ColumnMapping {
- src_attnum: i as i16 + 1,
- target_name: format!("c{i}"),
- target_type: "Int32".into(),
- type_pinned: false,
- })
- .collect(),
+ columns,
tees: Vec::new(),
}
}
-/// One 8 KiB heap page carrying `tuples` live int4-only tuples, laid out
-/// as PG writes them: line pointers up from the header, tuple bodies down
-/// from the page end.
-fn heap_page(tuples: usize, columns: usize, seed: i32) -> Vec {
- let body = TUPLE_HEADER + 4 * columns;
+/// Stored text datum: 1-byte header below 127 bytes, else 4-byte
+fn text_datum(len: usize) -> Vec {
+ let mut v = if len < 127 {
+ vec![(((len + 1) << 1) | 1) as u8]
+ } else {
+ (((len + 4) as u32) << 2).to_le_bytes().to_vec()
+ };
+ v.resize(v.len() + len, b'x');
+ v
+}
+
+/// One 8 KiB heap page carrying `tuples` live tuples, laid out as PG writes
+/// them: line pointers up from the header, MAXALIGNed tuple bodies down
+/// from the page end. Returns page, tuples written, and wrapping sum of
+/// int4 values
+fn heap_page(tuples: usize, shape: Shape, seed: i32) -> (Vec, usize, i64) {
+ let columns = shape.columns;
+ let text = (shape.text > 0).then(|| text_datum(shape.text));
+ let natts = columns + usize::from(text.is_some());
+ let body = TUPLE_HEADER + 4 * columns + text.as_ref().map_or(0, Vec::len);
+ let stride = body.next_multiple_of(8);
let mut page = vec![0u8; PAGE_BYTES];
let mut written = 0usize;
+ let mut digest = 0i64;
for i in 0..tuples {
- let off = PAGE_BYTES - (i + 1) * body;
+ let Some(off) = PAGE_BYTES.checked_sub((i + 1) * stride) else {
+ break;
+ };
if off < SIZE_OF_PAGE_HEADER + (i + 1) * SIZE_OF_ITEM_ID {
break;
}
let xmin = 100u32 + i as u32;
page[off..off + 4].copy_from_slice(&xmin.to_le_bytes());
- page[off + 18..off + 20].copy_from_slice(&(columns as u16).to_le_bytes());
+ page[off + 18..off + 20].copy_from_slice(&(natts as u16).to_le_bytes());
page[off + 20..off + 22].copy_from_slice(&0u16.to_le_bytes());
page[off + 22] = TUPLE_HEADER as u8;
for c in 0..columns {
let at = off + TUPLE_HEADER + c * 4;
let v = seed + (i as i32) * 31 + c as i32;
page[at..at + 4].copy_from_slice(&v.to_le_bytes());
+ digest = digest.wrapping_add(i64::from(v));
+ }
+ if let Some(t) = &text {
+ let at = off + TUPLE_HEADER + columns * 4;
+ page[at..at + t.len()].copy_from_slice(t);
}
let slot = SIZE_OF_PAGE_HEADER + i * SIZE_OF_ITEM_ID;
let raw = ((off as u32) & 0x7FFF) | (LP_NORMAL << 15) | (((body as u32) & 0x7FFF) << 17);
@@ -176,30 +231,32 @@ fn heap_page(tuples: usize, columns: usize, seed: i32) -> Vec {
written = i + 1;
}
let pd_lower = SIZE_OF_PAGE_HEADER + written * SIZE_OF_ITEM_ID;
- let pd_upper = PAGE_BYTES - written * body;
+ let pd_upper = PAGE_BYTES - written * stride;
page[12..14].copy_from_slice(&(pd_lower as u16).to_le_bytes());
page[14..16].copy_from_slice(&(pd_upper as u16).to_le_bytes());
- page
+ (page, written, digest)
}
-/// One tar part holding `segments_per_part` `base//` entries.
-async fn build_part(shape: Shape, part: usize) -> Vec {
+/// One tar part holding `segments_per_part` `base//` entries,
+/// with tuples and wrapping sum of int4 values written
+async fn build_part(shape: Shape, part: usize) -> (Vec, u64, i64) {
let mut writer = Writer::new(Vec::new());
+ let mut digest = 0i64;
+ let mut tuples = 0u64;
for s in 0..shape.segments_per_part {
let filenode = FIRST_FILENODE + (part * shape.segments_per_part + s) as u32;
let mut body = Vec::with_capacity(shape.pages_per_segment * PAGE_BYTES);
for p in 0..shape.pages_per_segment {
- body.extend_from_slice(&heap_page(
- shape.tuples_per_page,
- shape.columns,
- (p * 1_000) as i32,
- ));
+ let (page, n, d) = heap_page(shape.tuples_per_page, shape, (p * 1_000) as i32);
+ body.extend_from_slice(&page);
+ tuples += n as u64;
+ digest = digest.wrapping_add(d);
}
let path = format!("base/{DB_NODE}/{filenode}");
let entry = Entry::new(path, Kind::Regular, body.len() as u64, 0o600, 0);
writer.append(&entry, body.as_slice()).await.unwrap();
}
- writer.finish().await.unwrap()
+ (writer.finish().await.unwrap(), tuples, digest)
}
/// Object-store-shaped source over pre-built tars: one `pump_tar_to_sink`
@@ -273,17 +330,40 @@ impl Report {
}
}
-async fn bench_pump(shape: Shape) -> Report {
+/// Tars, their tuple count and int4 digest, and the catalog naming them
+async fn backup(shape: Shape) -> (Vec>, u64, i64, CatalogMap) {
let mut parts = Vec::with_capacity(shape.parts);
+ let (mut tuples, mut digest) = (0u64, 0i64);
for p in 0..shape.parts {
- parts.push(build_part(shape, p).await);
+ let (part, n, d) = build_part(shape, p).await;
+ parts.push(part);
+ tuples += n;
+ digest = digest.wrapping_add(d);
}
-
let mut catalog = CatalogMap::new();
for i in 0..shape.segments() {
- catalog.insert(descriptor(FIRST_FILENODE + i as u32, shape.columns));
+ catalog.insert(descriptor(FIRST_FILENODE + i as u32, shape));
}
+ (parts, tuples, digest, catalog)
+}
+fn pump_detail(
+ progress: &walshadow::backfill_bootstrap::BootstrapProgress,
+) -> Vec<(&'static str, f64)> {
+ let ld = |a: &AtomicU64| a.load(Ordering::Relaxed) as f64;
+ vec![
+ ("pages_walked", ld(&progress.page_walk.pages_walked)),
+ ("decode_secs", ld(&progress.page_walk.decode_nanos) / 1e9),
+ ("tap_secs", ld(&progress.pump.sink_chunk_nanos) / 1e9),
+ (
+ "channel_block_secs",
+ ld(&progress.page_walk.channel_block_nanos) / 1e9,
+ ),
+ ]
+}
+
+async fn bench_pump(shape: Shape) -> Report {
+ let (parts, tuples, expected, catalog) = backup(shape).await;
let tmp = tempfile::tempdir().unwrap();
let cfg = BootstrapConfig::new(tmp.path().join("data"));
let progress = cfg.progress.clone();
@@ -293,46 +373,48 @@ async fn bench_pump(shape: Shape) -> Report {
});
let started = Instant::now();
- let (mut rx, pump) = spawn_greenfield_bootstrap(cfg, source, catalog, false);
+ let (mut rx, pump) = spawn_greenfield_bootstrap(cfg, source, catalog.clone(), false);
let drained = tokio::spawn(async move {
- let mut n = 0u64;
+ let (mut n, mut digest) = (0u64, 0i64);
while let Some(slab) = rx.recv().await {
- // Touch the payload so the decode can't be optimised away
- n += slab.iter().map(|t| t.columns.len() as u64).sum::();
+ for t in &slab {
+ let rel = catalog.get(t.rfn.db_node, t.rfn.rel_node).unwrap();
+ let view = t.view(&rel).unwrap().unwrap();
+ n += 1;
+ for d in &view.datums {
+ if let DatumRef::Fixed(b) = d {
+ let v = i32::from_le_bytes((*b).try_into().unwrap());
+ digest = digest.wrapping_add(i64::from(v));
+ }
+ }
+ }
}
- n
+ (n, digest)
});
pump.await.unwrap().unwrap();
- let touched = drained.await.unwrap();
+ let (walked, digest) = drained.await.unwrap();
let elapsed = started.elapsed().as_secs_f64();
+ assert_eq!(digest, expected, "walked values differ from written pages");
+ assert_eq!(walked, tuples);
- let ld = |a: &AtomicU64| a.load(Ordering::Relaxed) as f64;
Report {
label: "pump".into(),
elapsed_secs: elapsed,
- tuples: touched / shape.columns as u64,
- bytes: ld(&progress.pump.bytes_tapped) as u64,
- detail: vec![
- ("pages_walked", ld(&progress.page_walk.pages_walked)),
- ("decode_secs", ld(&progress.page_walk.decode_nanos) / 1e9),
- ("tap_secs", ld(&progress.pump.sink_chunk_nanos) / 1e9),
- (
- "channel_block_secs",
- ld(&progress.page_walk.channel_block_nanos) / 1e9,
- ),
- ],
+ tuples,
+ bytes: progress.pump.bytes_tapped.load(Ordering::Relaxed),
+ detail: pump_detail(&progress),
}
}
async fn bench_drain(shape: Shape) -> Report {
let rels: Vec> = (0..shape.segments())
- .map(|i| descriptor(FIRST_FILENODE + i as u32, shape.columns))
+ .map(|i| descriptor(FIRST_FILENODE + i as u32, shape))
.collect();
let mut catalog = CatalogMap::new();
let mut tables = foldhash::HashMap::default();
for r in &rels {
catalog.insert(r.clone());
- tables.insert(r.rel_name.clone(), mapping(r.oid, shape.columns));
+ tables.insert(r.rel_name.clone(), mapping(r.oid, shape));
}
let routes = Arc::new(walshadow::mapping::shared(tables));
@@ -343,25 +425,32 @@ async fn bench_drain(shape: Shape) -> Report {
let (tup_tx, tup_rx) = tokio::sync::mpsc::channel::>(16);
let per_rel = shape.pages_per_segment as u64 * shape.tuples_per_page as u64;
- let columns = shape.columns;
+ let text = "x".repeat(shape.text);
let feeder = tokio::spawn(async move {
for r in rels {
// Slab at the walk's grain so the drain sees production shape
for slab_start in (0..per_rel).step_by(shape.tuples_per_page * 16) {
let end = (slab_start + (shape.tuples_per_page * 16) as u64).min(per_rel);
let slab: Vec = (slab_start..end)
- .map(|i| BackfillTuple {
- rfn: r.rfn,
- xid: 100 + i as u32,
- xmax: 0,
- infomask: 0,
- source_lsn: START_LSN,
- page_lsn: 0,
- blkno: (i / 100) as u32,
- offnum: (i % 100) as u16 + 1,
- columns: (0..columns)
- .map(|c| Some(ColumnValue::Int4(i as i32 + c as i32)))
- .collect(),
+ .map(|i| {
+ let mut cells: Vec<_> = (0..shape.columns)
+ .map(|c| Some(build::int4(i as i32 + c as i32)))
+ .collect();
+ if shape.text > 0 {
+ cells.push(Some(build::text(&text)));
+ }
+ BackfillTuple {
+ rfn: r.rfn,
+ xid: 100 + i as u32,
+ xmax: 0,
+ infomask: 0,
+ source_lsn: START_LSN,
+ page_lsn: 0,
+ blkno: (i / 100) as u32,
+ offnum: (i % 100) as u16 + 1,
+ data: TupleData::from_cells(&cells),
+ pending: None,
+ }
})
.collect();
if tup_tx.send(slab).await.is_err() {
@@ -390,22 +479,93 @@ async fn bench_drain(shape: Shape) -> Report {
let elapsed = started.elapsed().as_secs_f64();
feeder.await.unwrap();
tail.close(msg_tx, ack).await;
+ assert_eq!(outcome.rows_routed, shape.tuples());
Report {
label: "drain".into(),
elapsed_secs: elapsed,
tuples: outcome.rows_routed,
- // Int4 payload only; the interesting rate here is rows, not bytes
- bytes: outcome.rows_routed * 4 * shape.columns as u64,
+ // Row payload only; the interesting rate here is rows, not bytes
+ bytes: outcome.rows_routed * (4 * shape.columns + shape.text) as u64,
detail: vec![("seqs", outcome.next_seq as f64)],
}
}
+/// Backup pages through walk, drain, and batcher to sealed `InsertBatch`es,
+/// which drop instead of inserting
+async fn bench_load(shape: Shape) -> Report {
+ let (parts, tuples, _, catalog) = backup(shape).await;
+ let tables = catalog
+ .descriptors()
+ .map(|d| (d.rel_name.clone(), mapping(d.oid, shape)))
+ .collect();
+ let routes = Arc::new(walshadow::mapping::shared(tables));
+ let tmp = tempfile::tempdir().unwrap();
+ let cfg = BootstrapConfig::new(tmp.path().join("data"));
+ let progress = cfg.progress.clone();
+ let source = Box::new(MemSource {
+ parts,
+ parallelism: shape.parallelism,
+ });
+ let emitter = walshadow::ch_emitter::EmitterConfig::default();
+ let stats = Arc::new(walshadow::ch_emitter::EmitterStats::default());
+ let fatal = walshadow::pipeline::Fatal::new();
+ let batcher = walshadow::pipeline::batcher::BatcherConfig {
+ row_budget: emitter.row_budget,
+ byte_budget: emitter.byte_budget,
+ inserters: 1,
+ flush_timeout: std::time::Duration::from_secs(1),
+ };
+
+ let started = Instant::now();
+ let (msg_tx, ack, tail) =
+ walshadow::pipeline::tail::spawn_discard(batcher, stats.clone(), fatal.clone());
+ let (rx, pump) = spawn_greenfield_bootstrap(cfg, source, catalog.clone(), false);
+ let drain = bootstrap::drain(
+ rx,
+ catalog,
+ walshadow::emit::route::freeze_routes(&routes, None, &Default::default()),
+ msg_tx.clone(),
+ ack.clone(),
+ stats.clone(),
+ ToastResolver::disabled(),
+ bootstrap::Deferral::Rejected,
+ foldhash::HashSet::default(),
+ None,
+ );
+ let (outcome, pumped) = tokio::join!(drain, pump);
+ pumped.unwrap().unwrap();
+ let outcome = outcome.unwrap();
+ tail.finish(msg_tx, ack, outcome.next_seq, &fatal)
+ .await
+ .unwrap();
+ let elapsed = started.elapsed().as_secs_f64();
+ assert_eq!(outcome.rows_routed, tuples);
+ assert_eq!(
+ stats.insertbatch_rows_in.load(Ordering::Relaxed),
+ tuples,
+ "every row reached the batcher"
+ );
+
+ let mut detail = pump_detail(&progress);
+ detail.push((
+ "batches",
+ stats.insertbatch_batches_out.load(Ordering::Relaxed) as f64,
+ ));
+ Report {
+ label: "load".into(),
+ elapsed_secs: elapsed,
+ tuples,
+ bytes: progress.pump.bytes_tapped.load(Ordering::Relaxed),
+ detail,
+ }
+}
+
fn usage() -> ! {
eprintln!(
- "usage: bootstrap_pump [--list] [--stage pump|drain|all] [--parts N] \
+ "usage: bootstrap_pump [--list] [--stage pump|drain|load|all] [--parts N] \
[--segments-per-part N] [--pages N] [--tuples-per-page N] [--columns N] \
- [--parallelism N]"
+ [--text N] [--parallelism N]"
);
std::process::exit(2)
}
@@ -432,6 +592,7 @@ fn main() {
"--pages" => shape.pages_per_segment = val(&next).max(1),
"--tuples-per-page" => shape.tuples_per_page = val(&next).max(1),
"--columns" => shape.columns = val(&next).max(1),
+ "--text" => shape.text = val(&next),
"--parallelism" => shape.parallelism = val(&next).max(1),
// cargo bench passes --bench through
"--bench" => {
@@ -445,11 +606,13 @@ fn main() {
let rt = tokio::runtime::Runtime::new().unwrap();
println!(
- "shape: {} segments x {} pages x {} tuples x {} int4 cols, {} parts, parallelism {}",
+ "shape: {} segments x {} pages x {} tuples x {} int4 cols + {} text bytes, {} parts, \
+ parallelism {}",
shape.segments(),
shape.pages_per_segment,
shape.tuples_per_page,
shape.columns,
+ shape.text,
shape.parts,
shape.parallelism,
);
@@ -464,4 +627,7 @@ fn main() {
if stage == "drain" || stage == "all" {
rt.block_on(bench_drain(shape)).print();
}
+ if stage == "load" || stage == "all" {
+ rt.block_on(bench_load(shape)).print();
+ }
}
diff --git a/benches/jsonb.rs b/benches/jsonb.rs
index cdd31066..4e687f0d 100644
--- a/benches/jsonb.rs
+++ b/benches/jsonb.rs
@@ -5,7 +5,16 @@ use std::hint::black_box;
use std::time::{Duration, Instant};
use clap::Parser;
-use walshadow::codecs::{decode_jsonb_with_capacity, jsonb_text_len};
+use walshadow::codecs::{jsonb_text_len, render_jsonb_text};
+
+fn decode_jsonb_with_capacity(
+ body: &[u8],
+ len: usize,
+) -> Result {
+ let mut out = String::with_capacity(len);
+ render_jsonb_text(body, &mut out)?;
+ Ok(out)
+}
#[derive(Debug, Parser)]
struct Args {
diff --git a/benches/spool_read.rs b/benches/spool_read.rs
index 111e802d..d60b91e2 100644
--- a/benches/spool_read.rs
+++ b/benches/spool_read.rs
@@ -1,7 +1,7 @@
//! Deferred spool and plan spool write and replay, without servers.
//! `cargo bench --bench spool_read -- --rows 200000`
//! Add `--plan-only --planner --mapped --relations 8 --control-every 10000`
-//! to include planning, route caching within benchmark, and row placement.
+//! to include planning, route caching within benchmark, and column encoding.
//! Excludes WAL drain, TOAST I/O, control execution, and destination insertion.
use std::path::PathBuf;
@@ -11,18 +11,22 @@ use std::time::{Duration, Instant};
use clap::Parser;
use foldhash::{HashMap, HashMapExt};
use walrus::pg::walparser::RelFileNode;
-use walshadow::backup_page_walk::BackfillTuple;
-use walshadow::ch_emitter::EmitterStats;
+use walrus::pg::walparser::{
+ BlockLocation, RmId, XLogRecord, XLogRecordBlock, XLogRecordBlockHeader, XLogRecordHeader,
+};
+use walshadow::backup_page_walk::{BackfillTuple, TupleData};
+use walshadow::datum::build;
use walshadow::emit::route::RouteSnapshot;
-use walshadow::heap_decoder::{ColumnValue, DecodedHeap, DecodedTuple, DescribedHeap, HeapOp};
-use walshadow::mapping::{TableMapping, TableTarget};
-use walshadow::pipeline::decode::place_rows;
-use walshadow::pipeline::plan_spool::{PlanItem, PlanWriter};
+use walshadow::mapping::{ColumnMapping, TableMapping, TableTarget};
+use walshadow::pipeline::plan_spool::{PlanItem, PlanWriter, ViewChange};
use walshadow::pipeline::planner::{PlanRouteView, Planner};
-use walshadow::schema::{RelDescriptor, RelName, ReplIdent, SchemaEvent};
+use walshadow::schema::{
+ INT4OID, RelAttr, RelDescriptor, RelName, ReplIdent, SchemaEvent, TEXTOID,
+};
+use walshadow::spill::{DescribedInput, HeapInput};
use walshadow::spool::DeferredSpool;
use walshadow::toast::ToastResolver;
-use walshadow::xact_buffer::{DrainEntry, DrainedBatch, OrderedEvent};
+use walshadow::xact_buffer::{DrainEntry, DrainHeap, DrainedBatch, OrderedEvent};
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
@@ -39,7 +43,7 @@ struct Args {
dir: PathBuf,
#[arg(long)]
plan_only: bool,
- /// Run Planner and row placement with cached routes.
+ /// Run Planner with cached routes.
#[arg(long)]
planner: bool,
#[arg(long)]
@@ -92,6 +96,11 @@ async fn deferred(args: &Args, repeat: usize) -> anyhow::Result<()> {
let payload = "x".repeat(args.payload_bytes);
let mut spool = DeferredSpool::new(dir.path().join("deferred.spool"), 1 << 20);
let start = Instant::now();
+ let data = |i: usize| {
+ let cells = [Some(build::int4(i as i32)), Some(build::text(&payload))];
+ walshadow::heap_decoder::form_block_data(&[], &cells).into_boxed_slice()
+ };
+ let data_len = data(0).len();
for i in 0..args.rows {
spool
.push(BackfillTuple {
@@ -103,10 +112,8 @@ async fn deferred(args: &Args, repeat: usize) -> anyhow::Result<()> {
page_lsn: 0,
blkno: i as u32,
offnum: 1,
- columns: vec![
- Some(ColumnValue::Int4(i as i32)),
- Some(ColumnValue::Text(payload.clone())),
- ],
+ data: TupleData::Heap(data(i)),
+ pending: None,
})
.await?;
}
@@ -117,7 +124,7 @@ async fn deferred(args: &Args, repeat: usize) -> anyhow::Result<()> {
let mut n = 0;
while let Some(t) = reader.next().await? {
assert_eq!(t.xid, n as u32);
- assert!(matches!(&t.columns[1], Some(ColumnValue::Text(s)) if s.len() == payload.len()));
+ assert!(matches!(&t.data, TupleData::Heap(b) if b.len() == data_len));
n += 1;
}
reader.finish().await?;
@@ -134,18 +141,16 @@ struct BenchRoutes {
}
impl PlanRouteView for BenchRoutes {
- fn route_for(&mut self, heap: &DescribedHeap) -> Option> {
+ fn route_for(&mut self, rel: &Arc) -> Option> {
self.calls += 1;
- if let Some((desc, route)) = self.memo.get(&heap.descriptor.rel_name)
- && Arc::ptr_eq(desc, &heap.descriptor)
+ if let Some((desc, route)) = self.memo.get(&rel.rel_name)
+ && Arc::ptr_eq(desc, rel)
{
return route.clone();
}
let route = self.route.clone();
- self.memo.insert(
- heap.descriptor.rel_name.clone(),
- (heap.descriptor.clone(), route.clone()),
- );
+ self.memo
+ .insert(rel.rel_name.clone(), (rel.clone(), route.clone()));
route
}
@@ -157,6 +162,59 @@ impl PlanRouteView for BenchRoutes {
}
}
+fn attr(attnum: i16, type_oid: u32, type_len: i16) -> RelAttr {
+ RelAttr {
+ attnum,
+ name: format!("c{attnum}"),
+ type_oid,
+ typmod: -1,
+ not_null: false,
+ dropped: false,
+ type_name: String::new(),
+ type_byval: type_len > 0,
+ type_len,
+ type_align: 'i',
+ type_storage: if type_len > 0 { 'p' } else { 'x' },
+ missing_default: None,
+ }
+}
+
+/// XLOG_HEAP_INSERT block data: xl_heap_header, int4, 4-byte-header text
+fn insert_tuple(id: i32, payload: &[u8]) -> Vec {
+ let mut t = Vec::with_capacity(14 + payload.len());
+ t.extend_from_slice(&2u16.to_le_bytes());
+ t.extend_from_slice(&0u16.to_le_bytes());
+ t.push(24);
+ t.push(0);
+ t.extend_from_slice(&id.to_le_bytes());
+ t.extend_from_slice(&(((4 + payload.len()) as u32) << 2).to_le_bytes());
+ t.extend_from_slice(payload);
+ t
+}
+
+fn insert_record<'a>(xid: u32, rel: &RelDescriptor, tuple: &'a [u8]) -> XLogRecord<'a> {
+ XLogRecord {
+ header: XLogRecordHeader {
+ xact_id: xid,
+ resource_manager_id: RmId::Heap as u8,
+ ..Default::default()
+ },
+ blocks: vec![XLogRecordBlock {
+ header: XLogRecordBlockHeader {
+ location: BlockLocation {
+ rel: rel.rfn,
+ block_no: 0,
+ },
+ ..Default::default()
+ },
+ data: std::borrow::Cow::Borrowed(tuple),
+ ..Default::default()
+ }],
+ main_data: std::borrow::Cow::Borrowed(&[1, 0, 8][..]),
+ ..Default::default()
+ }
+}
+
async fn plan(args: &Args, repeat: usize) -> anyhow::Result<()> {
let dir = tempfile::tempdir_in(&args.dir)?;
let descriptors: Vec<_> = (0..args.relations)
@@ -174,15 +232,21 @@ async fn plan(args: &Args, repeat: usize) -> anyhow::Result<()> {
persistence: 'p',
ancestors: Vec::new(),
replident: ReplIdent::Default { pk_attnums: None },
- attributes: Vec::new(),
+ attributes: vec![attr(1, INT4OID, 4), attr(2, TEXTOID, -1)],
})
})
.collect();
let route = args.mapped.then(|| {
+ let column = |attnum: i16, ty: &str| ColumnMapping {
+ src_attnum: attnum,
+ target_name: format!("c{attnum}"),
+ target_type: ty.into(),
+ type_pinned: false,
+ };
RouteSnapshot::freeze(
Arc::new(TableMapping {
target: TableTarget::new("db", "spool_bench"),
- columns: Vec::new(),
+ columns: vec![column(1, "Int32"), column(2, "String")],
tees: Vec::new(),
}),
Arc::default(),
@@ -190,23 +254,16 @@ async fn plan(args: &Args, repeat: usize) -> anyhow::Result<()> {
)
});
let payload = "x".repeat(args.payload_bytes);
- let make_heap = |i: usize| {
+ let xid_of = |i: usize| 7 + (i % 4) as u32;
+ let make_input = |i: usize| {
let descriptor = &descriptors[i % descriptors.len()];
- DescribedHeap {
- decoded: DecodedHeap {
- rfn: descriptor.rfn,
- xid: 7 + (i % 4) as u32,
- source_lsn: i as u64,
- op: HeapOp::Insert,
- new: Some(DecodedTuple {
- columns: vec![
- Some(ColumnValue::Int4(i as i32)),
- Some(ColumnValue::Text(payload.clone())),
- ],
- partial: false,
- }),
- old: None,
- },
+ let tuple = insert_tuple(i as i32, payload.as_bytes());
+ DescribedInput {
+ input: HeapInput::from_parsed(
+ &insert_record(xid_of(i), descriptor, &tuple),
+ i as u64,
+ 1,
+ ),
descriptor: descriptor.clone(),
descriptor_valid_from: 1,
}
@@ -226,113 +283,102 @@ async fn plan(args: &Args, repeat: usize) -> anyhow::Result<()> {
calls: 0,
};
let resolver = ToastResolver::disabled();
+ // Drained slices built ahead so timing covers planning alone
+ let batches: Vec = if args.planner {
+ (0..args.rows)
+ .step_by(4096)
+ .map(|first| {
+ let end = (first + 4096).min(args.rows);
+ let mut heaps = Vec::with_capacity(end - first);
+ let mut ordered_events = Vec::new();
+ for i in first..end {
+ if let Some(event) = control(i) {
+ ordered_events.push(OrderedEvent {
+ heap_idx: heaps.len(),
+ row_idx: 0,
+ event,
+ });
+ }
+ heaps.push(DrainHeap::Input(Box::new(make_input(i))));
+ }
+ DrainedBatch {
+ heaps,
+ ordered_events,
+ chunks: Vec::new(),
+ new_rows: Default::default(),
+ truncate_rows: Vec::new(),
+ is_final: end == args.rows,
+ }
+ })
+ .collect()
+ } else {
+ Vec::new()
+ };
let start = Instant::now();
let sealed = if args.planner {
let mut planner = Planner::create(
dir.path().join("plan.spool"),
u64::MAX,
+ (4096, 1 << 20),
+ 0,
&mut view,
&resolver,
);
- for first in (0..args.rows).step_by(4096) {
- let end = (first + 4096).min(args.rows);
- let mut heaps = Vec::with_capacity(end - first);
- let mut ordered_events = Vec::new();
- for i in first..end {
- if let Some(event) = control(i) {
- ordered_events.push(OrderedEvent {
- heap_idx: heaps.len(),
- row_idx: 0,
- event,
- });
- }
- heaps.push(make_heap(i));
- }
- planner
- .plan_batch(DrainedBatch {
- heaps,
- ordered_events,
- chunks: Vec::new(),
- new_rows: Default::default(),
- truncate_rows: Vec::new(),
- is_final: end == args.rows,
- })
- .await?;
+ for batch in batches {
+ planner.plan_batch(batch).await?;
}
planner.seal(u64::MAX, 0)?
} else {
- let mut writer = PlanWriter::create(dir.path().join("plan.spool"), u64::MAX, 1 << 20);
+ let mut writer = PlanWriter::create(dir.path().join("plan.spool"), u64::MAX, 1 << 20)
+ .with_segment_limits(4096, 1 << 20);
for i in 0..args.rows {
if let Some(event) = control(i) {
view.apply(&event).await.map_err(anyhow::Error::msg)?;
- writer.push_control(event, 0);
+ writer.push_control(event, 0)?;
+ }
+ let rel = &descriptors[i % descriptors.len()];
+ let table = writer.table(rel, 1, || view.route_for(rel));
+ if !writer.routed(table) {
+ writer.discard(table, 1);
+ } else {
+ let input = make_input(i);
+ let record = input.input.to_xlog_record(rel.rfn);
+ let images = walshadow::heap_decoder::view_heap_record(&record, rel)?;
+ let change = ViewChange {
+ op: images[0].op,
+ xid: input.input.xid,
+ source_lsn: input.input.source_lsn,
+ new: images[0].new.as_ref(),
+ old: None,
+ new_patch: &[],
+ old_patch: &[],
+ };
+ writer.push_views(change, table)?;
}
- let heap = make_heap(i);
- let table = writer.table(&heap, || view.route_for(&heap));
- writer.push_heap(heap.decoded, table)?;
}
writer.seal(u64::MAX, 0)?
};
let write = start.elapsed();
let (size_bytes, spilled) = (sealed.size_bytes, sealed.path().is_some());
- let stats = EmitterStats::default();
- let mut staged = Vec::new();
- let mut pending = Vec::with_capacity(4096);
- let mut routed = 0;
- let mut controls = 0;
+ let routed = sealed.routed_rows;
let start = Instant::now();
let mut rd = sealed.replay()?;
let mut n = 0;
+ let mut controls = 0;
+ let mut n_segments = 0;
while let Some(item) = rd.next_item()? {
- let control = matches!(item, PlanItem::Control(_));
match item {
- PlanItem::Heap(h) => {
- assert_eq!(h.decoded.source_lsn, n as u64);
- assert_eq!(h.decoded.rfn, descriptors[n % descriptors.len()].rfn);
- assert_eq!(h.decoded.xid, 7 + (n % 4) as u32);
- n += 1;
- if args.planner {
- pending.push(h);
- }
- }
- PlanItem::Control(c) => {
- assert_eq!(c.heap_idx, n);
- controls += 1;
+ PlanItem::Segment(s) => {
+ n += s.into_chunks()?.iter().map(|c| c.rows()).sum::();
+ n_segments += 1;
}
+ PlanItem::Control { .. } => controls += 1,
PlanItem::Truncate { .. } => unreachable!(),
}
- if pending.len() == 4096 || (control && !pending.is_empty()) {
- routed += place_rows(
- &mut staged,
- &stats,
- 4096,
- 0,
- 0,
- u64::MAX,
- &rd.tables,
- std::mem::take(&mut pending),
- None,
- );
- staged.clear();
- }
- }
- if args.planner {
- routed += place_rows(
- &mut staged,
- &stats,
- 4096,
- 0,
- 0,
- u64::MAX,
- &rd.tables,
- pending,
- None,
- );
- staged.clear();
- assert_eq!(routed, if args.mapped { args.rows as u64 } else { 0 });
}
let replay = start.elapsed();
assert_eq!(n, if args.mapped { args.rows } else { 0 });
+ assert_eq!(routed, n as u64);
assert_eq!(
controls,
args.rows
@@ -342,7 +388,7 @@ async fn plan(args: &Args, repeat: usize) -> anyhow::Result<()> {
);
report(
if args.planner {
- "plan_replay_place"
+ "plan_replay_encode"
} else {
"plan_replay"
},
@@ -361,10 +407,11 @@ async fn plan(args: &Args, repeat: usize) -> anyhow::Result<()> {
+ (args.rows % interval).min(descriptors.len());
assert_eq!(view.calls, expected_calls);
println!(
- "plan_total repeat={repeat} total_s={:.6} route_calls={} tables={} routed={routed} spilled={}",
+ "plan_total repeat={repeat} total_s={:.6} route_calls={} tables={} routed={routed} segments={} spilled={}",
(write + replay).as_secs_f64(),
view.calls,
rd.tables.len(),
+ n_segments,
spilled
);
Ok(())
diff --git a/benches/xact_spill.rs b/benches/xact_spill.rs
index 1deba7f4..25adad1f 100644
--- a/benches/xact_spill.rs
+++ b/benches/xact_spill.rs
@@ -1,15 +1,26 @@
//! Transaction buffering and verified commit drain, without PostgreSQL or ClickHouse.
//! `cargo bench --bench xact_spill -- --buffer-bytes 1048576`
+//! `--stashed` buffers records as stashed for commit resolution, `--plan`
+//! drains into the planner with a mapped route instead of checking rows
use std::path::PathBuf;
use std::sync::Arc;
use std::time::{Duration, Instant};
use clap::Parser;
-use walrus::pg::walparser::RelFileNode;
-use walshadow::heap_decoder::{ColumnValue, DecodedHeap, DecodedTuple, DescribedHeap, HeapOp};
-use walshadow::schema::{RelDescriptor, RelName, ReplIdent};
-use walshadow::xact_buffer::{StashResolved, XactBuffer, XactBufferConfig};
+use walrus::pg::walparser::{
+ BlockLocation, RelFileNode, RmId, XLogRecord, XLogRecordBlock, XLogRecordBlockHeader,
+ XLogRecordHeader,
+};
+use walshadow::decode::datum::DatumRef;
+use walshadow::emit::route::RouteSnapshot;
+use walshadow::heap_decoder::view_heap_record;
+use walshadow::mapping::{ColumnMapping, TableMapping, TableTarget};
+use walshadow::pipeline::planner::{PlanRouteView, Planner};
+use walshadow::schema::{RelAttr, RelDescriptor, RelName, ReplIdent, TEXTOID};
+use walshadow::spill::{DescribedInput, HeapInput, RawRecord};
+use walshadow::toast::ToastResolver;
+use walshadow::xact_buffer::{DrainEntry, DrainHeap, StashResolved, XactBuffer, XactBufferConfig};
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
@@ -28,6 +39,24 @@ struct Args {
repeats: usize,
#[arg(long, default_value = "target")]
dir: PathBuf,
+ #[arg(long)]
+ stashed: bool,
+ #[arg(long)]
+ plan: bool,
+}
+
+struct Mapped(Arc);
+
+impl PlanRouteView for Mapped {
+ fn route_for(&mut self, _: &Arc) -> Option> {
+ Some(self.0.clone())
+ }
+
+ fn unmapped(&mut self, _: &RelDescriptor, _: u64) {}
+
+ async fn apply(&mut self, _: &DrainEntry) -> Result<(), String> {
+ Ok(())
+ }
}
fn main() -> anyhow::Result<()> {
@@ -57,9 +86,64 @@ async fn run(args: Args) -> anyhow::Result<()> {
persistence: 'p',
ancestors: Vec::new(),
replident: ReplIdent::Default { pk_attnums: None },
- attributes: Vec::new(),
+ attributes: vec![RelAttr {
+ attnum: 1,
+ name: "v".into(),
+ type_oid: TEXTOID,
+ typmod: -1,
+ not_null: false,
+ dropped: false,
+ type_name: "text".into(),
+ type_byval: false,
+ type_len: -1,
+ type_align: 'i',
+ type_storage: 'x',
+ missing_default: None,
+ }],
});
+ let mut routes = Mapped(RouteSnapshot::freeze(
+ Arc::new(TableMapping {
+ target: TableTarget::new("db", "spill_bench"),
+ columns: vec![ColumnMapping {
+ src_attnum: 1,
+ target_name: "v".into(),
+ target_type: "String".into(),
+ type_pinned: false,
+ }],
+ tees: Vec::new(),
+ }),
+ Arc::default(),
+ Default::default(),
+ ));
+ let resolver = ToastResolver::disabled();
let payload = "x".repeat(args.payload_bytes);
+ // XLOG_HEAP_INSERT block data: xl_heap_header, then one 4-byte-header text
+ let mut tuple = 1u16.to_le_bytes().to_vec();
+ tuple.extend_from_slice(&0u16.to_le_bytes());
+ tuple.push(24);
+ tuple.push(0);
+ tuple.extend_from_slice(&(((4 + payload.len()) as u32) << 2).to_le_bytes());
+ tuple.extend_from_slice(payload.as_bytes());
+ let record = |xid: u32| XLogRecord {
+ header: XLogRecordHeader {
+ xact_id: xid,
+ resource_manager_id: RmId::Heap as u8,
+ ..Default::default()
+ },
+ blocks: vec![XLogRecordBlock {
+ header: XLogRecordBlockHeader {
+ location: BlockLocation {
+ rel: descriptor.rfn,
+ block_no: 0,
+ },
+ ..Default::default()
+ },
+ data: std::borrow::Cow::Borrowed(&tuple[..]),
+ ..Default::default()
+ }],
+ main_data: std::borrow::Cow::Borrowed(&[1, 0, 8][..]),
+ ..Default::default()
+ };
println!("{args:?}");
for repeat in 0..args.repeats {
let dir = tempfile::tempdir_in(&args.dir)?;
@@ -72,53 +156,71 @@ async fn run(args: Args) -> anyhow::Result<()> {
let mut peak_memory = 0;
for xid in 1..=args.transactions {
let start = Instant::now();
+ let record = record(xid);
for row in 0..args.rows {
- buffer
- .on_heap(DescribedHeap {
- decoded: DecodedHeap {
- rfn: descriptor.rfn,
- xid,
- source_lsn: u64::from(xid) * (args.rows as u64 + 1) + row as u64,
- op: HeapOp::Insert,
- new: Some(DecodedTuple {
- columns: vec![Some(ColumnValue::Text(payload.clone()))],
- partial: false,
- }),
- old: None,
- },
- descriptor: descriptor.clone(),
- descriptor_valid_from: 1,
- })
- .await?;
+ let source_lsn = u64::from(xid) * (args.rows as u64 + 1) + row as u64;
+ if args.stashed {
+ let raw = RawRecord::from_parsed(&record, source_lsn, 0);
+ buffer.stash_raw(xid, raw).await?;
+ } else {
+ buffer
+ .on_input(DescribedInput {
+ input: HeapInput::from_parsed(&record, source_lsn, 1),
+ descriptor: descriptor.clone(),
+ descriptor_valid_from: 1,
+ })
+ .await?;
+ }
peak_memory = peak_memory.max(buffer.stats().bytes_in_memory);
}
write_time += start.elapsed();
spill_bytes += buffer.stats().spill_bytes_active;
let start = Instant::now();
let commit_lsn = u64::from(xid + 1) * (args.rows as u64 + 1);
+ let resolved = if args.stashed {
+ StashResolved::ordinary(xid, descriptor.clone(), 1)
+ } else {
+ StashResolved::nothing_stashed(xid)
+ };
let mut drain = buffer
- .drain_committed(
- StashResolved::nothing_stashed(xid),
- 0,
- commit_lsn,
- &[],
- false,
- )
+ .drain_committed(resolved, 0, commit_lsn, &[], false)
.await?;
let mut rows = 0;
+ if args.plan {
+ let mut planner = Planner::create(
+ dir.path().join(format!("plan-{xid}.spool")),
+ u64::MAX,
+ (4096, 1 << 20),
+ 0,
+ &mut routes,
+ &resolver,
+ );
+ while let Some(batch) = drain.next_batch(1024, 1 << 20, None).await? {
+ planner.plan_batch(batch).await?;
+ }
+ rows = planner.seal(commit_lsn, 0)?.routed_rows;
+ }
while let Some(batch) = drain.next_batch(1024, 1 << 20, None).await? {
+ if args.stashed {
+ rows += batch.heaps.len() as u64;
+ continue;
+ }
for heap in batch.heaps {
- assert_eq!(heap.decoded.xid, xid);
+ let DrainHeap::Input(input) = heap else {
+ panic!("ordinary rows drain as inputs");
+ };
+ assert_eq!(input.input.xid, xid);
assert_eq!(
- heap.decoded.source_lsn,
+ input.input.source_lsn,
u64::from(xid) * (args.rows as u64 + 1) + rows
);
- assert_eq!(heap.descriptor.as_ref(), descriptor.as_ref());
- assert_eq!(heap.descriptor_valid_from, 1);
- let tuple = heap.decoded.new.unwrap();
- assert_eq!(tuple.columns.len(), 1);
+ assert_eq!(input.descriptor.as_ref(), descriptor.as_ref());
+ assert_eq!(input.descriptor_valid_from, 1);
+ let record = input.input.to_xlog_record(descriptor.rfn);
+ let images = view_heap_record(&record, &descriptor)?;
+ let datums = &images[0].new.as_ref().expect("insert image").datums;
assert!(
- matches!(&tuple.columns[0], Some(ColumnValue::Text(s)) if s == &payload)
+ matches!(datums[..], [DatumRef::Varlena(b)] if b == payload.as_bytes())
);
rows += 1;
}
diff --git a/docs/configuration.md b/docs/configuration.md
index 9bb072db..04ec4e80 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -305,7 +305,8 @@ Both settings apply to initial load and WAL processing in every value mode
### Memory budget
`[memory] resident_payload_max` controls backpressure for payload bytes held
-while decoding and inserting rows. New work waits when budget is full. Default
+while decoding, planning and batching rows; a ClickHouse tail's own budget
+covers them once sealed into an INSERT. New work waits when budget is full. Default
is one half of cgroup memory limit, or host memory without a cgroup limit, with
a 512 MiB minimum. This default assumes a dedicated host or container. Set an
explicit limit in shared environments
@@ -335,10 +336,27 @@ ClickHouse tails bound sealed encoder buffers by startup `[ch] byte_budget`,
including spare vector capacity. Each batch targets that budget divided by
inserter count, keeping concurrent inserts within a byte limit. Partial batches
across tables share another `byte_budget`; a producer waiting to hand off a
-sealed batch can hold one additional batch. A single oversized row may exceed
-these limits. Oracle responses and native client buffers remain additional costs
+sealed batch can hold one additional batch. A batch above `byte_budget` waits
+until no other batch holds budget
Sealed-buffer permits stay held through resolution, retries and insert completion.
+After resolution each Native block is serialized once, compressed with
+`[ch] compression`, and kept in that form until acknowledged; its permit settles
+to memory those bytes occupy, which can exceed their length. Output beyond
+what a batch reserved, such as frame overhead on incompressible data, is held
+without waiting. Each resolver also
+reserves workspace on top of `byte_budget` for one block's compressed output
+and frame scratch, plus 64 MiB for oracle request and response when the tail
+has an oracle; resolvers count oracle bridge concurrency, or one without an
+oracle. Blocks needing more workspace than that reserve seal one at a time
+Inserters send prepared bytes unchanged to every destination.
+An INSERT carries several blocks, sealed at 1,048,576 rows or 16 MiB of column
+bytes and when oracle request size fills, while batch limits still bound each
+INSERT. `walshadow_resident_raw_bytes_total` over
+`walshadow_resident_stored_bytes_total` reports how far sealing shrinks
+resident blocks. `walshadow_output_resident_bytes` and
+`walshadow_batcher_open_bytes` report allocated bytes held by output pools and
+open batches
This pool is separate from decoded-payload admission to avoid circular waits.
Live byte-budget reductions lower flush thresholds; increasing sealed-buffer
capacity requires restarting walshadow
diff --git a/docs/destination-tables.md b/docs/destination-tables.md
index 884006f6..dbb1b5a1 100644
--- a/docs/destination-tables.md
+++ b/docs/destination-tables.md
@@ -198,9 +198,10 @@ By default, a tee uses its main destination's instance and database. Set
use `archive`, and another copy of orders goes to `[ch]`
Named connections accept `host`, `port`, `database`, `user`, `password`, `secure`,
-`tls_server_name`, and `compression`, with the same defaults as `[ch]`.
+and `tls_server_name`, with the same defaults as `[ch]`.
Credentials from `[ch]` are not reused for named connections. All destinations
-share pipeline limits, pool sizes, retry settings, and column layout. Reload
+share pipeline limits, pool sizes, retry settings, `[ch] compression`, and column
+layout. Reload
TOML to update connection names and table mappings. `config_table` has no instance
column. `default` is reserved, and unknown names cause configuration errors
@@ -343,7 +344,12 @@ Common mappings include:
| `inet`, `cidr`, `interval`, unknown types | `String` |
Nullable source columns become `Nullable(...)` unless used as ClickHouse sort
-keys. ClickHouse deployments using PostgreSQL `time` columns must enable
+keys or mapped to arrays or maps. Default array and map mappings turn NULL into
+empty containers
+Arrays, maps, and vectors currently use shadow PostgreSQL conversion, including
+when destination uses Native container types
+
+ClickHouse deployments using PostgreSQL `time` columns must enable
`Time64` support
Use a column rule to override an inferred type; see
diff --git a/docs/limitations.md b/docs/limitations.md
index 95d0cf3e..99030960 100644
--- a/docs/limitations.md
+++ b/docs/limitations.md
@@ -58,8 +58,10 @@ and workload limits before attaching
## Types shadow PostgreSQL converts
Values outside walshadow's own codec set (arrays, `hstore`, enums, ranges,
-domains, extension types) are converted by shadow PostgreSQL, one request per
-insert batch
+domains, extension types) are converted by shadow PostgreSQL in batches, one
+request per Native block needing conversion. Native array or map destination
+types still require this conversion; prepared sending and compression do not
+remove shadow requests
- a value shadow PostgreSQL cannot convert stops the batch and names the
column and row, rather than writing a substituted value
diff --git a/pgext/native.c b/pgext/native.c
index 799b77e0..af47ea7f 100644
--- a/pgext/native.c
+++ b/pgext/native.c
@@ -39,6 +39,9 @@ typedef struct WsNativeCol {
FmgrInfo infn;
Oid ioparam;
bool infn_ready;
+ FmgrInfo recvfn;
+ Oid recv_ioparam;
+ bool recvfn_ready;
FmgrInfo expander;
@@ -166,6 +169,43 @@ ws_input_datum(WsNativeCol* col, const char* body, uint32 len) {
return val;
}
+/* COPY fields carry typsend bytes, typreceive must consume all of them */
+static Datum
+ws_receive_datum(WsNativeCol* col, const char* body, uint32 len) {
+ StringInfoData buf;
+ Datum val;
+
+ if (!col->recvfn_ready) {
+ Oid recvfunc;
+
+ getTypeBinaryInputInfo(col->source_oid, &recvfunc, &col->recv_ioparam);
+ fmgr_info_cxt(recvfunc, &col->recvfn, CurrentMemoryContext);
+ col->recvfn_ready = true;
+ }
+ /* Receive functions expect a writable, NUL-terminated buffer */
+ buf.data = (char*)palloc(len + 1);
+ memcpy(buf.data, body, len);
+ buf.data[len] = '\0';
+ buf.len = (int)len;
+ buf.maxlen = (int)len + 1;
+ buf.cursor = 0;
+ val =
+ ReceiveFunctionCall(&col->recvfn, &buf, col->recv_ioparam, col->source_typmod);
+ if (buf.cursor != buf.len) {
+ ereport(
+ ERROR,
+ (errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
+ errmsg(
+ "incorrect binary data format for type %u: %d of %u bytes consumed",
+ col->source_oid,
+ buf.cursor,
+ len
+ ))
+ );
+ }
+ return val;
+}
+
/* Require one expander owned by source type's extension */
static bool
ws_find_hstore_expander(Oid source_oid, Oid* out) {
@@ -274,7 +314,8 @@ ws_append_cell(pgch_writer* w, size_t col, WsNativeCol* c, StringInfo req) {
ws_append_default(w, col, c);
return;
}
- if (tag != WS_CELL_DISK_RAW && tag != WS_CELL_TEXT && tag != WS_CELL_LITERAL) {
+ if (tag != WS_CELL_DISK_RAW && tag != WS_CELL_TEXT && tag != WS_CELL_LITERAL &&
+ tag != WS_CELL_BINARY) {
ereport(
ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -306,8 +347,17 @@ ws_append_cell(pgch_writer* w, size_t col, WsNativeCol* c, StringInfo req) {
);
return;
}
- val = tag == WS_CELL_DISK_RAW ? ws_reconstruct_datum(c, body, len)
- : ws_input_datum(c, body, len);
+ switch (tag) {
+ case WS_CELL_DISK_RAW:
+ val = ws_reconstruct_datum(c, body, len);
+ break;
+ case WS_CELL_BINARY:
+ val = ws_receive_datum(c, body, len);
+ break;
+ default:
+ val = ws_input_datum(c, body, len);
+ break;
+ }
val = ws_expand(c, val, &valtype);
pgch_append_datum(w, col, val, valtype, false);
}
diff --git a/pgext/walshadow.h b/pgext/walshadow.h
index bae035e6..b9d003c7 100644
--- a/pgext/walshadow.h
+++ b/pgext/walshadow.h
@@ -8,7 +8,7 @@
/* Bumped when a request or response layout changes, or when an op's reading
* of an unchanged layout changes */
-#define WS_PROTO_VERSION 7
+#define WS_PROTO_VERSION 8
/* Bumped when any catalog projection changes shape */
#define WS_PROJECTION_VERSION 2
@@ -38,6 +38,7 @@
#define WS_CELL_DISK_RAW 0x01
#define WS_CELL_TEXT 0x02
#define WS_CELL_LITERAL 0x03
+#define WS_CELL_BINARY 0x04
/* Match daemon MAX_REQUEST_BYTES, exceed maximum inline TOAST value */
#define WS_MAX_REQUEST_BYTES (256 * 1024 * 1024)
diff --git a/plans/performance.md b/plans/performance.md
index 082e35c2..e9948261 100644
--- a/plans/performance.md
+++ b/plans/performance.md
@@ -47,6 +47,27 @@ replay can temporarily duplicate rows
Use existing stage metrics to locate missing attribution before adding counters
Check process uptime and cumulative series remain valid across phase changes
+## Encoding and memory baseline
+
+Measure narrow fixed rows, short strings, wide JSONB/text, TOAST-heavy updates,
+mixed oracle tables, many small transactions, and a spilling large transaction
+against `more-types` (`69f1700`), before direct encoding, using `xact_spill`,
+`spool_read`, and `bootstrap_pump` allocation reports, and source-to-destination
+with [benchmark driver](../bench/README.md). Keep worker counts, batching
+limits, and codec settings unchanged. Use compressible and incompressible
+payloads, healthy and stalled destinations
+
+Record rows/s, source WAL bytes/s, CPU, allocation count, peak/steady RSS, spill
+volume, durable lag, blocks per INSERT, and ClickHouse part creation. Record
+retained bytes by owner from `walshadow_` metrics `xact_bytes_in_memory`,
+`resident_payload_bytes`, `output_resident_bytes`, `batcher_open_bytes`, and
+`resident_raw_bytes_total` over
+`resident_stored_bytes_total`, then compare their
+sum against RSS to size allocator and client overhead. Attribute savings
+separately to compact transaction storage, direct encoding, and compression
+Choose block size, seal workspace reserve, and default compression from measured
+CPU, memory, and part tradeoffs; do not claim gains from wire compression alone
+
## Candidate changes
| Measured limit | Candidate |
@@ -59,6 +80,9 @@ Check process uptime and cumulative series remain valid across phase changes
| Per-record allocation dominates | Evaluate inline storage for small block-reference lists |
| Catalog refresh work dominates | Narrow invalidation only after measuring recapture and cache cost |
| TOAST fetch or materialization dominates | Compare ClickHouse and [shadow modes](../architecture/shadow-toast.md) |
+| Oracle cells dominate output memory | Pack one column's cells into a shared buffer instead of one allocation each |
+| Open transaction inputs dominate RSS | Compress raw transaction segments in memory; keep spill regardless of ratio |
+| Commit-time encoding dominates latency | Encode before commit, after defining unresolved-value and rollback state |
Bootstrap workers change ordering: assign acknowledgement sequence before
parallel decode and account for concurrent deferred-TOAST writers. Preserve
@@ -89,10 +113,11 @@ Try these in order:
insert latency, part count, RSS, and durable completion counts
2. Profile serial WAL dispatch before splitting it further
[Queue worker](../src/source/queueing_record_sink.rs) now dispatches batches
- with heap decode started ahead of dispatch. Measure transaction buffer lock
- waits, commit draining, shadow replay, and downstream waits before changing
- execution. Preserve commit/DDL order, byte-before-record reachability, and
- contiguous durable acknowledgements
+ with TOAST reads started by a prescan ahead of dispatch. Ordinary heaps stay
+ WAL bytes until commit planning. Measure transaction buffer lock waits,
+ commit-time encoding, plan spill, oracle conversion, block sealing, shadow
+ replay, and downstream waits before changing execution. Preserve commit/DDL
+ order, byte-before-record reachability, and contiguous durable acknowledgements
## Bootstrap worker shape
diff --git a/plans/tier2.md b/plans/tier2.md
index b183894e..5153d553 100644
--- a/plans/tier2.md
+++ b/plans/tier2.md
@@ -5,11 +5,11 @@ process and writes ClickHouse wire columns without PostgreSQL. Tier 3 defers to
shadow PostgreSQL through [conversion client](../src/ops/oracle.rs), described in
[value architecture](../architecture/values.md)
-Arrays, `hstore` maps, and pgvector vectors already resolve correctly through
-tier 3. Promotion buys latency only. One tier-3 column makes every batch for that
-table wait on a shadow round trip, so a single `int[]` column costs the round trip
-even when every other column encodes locally. Use `oracle_resolve_seconds_total`
-and `oracle_blocks_total` as baseline and acceptance measure
+Arrays, `hstore` maps, and pgvector vectors already use tier 3. Promotion removes
+shadow conversion work and round trips without changing destination values. One
+tier-3 column makes each Native block needing conversion wait on shadow, even
+when every other column encodes locally. Measure oracle requests and CPU beside
+latency, allocations, and retained bytes
## Why jsonb was cheap and containers are not
@@ -19,14 +19,18 @@ row. `ColumnBuf` offers `Fixed`, `String`, and nullable pairs, and
accepts `JSON` as a string body, so `jsonb` needed a decoder and nothing else
Containers break that assumption. `Array(T)` on wire is a `UInt64` offsets column
-plus a flattened element column carrying its own null map. Rows contribute
-variable element counts to shared element storage, so one row no longer maps to
-one buffer write. `Array` reports `elem_size` zero, so an array target reaching
+plus a flattened element column, with a null map when `T` is nullable. Rows
+contribute variable element counts to shared storage, so one row no longer maps
+to one buffer write. `Array` reports `elem_size` zero, so an array target reaching
`new_for_ast` would silently take the `String` shape and write wrong bytes. Two
-guards prevent that: `composite_target` routes `Kind::Array | Map | Object` to the
-oracle, and `local_matrix_covers` omits array type OIDs
+guards prevent that: `composite_target` routes `Array`, `Map`, `Object`, and
+`QBit` to oracle, and `local_matrix_covers` omits array type OIDs
-Promotion cost is therefore dominated by wire shape and value model, not by codecs
+Commit planning already reads ordinary WAL tuples through borrowed datums and
+writes encoded column chunks. Large plans spill those chunks; conversion workers
+combine local and oracle columns, then seal prepared Native blocks. Extend this
+path with container buffers and a borrowed element walk. Nested owned values
+are not a prerequisite
## Dependencies need no change
@@ -37,21 +41,25 @@ through `chc_build_array`, and builds `Array(Tuple(..))` for maps through
One optional upstream change is ergonomic. `ColumnBuilder::array` and
`ColumnBuilder::nullable` store a raw pointer into the child builder, so every
-wrap level must outlive the level above it. The inserter already holds one vector
-per level, leaves then roots. `Array(Nullable(T))` needs three levels and `Map`
-needs four, so each adds a vector and matching index discipline. An arena owning
-nodes and returning indices would collapse those vectors into one structure
-Decide it on readability, do not expect throughput from it
+wrap level must outlive the level above it. Conversion workers currently hold
+leaves then roots. Containers need deeper builder storage, with tuple pointer
+scratch for maps. Choose stable per-level storage or an owning arena after
+checking lifetimes. Treat this as builder organization, not a throughput claim
## Work in walshadow
| Area | Change |
|---|---|
-| Wire shape | `ColumnBuf` array variant holding row offsets plus element buffer, and a build chain deep enough for `Array(Nullable(T))` |
-| Value model | Nested `ColumnValue` array variant, spill tag and body encoding, `SPILL_VERSION` bump, recursive `approx_bytes` |
-| Decode | Generalize on-disk `ArrayType` walk over element type, including null bitmap |
-| Gates | Admit covered array OIDs in `local_matrix_covers`, narrow `composite_target` to shapes still lacking a local buffer |
-| Type bridge | None, `array_ch_type` already fixes target types |
+| Wire shape | Recursive array buffer with row offsets and child buffer; add tuple children for maps |
+| Buffer operations | Append empty/default rows, account allocated capacity, compare shapes, and merge chunks with offsets rebased by child element count |
+| Plan spill | Serialize and restore recursive column buffers; bump `PLAN_VERSION` when adding buffer tags |
+| Decode | Walk borrowed on-disk `ArrayType` bodies, including null bitmap, into child buffers |
+| Adapters | Reuse container encoding from borrowed datums and owned raw pending values; handle COPY text and fast defaults explicitly |
+| Gates | Admit only supported source/target pairs and input representations; retain oracle routes for other shapes |
+| Type bridge | Preserve existing mappings; define multidimensional behavior against configured target |
+
+Keep raw transaction spill unchanged when existing pending-value forms suffice
+Version transaction or deferred-load formats only if their representation changes
Give the walk its own `CodecError` variants for a rejected array header and an
uncovered element type, so a malformed body is distinguishable from a type the
@@ -59,12 +67,15 @@ table does not admit
### Pair source shape with target shape
-`ColumnEncoding::choose` tests target and source independently. Once array OIDs
-pass `local_matrix_covers`, an operator pin mapping an array column to `String`
-would select local encoding with no buffer able to hold the value. Make the
-choice consider both, so a pin to a scalar target still routes to the oracle
-That keeps existing table configuration as the recovery valve for a divergence,
-with no release needed
+Choose encoding from source type, element codec, and target shape together
+Admitting an array OID must not enable local encoding for a `String` override,
+unsupported nested target, or `QBit`. Keep those pairs on oracle until covered
+
+Column-wide admission also affects resolved TOAST values and COPY rows.
+Resolved values retain raw bodies in existing pending cells; COPY casts
+containers to text. Support that text form or introduce an explicit oracle fallback before
+opening gates. Fast defaults need equivalent handling. An unsupported input
+must not enter a local-only buffer merely because WAL encoding works
### Element metadata
@@ -72,12 +83,17 @@ The walk needs element length, alignment, and by-value flag. `RelAttr` carries
those for the column type, which for an array column describes the array rather
than the element, and carries no `typelem`
-Prefer a static element table keyed by array type name, matching the mapping
-`type_bridge::array_ch_type` already performs. Built-in array element OIDs are
-fixed, and one table can supply length and alignment beside the ClickHouse type
-This leaves both catalog paths untouched. Add `typelem` and element metadata to
-`RelAttr` only when domains or extension arrays need coverage, and expect the
-pinned worker and SQL snapshot paths to agree once it exists
+Start with a static layout table keyed by built-in element OID. Validate header
+element OID against admitted source type; type name alone is insufficient
+
+Separate `domain` branch resolves domain base OIDs and typmods during catalog
+construction and reads `typelem`, but does not carry element layout into
+`RelAttr`. Reuse that resolution when composing changes. Array bodies still
+carry original element OIDs, including domain OIDs after descriptor
+normalization. Retain enough metadata to resolve those identities at record
+boundary, or keep such arrays on oracle. Cover domains over arrays and arrays
+of domains separately. Keep pinned worker, SQL snapshot, and source bootstrap
+metadata consistent
Wrong alignment misreads silently instead of failing. Take alignment per element
from PostgreSQL `src/include/catalog/pg_type.dat` and assert it per type, rather
@@ -94,10 +110,16 @@ reproduce results `tests/oracle_types_e2e.rs` already asserts for tier 3
| NULL column | Empty array |
| Empty array | Empty array |
| Array holding NULL elements | Null elements in `Array(Nullable(T))` |
-| Multi-dimensional array | Oracle, target stays `String` |
+| Multi-dimensional array | Preserve configured target; use oracle for supported nested overrides until local nested encoding exists |
| Non-zero lower bound | Elements in stored order, bounds dropped |
| Element type outside the table | Oracle |
+Default mapping uses array type name without inspecting dimensions, so a
+multidimensional value does not automatically become `String`. Prove behavior
+for default one-layer target, explicit nested target, and `String` override
+Reject unsupported shape before emitting a partial row; never change target
+schema from a value's dimensions
+
`decode_text_array` refuses a null bitmap, correct for its own caller and wrong
for a column codec. Runtime configuration reads its own key columns before the
bridge exists and wants `Vec`, so let it reuse the general walk behind a
@@ -105,28 +127,38 @@ nulls-refusing adapter, as noted in [runtime configuration](runtime_config.md)
## Staging
-1. Decode only: general `ArrayType` walk, nested `ColumnValue`, spill round trip,
- element metadata table. Gates stay closed, so the oracle keeps serving arrays
- and behavior does not move
-2. Wire shape: array `ColumnBuf` and the deeper build chain, proven on
- `Array(Nullable(T))` over both fixed and string elements
-3. Open gates per element family, paired source and target shapes first
-4. `vector` and `halfvec`: pgvector stores a dimension header plus float payload
- rather than `ArrayType`, so it needs its own codec feeding the same array
- buffer with non-nullable elements
-5. `hstore`: `Map(String, Nullable(String))` is
+1. Add recursive array buffers, merge/accounting/spill operations, and builder
+ storage. Prove `Array(Nullable(T))` over fixed and string elements while
+ production gates stay closed
+2. Add borrowed `ArrayType` walk and element metadata. Reuse it for owned raw
+ pending bodies; cover COPY text, fast defaults, compressed and external values
+3. Open gates per source/target pair after local/oracle parity across input
+ paths. Keep unsupported pairs on oracle
+4. Add `vector` and `halfvec` codecs feeding same array buffer with non-nullable
+ elements. Their dimension header and float payload bypass `ArrayType`; this
+ can be first codec after buffer support if minimizing initial scope
+5. Add `hstore`: `Map(String, Nullable(String))` is
`Array(Tuple(String, Nullable(String)))`, needing tuple children, pointer
scratch, and expansion equivalent to the module's `hstore_to_matrix`
-Steps 1 and 2 carry no divergence risk because gates stay shut. Step 3 is where
-divergence can first reach a destination
+Keep existing production behavior while gates stay closed. Open each later
+family only after proving its supported representations and overrides
## Completion
Require identical destination contents through local and oracle paths for every
admitted element type, across insert, update, delete, empty array, NULL column,
-null elements, and TOASTed arrays. Cover spill and restore of nested values, and
-a schema change adding an array column with a fast default
+null elements, non-default lower bounds, and TOASTed arrays. Extend
+[direct/owned parity](../src/emit/direct_parity.rs), then compare destination
+contents against oracle independently across streaming, COPY, and backup loads
+Cover raw transaction spill, recursive plan spill, merging chunks with empty and
+NULL-bearing rows, and a schema change adding an array column with a fast
+default. Test malformed headers separately from unsupported element types
+
+When composing domain support, include original domain element OIDs, nested
+domain bases, typmods, and domains created in same transaction as their table
+Round-trip prepared container blocks with none/LZ4/ZSTD and compression-mismatch
+rebuilds; retain existing INSERT retry and acknowledgement guarantees
Measure oracle round trips removed per table shape before widening element
coverage
diff --git a/plans/value_coercion.md b/plans/value_coercion.md
index a3b13349..1c5068b2 100644
--- a/plans/value_coercion.md
+++ b/plans/value_coercion.md
@@ -112,7 +112,7 @@ containers for unsupported elements
- Verify automatic numeric mappings at precision 76/77/1000, negative scale,
scale greater than precision, and unconstrained numeric against source text
Include trailing fractional zeros, SQL NULL, and String fast default `'NaN'`;
- compare decoded default semantics, not SQL spelling such as `unhex('4e614e')`
+ compare decoded default semantics, not SQL spelling
- Preserve finite calendar boundaries; reject adjacent out-of-range values
- Reject Decimal values exceeding declared precision even when wire width fits
- Accept `1.230` at scale 2; reject `1.234`, including negative equivalents
diff --git a/plans/verification.md b/plans/verification.md
index d09abcf7..274ca414 100644
--- a/plans/verification.md
+++ b/plans/verification.md
@@ -38,6 +38,12 @@ Extend existing TOAST and type suites with:
and MD5s after WAL replay. Test incompressible bodies, cap boundaries, explicit
native JSON mapping, and restart. Do not infer native JSON semantics from
automatic String mapping or row count
+- Compressed external, unchanged, and same-transaction values under shadow and
+ disabled modes, matching ClickHouse-mode cases in
+ [TOAST harness](../tests/toast_e2e.rs)
+- One fixture loaded by COPY, direct bootstrap, base backup, and object store,
+ with pending transactions, deferred TOAST, and restart; compare destination
+ rows across modes as well as against source
For a fixture containing 8 KiB, 32 MiB, and 128 MiB TEXT rows, assert total
length of `167780352` bytes after all inserts succeed. Compare every expected
@@ -89,6 +95,8 @@ retry exhaustion, supervisor restart, and recovery after connectivity returns
Cover a transaction spanning WAL segments, partial batch success, and later
inserts completing before an earlier insert. Compare final rows and durable
progress with uninterrupted execution. Verify missing retained WAL fails clearly
+Add crash points while a spilled plan is written and while it replays, then
+assert restart clears scratch and converges
Prepared COMMIT and ABORT already dispatch through ordinary drain/discard logic,
and open transactions constrain restart floor. Verify full daemon restart after
diff --git a/src/backfill/backfill_bootstrap.rs b/src/backfill/backfill_bootstrap.rs
index 56165bff..5f965aa5 100644
--- a/src/backfill/backfill_bootstrap.rs
+++ b/src/backfill/backfill_bootstrap.rs
@@ -39,7 +39,6 @@ use crate::backfill::backup_sink::{
CatalogFilenodes, DiskLanderSink, DiskLanderStats, MultiplexSink,
};
use crate::backfill::backup_source::{BackupSink, BackupSource, EndInfo, PumpStats, StartInfo};
-use crate::decode::decoder_sink::TupleObserver;
use crate::schema::{RelAttr, RelDescriptor, RelName, ReplIdent};
/// Live counter handles, readable while the pump runs. The pump publishes
@@ -154,10 +153,9 @@ pub fn tap_filenode_set(
/// `rx.recv() == None` signals channel close.
///
/// ```ignore
-/// let (rx, pump) = spawn_greenfield_bootstrap(cfg, source, catalog_map, store_toast);
-/// let drained = tokio::spawn(drain_backfill(rx, observer));
+/// let (mut rx, pump) = spawn_greenfield_bootstrap(cfg, source, catalog_map, store_toast);
+/// while let Some(slab) = rx.recv().await { /* consume */ }
/// let outcome = pump.await?.context("bootstrap pump")?;
-/// let shipped = drained.await?;
/// ```
///
/// Channel is bounded ([`BOOTSTRAP_TUPLE_CHANNEL_CAP`]): PageWalkSink::chunk
@@ -223,8 +221,8 @@ pub fn spawn_greenfield_bootstrap(
}
/// Test/fixture wrapper: run pump to completion, collect every
-/// `BackfillTuple` into a `Vec`. Production callers use
-/// [`spawn_greenfield_bootstrap`] + [`drain_backfill`] to bound memory by
+/// `BackfillTuple` into a `Vec`. Production callers drain
+/// [`spawn_greenfield_bootstrap`] concurrently to bound memory by
/// the emitter's drain rate, not the source's full tuple count
pub async fn run_greenfield_bootstrap(
cfg: BootstrapConfig,
@@ -245,54 +243,6 @@ pub async fn run_greenfield_bootstrap(
Ok((outcome, tuples))
}
-/// Drain backfill tuples into a [`TupleObserver`]. Each `BackfillTuple`
-/// becomes a synthetic
-/// [`CommittedTuple`](crate::decode::heap_decoder::CommittedTuple) with `op = Insert`,
-/// `commit_ts = 0`, `commit_lsn = source_lsn` (BASE_BACKUP `start_lsn`).
-/// Synthesises committed rows from on-disk pages, bypassing the daemon's
-/// xact-buffer + decoder chain.
-///
-/// `PageWalkSink` emits all rows for one rfn contiguously, so drive
-/// `on_xact_end` on every rfn flip: CH Native protocol forbids a new `Query`
-/// while a prior INSERT's data stream is open; without per-table flushes the
-/// second `send_query` races the first INSERT's body bytes and CH silently
-/// drops everything on the connection.
-///
-/// Final `on_xact_end` after channel-close lets the transitional emitter
-/// release CH state before the daemon swaps to the shadow-catalog emitter.
-pub async fn drain_backfill(
- mut rx: mpsc::Receiver>,
- observer: &mut O,
-) -> Result {
- let mut shipped: u64 = 0;
- let mut last_rfn: Option = None;
- let mut last_lsn: u64 = 0;
- while let Some(slab) = rx.recv().await {
- for tuple in slab {
- if let Some(prev) = last_rfn
- && prev != tuple.rfn
- {
- observer.on_xact_end(last_lsn).await.map_err(|e| {
- anyhow::anyhow!("bootstrap drain: emitter rejected mid-table xact end: {e}")
- })?;
- }
- last_rfn = Some(tuple.rfn);
- last_lsn = tuple.source_lsn;
- let committed = tuple.into_committed_insert();
- observer
- .on_tuple(&committed)
- .await
- .context("bootstrap drain: emitter rejected tuple")?;
- shipped += 1;
- }
- }
- observer
- .on_xact_end(last_lsn)
- .await
- .context("bootstrap drain: emitter rejected xact end")?;
- Ok(shipped)
-}
-
/// Build a [`CatalogMap`] by enumerating every source PG user relation
/// upfront. Mirrors `ShadowCatalog::{fetch_by_rfn,fetch_attributes,
/// fetch_replident}` (shadow_catalog.rs) but eager rather than lazy by
@@ -467,7 +417,7 @@ mod tests {
use super::*;
use crate::backfill::backup_page_walk::{PAGE_BYTES, make_rel, synth_single_tuple_page};
use crate::backfill::backup_source::{BackupSink, BackupSource, FileMeta};
- use crate::decode::heap_decoder::HeapOp;
+ use crate::decode::datum::build;
use async_trait::async_trait;
use walrus::pg::backup::tarball::{Entry, Kind, Reader, Writer};
@@ -635,178 +585,7 @@ mod tests {
let tuple = &tuples[0];
assert_eq!(tuple.source_lsn, 0xDEAD_BEEF);
assert_eq!(tuple.xid, 99);
- assert_eq!(tuple.columns.len(), 1);
- assert!(matches!(
- tuple.columns[0],
- Some(crate::decode::heap_decoder::ColumnValue::Int4(42))
- ));
- }
-
- #[tokio::test]
- async fn drain_backfill_synthesises_inserts_into_observer() {
- use crate::decode::decoder_sink::CollectingTupleObserver;
-
- let (tx, rx) = mpsc::channel::>(64);
- let rfn = RelFileNode {
- spc_node: 1663,
- db_node: 5,
- rel_node: 16400,
- };
- for v in 0..3 {
- tx.send(vec![BackfillTuple {
- rfn,
- xid: 100 + v,
- xmax: 0,
- infomask: 0,
- source_lsn: 0xCAFE,
- page_lsn: 0,
- blkno: 0,
- offnum: 0,
- columns: vec![Some(crate::decode::heap_decoder::ColumnValue::Int4(
- v as i32,
- ))],
- }])
- .await
- .unwrap();
- }
- drop(tx);
-
- let mut observer = CollectingTupleObserver::default();
- let shipped = drain_backfill(rx, &mut observer).await.unwrap();
- assert_eq!(shipped, 3);
- assert_eq!(observer.tuples.len(), 3);
- for (i, c) in observer.tuples.iter().enumerate() {
- assert_eq!(c.commit_ts, 0);
- assert_eq!(c.commit_lsn, 0xCAFE);
- assert_eq!(c.decoded.op, HeapOp::Insert);
- assert!(c.decoded.old.is_none());
- let new = c.decoded.new.as_ref().unwrap();
- assert!(!new.partial);
- assert!(matches!(
- new.columns[0],
- Some(crate::decode::heap_decoder::ColumnValue::Int4(v)) if v == i as i32
- ));
- }
- }
-
- #[derive(Default)]
- struct CountingObserver {
- on_tuple_calls: u32,
- on_xact_end_calls: u32,
- reject_xact_end: bool,
- }
-
- impl crate::decode::decoder_sink::TupleObserver for CountingObserver {
- fn on_tuple<'a>(
- &'a mut self,
- _committed: &'a crate::decode::heap_decoder::CommittedTuple,
- ) -> std::pin::Pin<
- Box<
- dyn std::future::Future<
- Output = Result<(), crate::decode::decoder_sink::DecoderSinkError>,
- > + Send
- + 'a,
- >,
- > {
- self.on_tuple_calls += 1;
- Box::pin(async { Ok(()) })
- }
- fn on_xact_end<'a>(
- &'a mut self,
- commit_lsn: u64,
- ) -> std::pin::Pin<
- Box<
- dyn std::future::Future<
- Output = Result,
- > + Send
- + 'a,
- >,
- > {
- self.on_xact_end_calls += 1;
- let reject = self.reject_xact_end;
- Box::pin(async move {
- use crate::decode::decoder_sink::DecoderSinkError;
- (!reject)
- .then_some(commit_lsn)
- .ok_or(DecoderSinkError::Observer("full".into()))
- })
- }
- }
-
- #[tokio::test]
- async fn drain_backfill_calls_on_xact_end_after_channel_closes() {
- let rfn = RelFileNode {
- spc_node: 1663,
- db_node: 5,
- rel_node: 16400,
- };
- let (tx, rx) = mpsc::channel::>(64);
- for v in 0..4u32 {
- tx.send(vec![BackfillTuple {
- rfn,
- xid: v,
- xmax: 0,
- infomask: 0,
- source_lsn: 1,
- page_lsn: 0,
- blkno: 0,
- offnum: 0,
- columns: vec![Some(crate::decode::heap_decoder::ColumnValue::Int4(
- v as i32,
- ))],
- }])
- .await
- .unwrap();
- }
- drop(tx);
-
- let mut obs = CountingObserver::default();
- let shipped = drain_backfill(rx, &mut obs).await.unwrap();
- assert_eq!(shipped, 4);
- assert_eq!(obs.on_tuple_calls, 4);
- // One synthetic xact closes the backfill
- assert_eq!(obs.on_xact_end_calls, 1);
- }
-
- #[tokio::test]
- async fn drain_backfill_surfaces_a_rejected_table_flip() {
- let tuple = |rel_node| BackfillTuple {
- rfn: RelFileNode {
- spc_node: 1663,
- db_node: 5,
- rel_node,
- },
- xid: 1,
- xmax: 0,
- infomask: 0,
- source_lsn: 1,
- page_lsn: 0,
- blkno: 0,
- offnum: 0,
- columns: Vec::new(),
- };
- let (tx, rx) = mpsc::channel::>(1);
- tx.send(vec![tuple(16400), tuple(16401)]).await.unwrap();
- drop(tx);
- let mut obs = CountingObserver {
- reject_xact_end: true,
- ..Default::default()
- };
- let err = drain_backfill(rx, &mut obs).await.unwrap_err();
- assert!(err.to_string().contains("mid-table xact end"), "{err}");
- assert_eq!(obs.on_tuple_calls, 1);
- }
-
- #[tokio::test]
- async fn drain_backfill_calls_on_xact_end_even_on_empty_channel() {
- // Sender dropped without a tuple; `on_xact_end` still fires so the
- // transitional emitter's INSERT cleanup runs unconditionally
- let (tx, rx) = mpsc::channel::>(64);
- drop(tx);
- let mut obs = CountingObserver::default();
- let shipped = drain_backfill(rx, &mut obs).await.unwrap();
- assert_eq!(shipped, 0);
- assert_eq!(obs.on_tuple_calls, 0);
- assert_eq!(obs.on_xact_end_calls, 1);
+ let columns = tuple.cells(&make_rel()).unwrap();
+ assert_eq!(columns, [Some(build::int4(42))]);
}
}
diff --git a/src/backfill/backfill_staging.rs b/src/backfill/backfill_staging.rs
index 01667872..56c0c88e 100644
--- a/src/backfill/backfill_staging.rs
+++ b/src/backfill/backfill_staging.rs
@@ -337,7 +337,7 @@ mod tests {
let sql = "DROP TABLE IF EXISTS staging";
for retries in 0..=2 {
let (config, server) =
- crate::ch::test_support::retry_server(retries, sql, false, false).await;
+ crate::ch::test_support::retry_server(retries, sql, 0, false, None).await;
let mut session = StagingSession::connect(DestEmitter::new(Arc::new(config), None))
.await
.unwrap();
@@ -349,7 +349,7 @@ mod tests {
#[tokio::test]
async fn exchange_ambiguity_does_not_retry() {
let sql = "EXCHANGE TABLES staging AND live";
- let (config, server) = crate::ch::test_support::retry_server(2, sql, false, false).await;
+ let (config, server) = crate::ch::test_support::retry_server(2, sql, 0, false, None).await;
let mut session = StagingSession::connect(DestEmitter::new(Arc::new(config), None))
.await
.unwrap();
diff --git a/src/backfill/backup_backfill.rs b/src/backfill/backup_backfill.rs
index dee7cd53..5c4e3034 100644
--- a/src/backfill/backup_backfill.rs
+++ b/src/backfill/backup_backfill.rs
@@ -1122,6 +1122,7 @@ impl RecordSink for PrescanSink {
#[cfg(test)]
mod tests {
use super::*;
+ use crate::backfill::backup_page_walk::TupleData;
use crate::decode::visibility::{
HEAP_XMAX_INVALID, HEAP_XMAX_IS_MULTI, HEAP_XMIN_COMMITTED, HEAP_XMIN_INVALID,
PgMultiXactAccum, PgXactAccum, PgXactView,
@@ -1613,7 +1614,8 @@ mod tests {
page_lsn: 0,
blkno: 0,
offnum: 0,
- columns: Vec::new(),
+ data: TupleData::Heap(Box::default()),
+ pending: None,
}
}
diff --git a/src/backfill/backup_page_walk.rs b/src/backfill/backup_page_walk.rs
index 1a764f66..a8aa29a6 100644
--- a/src/backfill/backup_page_walk.rs
+++ b/src/backfill/backup_page_walk.rs
@@ -1,5 +1,5 @@
//! Page-walk Tap sink. Decodes user-heap tar entries through the same
-//! heap decoder the WAL hot path uses (`decode_block_data`).
+//! heap decoder the WAL hot path uses (`view_block_data`).
//! See [architecture/bootstrap.md](../../architecture/bootstrap.md).
//!
//! ## V1 limits
@@ -27,12 +27,15 @@ use walrus::pg::walparser::{Oid, RelFileNode};
use crate::backfill::backup_source::{
BackupSink, EntrySink, FileAction, FileKind, FileMeta, StartInfo,
};
+use crate::backfill::copy_backfill::CopyRow;
use crate::backfill::pg_path::{BaseRelFile, RelFork, parse_base_path};
use crate::backfill::walk_barrier::WalkBarrier;
-use crate::decode::heap_decoder::{
- ColumnValue, CommittedTuple, DecodeError, DecodedHeap, DecodedTuple, HeapOp, decode_block_data,
+use crate::decode::heap_decoder::{DecodeError, TupleView, view_block_data};
+use crate::decode::visibility::{
+ PendingXids, PgMultiXactAccum, PgXactAccum, PgXactView, SealedPatch,
};
-use crate::decode::visibility::{PgMultiXactAccum, PgXactAccum, PgXactView, SealedPatch};
+#[cfg(any(test, feature = "test-support"))]
+use crate::decode::{datum::Datum, heap_decoder::decode_block_data};
use crate::schema::RelDescriptor;
use foldhash::{HashMap, HashMapExt, HashSet};
@@ -80,8 +83,8 @@ pub enum PageWalkError {
},
}
-/// One decoded tuple from a backup page; a synthetic INSERT at
-/// `_lsn = start_lsn`.
+/// One tuple from a backup page or COPY; a synthetic INSERT at
+/// `_lsn = start_lsn`. Carries row bytes, decoded only at encode
#[derive(Debug, Clone)]
pub struct BackfillTuple {
pub rfn: RelFileNode,
@@ -100,38 +103,57 @@ pub struct BackfillTuple {
/// ([`crate::toast::ToastRow`])
pub blkno: u32,
pub offnum: u16,
- /// Attnum-1 indexed, matching `RelDescriptor.attributes`
- pub columns: Vec>,
+ pub data: TupleData,
+ /// Xids an undecided row waits on, set once it enters a pending table
+ pub pending: Option,
+}
+
+/// Row bytes of a [`BackfillTuple`]
+#[derive(Debug, Clone, PartialEq)]
+pub enum TupleData {
+ /// On-page tuple from `t_infomask2` on: the `block.data` shape heap
+ /// decode reads
+ Heap(Box<[u8]>),
+ Copy(CopyRow),
+}
+
+impl TupleData {
+ /// Heap image of `values`, see
+ /// [`form_block_data`](crate::decode::heap_decoder::form_block_data)
+ #[cfg(any(test, feature = "test-support"))]
+ pub fn from_cells(values: &[Option]) -> Self {
+ Self::Heap(crate::decode::heap_decoder::form_block_data(&[], values).into())
+ }
+
+ pub fn len(&self) -> usize {
+ match self {
+ Self::Heap(b) => b.len(),
+ Self::Copy(r) => r.bytes().len(),
+ }
+ }
+
+ pub fn is_empty(&self) -> bool {
+ self.len() == 0
+ }
}
impl BackfillTuple {
- pub fn has_mapped_external(&self, mapping: &crate::mapping::TableMapping) -> bool {
- mapping.columns.iter().any(|c| {
- usize::try_from(i32::from(c.src_attnum) - 1)
- .ok()
- .and_then(|i| self.columns.get(i))
- .is_some_and(|v| matches!(v, Some(ColumnValue::ExternalToast(_))))
- })
+ /// Tuple image in place; `None` for COPY rows
+ pub fn view<'a>(&'a self, rel: &RelDescriptor) -> Result>, DecodeError> {
+ match &self.data {
+ TupleData::Heap(b) => view_block_data(b, rel).map(Some),
+ TupleData::Copy(_) => Ok(None),
+ }
}
- /// start_lsn rides as both source_lsn and commit_lsn so
- /// ReplacingMergeTree(_lsn) collapses duplicates the WAL decoder
- /// re-emits for records in [start_lsn, end_lsn]
- pub fn into_committed_insert(self) -> CommittedTuple {
- CommittedTuple {
- decoded: DecodedHeap {
- rfn: self.rfn,
- xid: self.xid,
- source_lsn: self.source_lsn,
- op: HeapOp::Insert,
- new: Some(DecodedTuple {
- columns: self.columns,
- partial: false,
- }),
- old: None,
- },
- commit_ts: 0,
- commit_lsn: self.source_lsn,
+ /// Owned cells, attnum-1 indexed
+ #[cfg(any(test, feature = "test-support"))]
+ pub fn cells(&self, rel: &RelDescriptor) -> Result>, String> {
+ match &self.data {
+ TupleData::Heap(b) => decode_block_data(b, rel)
+ .map(|t| t.columns)
+ .map_err(|e| e.to_string()),
+ TupleData::Copy(row) => Ok(crate::backfill::copy_backfill::copy_cells(row, rel)),
}
}
}
@@ -321,19 +343,20 @@ impl<'a> PageWalker<'a> {
continue;
}
let tuple_bytes = &page[lp_off..lp_off + lp_len];
- match decode_on_page_tuple(tuple_bytes, self.rel) {
- Some((xid, xmax, infomask, columns)) => {
+ match on_page_tuple(tuple_bytes, self.rel) {
+ Some((header, data)) => {
out.push(BackfillTuple {
rfn: self.rel.rfn,
- xid,
- xmax,
- infomask,
+ xid: header.xmin,
+ xmax: header.xmax,
+ infomask: header.infomask,
source_lsn: self.source_lsn,
page_lsn,
blkno: block_no,
// OffsetNumber is 1-based (PG off/itemid.h)
offnum: (i + 1) as u16,
- columns,
+ data: TupleData::Heap(data.into()),
+ pending: None,
});
tally.tuples_emitted += 1;
}
@@ -396,42 +419,60 @@ pub(crate) fn toast_rows_from_page(
) -> Vec {
let mut rows = Vec::new();
for offnum in 1..=page_max_offnum(page) {
- let Some(tuple) = page_tuple_bytes(page, offnum) else {
- continue;
- };
- let Some((_, _, _, mut columns)) = decode_on_page_tuple(tuple, rel) else {
+ let Some((_, data)) = page_tuple_bytes(page, offnum).and_then(|t| on_page_tuple(t, rel))
+ else {
continue;
};
- let Some((chunk_id, chunk_seq, chunk_data)) =
- crate::decode::heap_decoder::take_toast_chunk_columns(&mut columns)
- else {
+ let Some(row) = toast_row(data, rel, blkno, offnum, lsn) else {
continue;
};
- rows.push(crate::toast::ToastRow {
- db_oid: rel.rfn.db_node,
- toast_relid: rel.oid,
- blkno,
- offnum,
- chunk_id,
- chunk_seq,
- chunk_data: bytes::Bytes::from(chunk_data),
- lsn,
- });
+ rows.push(row);
}
rows
}
-/// Decode one on-page tuple into `(xmin, xmax, infomask, columns)`.
+/// Chunk row of one `pg_toast_*` tuple's `block.data` image
+pub(crate) fn toast_row(
+ data: &[u8],
+ rel: &RelDescriptor,
+ blkno: u32,
+ offnum: u16,
+ lsn: u64,
+) -> Option {
+ let view = view_block_data(data, rel).ok()?;
+ let (chunk_id, chunk_seq, chunk_data) =
+ crate::decode::heap_decoder::toast_chunk_datums(&view.datums)?;
+ Some(crate::toast::ToastRow {
+ db_oid: rel.rfn.db_node,
+ toast_relid: rel.oid,
+ blkno,
+ offnum,
+ chunk_id,
+ chunk_seq,
+ chunk_data: bytes::Bytes::copy_from_slice(chunk_data),
+ lsn,
+ })
+}
+
+/// `HeapTupleHeaderData` fields the gate reads
+#[derive(Clone, Copy, Debug)]
+pub(crate) struct OnPageHeader {
+ pub xmin: u32,
+ pub xmax: u32,
+ pub infomask: u16,
+}
+
+/// Split one on-page tuple into its header fields and `block.data` image.
/// On-disk shape carries a full `HeapTupleHeaderData` (23 bytes); from
/// `t_infomask2` on it matches the `xl_heap_header` (5 bytes) shape PG
/// strips into WAL via
/// `XLogRegisterBufData(0, tup->t_data + SizeofHeapTupleHeader, ...)`,
/// so the shared heap decoder reads it in place. `None` on truncated /
-/// malformed header.
-pub(crate) fn decode_on_page_tuple(
- tuple: &[u8],
+/// malformed header or attributes.
+pub(crate) fn on_page_tuple<'a>(
+ tuple: &'a [u8],
rel: &RelDescriptor,
-) -> Option<(u32, u32, u16, Vec>)> {
+) -> Option<(OnPageHeader, &'a [u8])> {
use crate::decode::heap_decoder::SIZE_OF_HEAP_TUPLE_HEADER;
const T_INFOMASK2_OFF: usize = 18;
@@ -443,17 +484,18 @@ pub(crate) fn decode_on_page_tuple(
// t_infomask2 18..20 t_infomask 20..22
// t_hoff 22 (offset to user data, 8-byte aligned)
// t_bits[] 23.. (NULL bitmap, only if HEAP_HASNULL)
- let xmin = u32::from_le_bytes(tuple[0..4].try_into().unwrap());
- let xmax = u32::from_le_bytes(tuple[4..8].try_into().unwrap());
- let t_infomask = u16::from_le_bytes(tuple[20..22].try_into().unwrap());
+ let header = OnPageHeader {
+ xmin: u32::from_le_bytes(tuple[0..4].try_into().unwrap()),
+ xmax: u32::from_le_bytes(tuple[4..8].try_into().unwrap()),
+ infomask: u16::from_le_bytes(tuple[20..22].try_into().unwrap()),
+ };
let t_hoff = tuple[22] as usize;
if t_hoff < SIZE_OF_HEAP_TUPLE_HEADER || t_hoff > tuple.len() {
return None;
}
- match decode_block_data(&tuple[T_INFOMASK2_OFF..], rel) {
- Ok(d) => Some((xmin, xmax, t_infomask, d.columns)),
- Err(_) => None,
- }
+ let data = &tuple[T_INFOMASK2_OFF..];
+ crate::decode::heap_decoder::scan_block_data(data, rel).ok()?;
+ Some((header, data))
}
/// Tap router: resolves a segment's descriptor at `begin` and hands back a
@@ -1145,6 +1187,7 @@ pub(crate) fn synth_single_tuple_page(value: i32) -> [u8; PAGE_BYTES] {
#[cfg(test)]
mod tests {
use super::*;
+ use crate::decode::datum::Datum;
use crate::backfill::backup_source::EndInfo;
use crate::backfill::backup_source::testing::expect_tap;
@@ -1183,6 +1226,14 @@ mod tests {
assert_eq!(sink.source_lsn(), 0xABCD_1234);
}
+ /// Leading int4 of a tuple walked against [`make_rel`]'s layout
+ fn int4(t: &BackfillTuple) -> Option {
+ match t.cells(&make_rel()).unwrap().first() {
+ Some(Some(Datum::Fixed(b))) => Some(i32::from_le_bytes(b.as_slice().try_into().ok()?)),
+ _ => None,
+ }
+ }
+
#[test]
fn page_walker_emits_single_tuple() {
let rel = make_rel();
@@ -1194,8 +1245,7 @@ mod tests {
assert_eq!(out.len(), 1);
assert_eq!(out[0].source_lsn, 0xABCD);
assert_eq!(out[0].xid, 99);
- assert_eq!(out[0].columns.len(), 1);
- assert!(matches!(out[0].columns[0], Some(ColumnValue::Int4(42))));
+ assert_eq!(int4(&out[0]), Some(42));
assert_eq!(tally.pages_walked, 1);
assert_eq!(tally.tuples_emitted, 1);
}
@@ -1425,10 +1475,7 @@ mod tests {
assert_eq!(captured.len(), 1);
assert_eq!(captured[0].source_lsn, 0x1234_5678);
assert_eq!(captured[0].xid, 99);
- assert!(matches!(
- captured[0].columns[0],
- Some(ColumnValue::Int4(99))
- ));
+ assert!(matches!(int4(&captured[0]), Some(99)));
assert_eq!(ld(&sink.stats.files_seen), 1);
assert_eq!(ld(&sink.stats.files_walked), 1);
assert_eq!(ld(&sink.stats.pages_walked), 1);
@@ -1692,7 +1739,7 @@ mod tests {
assert_eq!(ld(&sink.stats.tail_bytes_dropped), 0);
for (i, t) in captured.iter().enumerate() {
assert_eq!(t.blkno, i as u32, "page {i} kept its block number");
- assert!(matches!(t.columns[0], Some(ColumnValue::Int4(v)) if v == i as i32));
+ assert_eq!(int4(t), Some(i as i32));
}
}
@@ -1821,18 +1868,15 @@ mod tests {
// Each tuple carries its own file's rfn + value; a shared slot
// would attribute both to rel B
- let mut by_rel: Vec<_> = sink
- .captured()
- .into_iter()
- .map(|t| (t.rfn.rel_node, t.columns))
- .collect();
- by_rel.sort_by_key(|(rel_node, _)| *rel_node);
- assert_eq!(
- by_rel,
- [
- (16400, vec![Some(ColumnValue::Int4(100))]),
- (16401, vec![Some(ColumnValue::Int4(200))]),
- ]
- );
+ let captured = sink.captured();
+ let mut by_rel: HashMap = HashMap::new();
+ for t in &captured {
+ if let Some(v) = int4(t) {
+ by_rel.insert(t.rfn.rel_node, v);
+ }
+ }
+ assert_eq!(captured.len(), 2);
+ assert_eq!(by_rel.get(&16400), Some(&100), "entry A decoded vs rel A");
+ assert_eq!(by_rel.get(&16401), Some(&200), "entry B decoded vs rel B");
}
}
diff --git a/src/backfill/copy_backfill.rs b/src/backfill/copy_backfill.rs
index a3413e40..6de32154 100644
--- a/src/backfill/copy_backfill.rs
+++ b/src/backfill/copy_backfill.rs
@@ -34,13 +34,14 @@
//! ## Field decode
//!
//! Binary COPY carries each field in `typsend` wire form (big-endian), not
-//! the on-disk datum form the WAL heap decoder reads. Fixed-width types and
-//! byte/text strings decode natively into the same [`ColumnValue`] variants
-//! the WAL path produces; `numeric` and `jsonb` select as `::text`
-//! (`numeric_out` and `jsonb_out` are the forms the WAL path renders too);
-//! every out-of-matrix type also selects
-//! as `::text` and ships as [`ColumnValue::PgPendingText`], so the oracle
-//! converts it through `typinput` exactly as it does a WAL-path default
+//! the on-disk datum form the WAL heap decoder reads. Each row converts to
+//! on-disk form as it arrives: fixed-width fields byte-swap, `name` pads to
+//! `NAMEDATALEN`, `numeric_send` rebuilds the stored `numeric` body, and
+//! byte/text strings pass unchanged. Types only the oracle converts keep
+//! their `typsend` bytes, which the oracle reads through `typreceive`.
+//! `jsonb`, other types the local matrix renders, and types lacking binary
+//! I/O select as `::text` and stay `typoutput` text, which the oracle
+//! converts through `typinput` exactly as it does a WAL-path default
//! (architecture/values.md).
//!
//! ## Resume ledger
@@ -76,13 +77,14 @@ use walrus::pg::replication::conn::PgConfig;
use crate::backfill::backfill_staging::{self, StagingPlan, StagingRel, StagingSession};
use crate::backfill::backfill_types::{BackupRequest, PassContext, PassOutcome};
use crate::backfill::backup_checkpoint::BackupCheckpoint;
-use crate::backfill::backup_page_walk::{BOOTSTRAP_TUPLE_CHANNEL_CAP, BackfillTuple, CatalogMap};
+use crate::backfill::backup_page_walk::{
+ BOOTSTRAP_TUPLE_CHANNEL_CAP, BackfillTuple, CatalogMap, TupleData,
+};
use crate::backfill::opt_in::Backfiller;
use crate::backfill::visibility_pending::{self, SharedPendingLedger};
use crate::catalog::shadow_catalog::ShadowCatalog;
use crate::config::{DestEmitter, ResolvedConfig};
-use crate::decode::codecs::NumericKind;
-use crate::decode::heap_decoder::ColumnValue;
+use crate::decode::datum::DatumRef;
use crate::emit::ch_emitter::{EmitterStats, ToastMode};
use crate::emit::pipeline::tail::OwnedTail;
use crate::emit::pipeline::{Fatal, bootstrap};
@@ -94,7 +96,7 @@ use crate::pos::{Pos, Snapshot};
use crate::runtime_config::InitialLoadMode;
use crate::schema::{
BOOLOID, BPCHAROID, BYTEAOID, CHAROID, CIDOID, DATEOID, FLOAT4OID, FLOAT8OID, INT2OID, INT4OID,
- INT8OID, JSONBOID, JSONOID, MONEYOID, NAMEOID, NUMERICOID, OID8OID, OIDOID, PG_LSNOID,
+ INT8OID, JSONOID, MONEYOID, NAMEOID, NUMERICOID, OID8OID, OIDOID, PG_LSNOID, RelAttr,
RelDescriptor, RelName, TEXTOID, TIMEOID, TIMESTAMPOID, TIMESTAMPTZOID, UUIDOID, VARCHAROID,
XID8OID, XIDOID,
};
@@ -403,9 +405,10 @@ impl Ledger {
// ---------------------------------------------------------------------------
/// How one selected column decodes from the wire. Native kinds read `typsend`
-/// output; `NumericText`/`CastText` columns were cast to `text` in the SELECT.
+/// output; `Cast` columns were cast to `text` in the SELECT; `Send` columns
+/// keep `typsend` bytes for the oracle's `typreceive`
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-enum WireKind {
+pub(crate) enum WireKind {
Bool,
Char,
Int2,
@@ -424,20 +427,20 @@ enum WireKind {
Text,
Name,
Json,
- NumericText,
- CastText,
+ Numeric,
+ Cast,
+ Send,
}
struct ColPlan {
attnum: i16,
- kind: WireKind,
- type_oid: u32,
}
struct CopyPlan {
select: String,
cols: Vec,
- natts: usize,
+ /// Indexed by attnum - 1
+ kinds: Vec,
}
fn wire_kind(type_oid: u32) -> Option {
@@ -460,42 +463,89 @@ fn wire_kind(type_oid: u32) -> Option {
TEXTOID | VARCHAROID | BPCHAROID => WireKind::Text,
NAMEOID => WireKind::Name,
JSONOID => WireKind::Json,
+ NUMERICOID => WireKind::Numeric,
_ => return None,
})
}
-fn column_plan(desc: &RelDescriptor) -> CopyPlan {
+/// Types no local reader converts, so their values always reach the oracle
+fn oracle_only(att: &RelAttr) -> bool {
+ wire_kind(att.type_oid).is_none()
+ && !crate::decode::heap_decoder::local_matrix_covers(att.type_oid, att.type_len)
+}
+
+/// Oracle-only types of `desc` with binary I/O. Shadow replays source's
+/// catalog, so one source `pg_type` read covers `typsend` and the oracle's
+/// `typreceive`
+async fn binary_io_types(
+ client: &tokio_postgres::Client,
+ desc: &RelDescriptor,
+) -> anyhow::Result> {
+ let oids: Vec = desc
+ .attributes
+ .iter()
+ .filter(|a| !a.dropped && oracle_only(a))
+ .map(|a| a.type_oid)
+ .collect();
+ if oids.is_empty() {
+ return Ok(HashSet::new());
+ }
+ let rows = client
+ .query(
+ "SELECT oid FROM pg_type WHERE oid = ANY($1) \
+ AND typsend::oid <> 0 AND typreceive::oid <> 0",
+ &[&oids],
+ )
+ .await
+ .context("backfill: binary I/O types")?;
+ Ok(rows.iter().map(|r| r.get(0)).collect())
+}
+
+/// Wire form each attribute selects as, attnum - 1 indexed. Oracle-only
+/// types in `binary` keep `typsend` bytes, the rest cast to `text`. jsonb's
+/// send form is its text behind a version byte, so cast it too
+pub(crate) fn wire_kinds(desc: &RelDescriptor, binary: &HashSet) -> Vec {
+ let natts = desc
+ .attributes
+ .iter()
+ .map(|a| a.attnum.max(0) as usize)
+ .max()
+ .unwrap_or(0);
+ let mut kinds = vec![WireKind::Cast; natts];
+ for a in desc.attributes.iter().filter(|a| a.attnum >= 1) {
+ kinds[a.attnum as usize - 1] = match wire_kind(a.type_oid) {
+ Some(kind) => kind,
+ None if oracle_only(a) && binary.contains(&a.type_oid) => WireKind::Send,
+ None => WireKind::Cast,
+ };
+ }
+ kinds
+}
+
+fn column_plan(desc: &RelDescriptor, binary: &HashSet) -> CopyPlan {
+ let kinds = wire_kinds(desc, binary);
let mut select = String::new();
let mut cols = Vec::with_capacity(desc.attributes.len());
- let mut natts = 0usize;
for a in &desc.attributes {
- natts = natts.max(a.attnum.max(0) as usize);
if a.dropped {
continue;
}
let ident = quote_ident(&a.name);
- let (expr, kind) = match wire_kind(a.type_oid) {
- Some(k) => (ident, k),
- // jsonb's send form prefixes a version byte; its text is the
- // document the WAL path renders, so take that instead
- None if a.type_oid == JSONBOID => (format!("{ident}::text"), WireKind::Json),
- None if a.type_oid == NUMERICOID => (format!("{ident}::text"), WireKind::NumericText),
- None => (format!("{ident}::text"), WireKind::CastText),
+ let expr = if kinds[(a.attnum - 1).max(0) as usize] == WireKind::Cast {
+ format!("{ident}::text")
+ } else {
+ ident
};
if !select.is_empty() {
select.push_str(", ");
}
select.push_str(&expr);
- cols.push(ColPlan {
- attnum: a.attnum,
- kind,
- type_oid: a.type_oid,
- });
+ cols.push(ColPlan { attnum: a.attnum });
}
CopyPlan {
select,
cols,
- natts,
+ kinds,
}
}
@@ -539,7 +589,7 @@ pub async fn copy_rows_into(
stats: &EmitterStats,
range: BlockRange,
) -> anyhow::Result {
- let plan = column_plan(desc);
+ let plan = column_plan(desc, &binary_io_types(client, desc).await?);
let sql = format!(
"COPY (SELECT {} FROM ONLY {}.{}{}) TO STDOUT (FORMAT binary)",
plan.select,
@@ -556,23 +606,15 @@ pub async fn copy_rows_into(
let mut slab_bytes = 0usize;
while let Some(row) = stream.next().await {
let row = row.context("backfill: COPY stream")?;
- let mut payload_bytes = 0u64;
- let mut columns: Vec> = vec![None; plan.natts];
+ let mut fields: Vec > = vec![None; plan.kinds.len()];
for (i, cp) in plan.cols.iter().enumerate() {
- let raw: Option<&[u8]> = row.try_get(i).context("backfill: COPY field")?;
- payload_bytes += raw.map_or(0, |bytes| bytes.len() as u64);
- let v = raw
- .map(|raw| decode_field(cp.kind, cp.type_oid, raw))
- .transpose()
- .map_err(anyhow::Error::msg)?
- .unwrap_or(ColumnValue::Null);
- columns[(cp.attnum - 1).max(0) as usize] = Some(v);
- }
- slab_bytes += columns
- .iter()
- .flatten()
- .map(ColumnValue::approx_bytes)
- .sum::();
+ fields[(cp.attnum - 1).max(0) as usize] =
+ row.try_get(i).context("backfill: COPY field")?;
+ }
+ let payload_bytes = fields.iter().flatten().map(|f| f.len() as u64).sum::();
+ let data = CopyRow::from_send(&plan.kinds, &fields)
+ .map_err(|e| anyhow::anyhow!("backfill: {e}"))?;
+ slab_bytes += data.bytes().len();
slab.push(BackfillTuple {
rfn: desc.rfn,
xid: 0,
@@ -583,7 +625,8 @@ pub async fn copy_rows_into(
// COPY rows have no on-page TID
blkno: 0,
offnum: 0,
- columns,
+ data: TupleData::Copy(data),
+ pending: None,
});
rows += 1;
stats.backfill_copy_rows.fetch_add(1, Ordering::Relaxed);
@@ -684,54 +727,254 @@ fn fixed(raw: &[u8], what: &str) -> Result<[u8; N], String> {
.map_err(|_| format!("{what}: expected {N} bytes, got {}", raw.len()))
}
-fn text_or_bytea(raw: &[u8], wrap: fn(String) -> ColumnValue) -> ColumnValue {
- simdutf8::basic::from_utf8(raw)
- .map_or_else(|_| ColumnValue::Bytea(raw.to_vec()), |s| wrap(s.into()))
+/// One binary COPY row: per attribute, `typsend` bytes or NULL. Laid out
+/// `[natts: u16][end: u32 per attribute][field bytes]`, an end's high bit
+/// marking NULL and the next marking a field kept in `typsend` form
+#[derive(Clone, Debug, PartialEq)]
+pub struct CopyRow(Box<[u8]>);
+
+const COPY_NULL: u32 = 1 << 31;
+const COPY_SEND: u32 = 1 << 30;
+/// PostgreSQL builds each COPY row in one palloc, so 1 GiB bounds it
+const COPY_END: u32 = COPY_SEND - 1;
+
+impl CopyRow {
+ /// `fields` indexed by attnum - 1
+ pub fn new(fields: &[Option<&[u8]>]) -> Self {
+ let head = 2 + 4 * fields.len();
+ let body = fields.iter().flatten().map(|f| f.len()).sum::();
+ let mut out = Vec::with_capacity(head + body);
+ out.extend_from_slice(&(fields.len() as u16).to_le_bytes());
+ let mut end = 0u32;
+ for f in fields {
+ end += f.map_or(0, |f| f.len() as u32);
+ let mark = if f.is_none() { COPY_NULL } else { 0 };
+ out.extend_from_slice(&(end | mark).to_le_bytes());
+ }
+ for f in fields.iter().flatten() {
+ out.extend_from_slice(f);
+ }
+ Self(out.into())
+ }
+
+ /// Row of `typsend` fields indexed by attnum - 1, each converted to its
+ /// on-disk form per `kinds`
+ pub(crate) fn from_send(kinds: &[WireKind], fields: &[Option<&[u8]>]) -> Result {
+ let head = 2 + 4 * fields.len();
+ let body = fields.iter().flatten().map(|f| f.len()).sum::();
+ let mut out = Vec::with_capacity(head + body);
+ out.extend_from_slice(&(fields.len() as u16).to_le_bytes());
+ out.resize(head, 0);
+ for (i, f) in fields.iter().enumerate() {
+ let kind = kinds.get(i).copied().unwrap_or(WireKind::Cast);
+ if let Some(raw) = f {
+ store_field(kind, raw, &mut out)?;
+ }
+ let end = u32::try_from(out.len() - head)
+ .ok()
+ .filter(|&end| end <= COPY_END)
+ .ok_or("COPY row past 1 GiB")?;
+ let mark = match (f, kind) {
+ (None, _) => COPY_NULL,
+ (Some(_), WireKind::Send) => COPY_SEND,
+ _ => 0,
+ };
+ out[2 + 4 * i..][..4].copy_from_slice(&(end | mark).to_le_bytes());
+ }
+ Ok(Self(out.into()))
+ }
+
+ /// Spooled bytes, see [`Self::from_bytes`]
+ pub fn bytes(&self) -> &[u8] {
+ &self.0
+ }
+
+ /// Row [`Self::bytes`] wrote; `None` when the header overruns them
+ pub fn from_bytes(bytes: Box<[u8]>) -> Option {
+ let row = Self(bytes);
+ let natts = row.natts()?;
+ let last = natts.checked_sub(1).map_or(Some(0), |i| row.end(i))?;
+ (row.0.len() == 2 + 4 * natts + last as usize).then_some(row)
+ }
+
+ fn natts(&self) -> Option {
+ Some(u16::from_le_bytes(self.0.get(..2)?.try_into().ok()?) as usize)
+ }
+
+ fn end(&self, i: usize) -> Option {
+ let at = 2 + 4 * i;
+ let word = u32::from_le_bytes(self.0.get(at..at + 4)?.try_into().ok()?);
+ Some(word & COPY_END)
+ }
+
+ /// Field of attribute index `i`; `None` past the row or NULL
+ pub fn field(&self, i: usize) -> Option<&[u8]> {
+ self.slot(i).map(|(raw, _)| raw)
+ }
+
+ /// Field of attribute index `i` and whether it kept `typsend` form
+ fn slot(&self, i: usize) -> Option<(&[u8], bool)> {
+ let natts = self.natts()?;
+ if i >= natts {
+ return None;
+ }
+ let at = 2 + 4 * i;
+ let word = u32::from_le_bytes(self.0[at..at + 4].try_into().ok()?);
+ if word & COPY_NULL != 0 {
+ return None;
+ }
+ let start = if i == 0 { 0 } else { self.end(i - 1)? };
+ let base = 2 + 4 * natts;
+ let raw = self
+ .0
+ .get(base + start as usize..base + (word & COPY_END) as usize)?;
+ Some((raw, word & COPY_SEND != 0))
+ }
}
-/// Decode non-NULL wire field into heap-decoder value shape
-fn decode_field(kind: WireKind, type_oid: u32, raw: &[u8]) -> Result {
- Ok(match kind {
- WireKind::Bool => ColumnValue::Bool(fixed::<1>(raw, "bool")?[0] != 0),
- WireKind::Char => ColumnValue::Char(fixed::<1>(raw, "char")?[0] as i8),
- WireKind::Int2 => ColumnValue::Int2(i16::from_be_bytes(fixed(raw, "int2")?)),
- WireKind::Int4 => ColumnValue::Int4(i32::from_be_bytes(fixed(raw, "int4")?)),
- WireKind::Int8 => ColumnValue::Int8(i64::from_be_bytes(fixed(raw, "int8")?)),
- WireKind::Oid => ColumnValue::Oid(u32::from_be_bytes(fixed(raw, "oid")?)),
- WireKind::Oid8 => ColumnValue::Oid8(u64::from_be_bytes(fixed(raw, "oid8")?)),
- WireKind::Float4 => ColumnValue::Float4(f32::from_be_bytes(fixed(raw, "float4")?)),
- WireKind::Float8 => ColumnValue::Float8(f64::from_be_bytes(fixed(raw, "float8")?)),
- WireKind::Date => ColumnValue::Date(i32::from_be_bytes(fixed(raw, "date")?)),
- WireKind::Time => ColumnValue::Time(i64::from_be_bytes(fixed(raw, "time")?)),
- WireKind::Timestamp => ColumnValue::Timestamp(i64::from_be_bytes(fixed(raw, "timestamp")?)),
- WireKind::TimestampTz => {
- ColumnValue::TimestampTz(i64::from_be_bytes(fixed(raw, "timestamptz")?))
- }
- WireKind::Uuid => ColumnValue::Uuid(fixed(raw, "uuid")?),
- WireKind::Bytea => ColumnValue::Bytea(raw.to_vec()),
- WireKind::Text => text_or_bytea(raw, ColumnValue::Text),
- // Preserve source type for typinput
- WireKind::CastText => match simdutf8::basic::from_utf8(raw) {
- Ok(s) => ColumnValue::PgPendingText {
- type_oid,
- text: s.to_owned(),
- },
- Err(_) => ColumnValue::Bytea(raw.to_vec()),
- },
- WireKind::Name => text_or_bytea(raw, ColumnValue::Name),
- WireKind::Json => text_or_bytea(raw, ColumnValue::Json),
- // numeric_out text form; specials carry their flag
- WireKind::NumericText => {
- let s = simdutf8::basic::from_utf8(raw)
- .map_err(|_| "numeric::text not utf8".to_string())?;
- ColumnValue::Numeric(match s {
- "NaN" => NumericKind::NaN,
- "Infinity" => NumericKind::PInf,
- "-Infinity" => NumericKind::NInf,
- _ => NumericKind::Finite(s.into()),
- })
+/// Attribute index `i` of a row [`CopyRow::from_send`] stored; NULL past the row
+pub(crate) fn copy_slot<'a>(row: &'a CopyRow, rel: &RelDescriptor, i: usize) -> DatumRef<'a> {
+ let (Some(att), Some((raw, send))) = (rel.attributes.get(i), row.slot(i)) else {
+ return DatumRef::Null;
+ };
+ if send {
+ return DatumRef::Send(raw);
+ }
+ match wire_kind(att.type_oid) {
+ Some(WireKind::Bytea | WireKind::Text | WireKind::Json | WireKind::Numeric) => {
+ DatumRef::Varlena(raw)
}
- })
+ Some(_) => DatumRef::Fixed(raw),
+ None => DatumRef::Text(raw),
+ }
+}
+
+/// Owned datums of a COPY row, attnum-1 indexed
+#[cfg(any(test, feature = "test-support"))]
+pub(crate) fn copy_cells(
+ row: &CopyRow,
+ rel: &RelDescriptor,
+) -> Vec> {
+ (0..rel.attributes.len())
+ .map(|i| crate::decode::heap_decoder::resolve(&rel.attributes[i], copy_slot(row, rel, i)))
+ .collect()
+}
+
+/// Append non-NULL wire field `raw` in its on-disk form
+fn store_field(kind: WireKind, raw: &[u8], out: &mut Vec) -> Result<(), String> {
+ let swapped = |out: &mut Vec, width: usize, name: &str| {
+ if raw.len() != width {
+ return Err(format!("{name}: want {width} bytes, got {}", raw.len()));
+ }
+ out.extend(raw.iter().rev());
+ Ok(())
+ };
+ match kind {
+ WireKind::Bool => swapped(out, 1, "bool"),
+ WireKind::Char => swapped(out, 1, "char"),
+ WireKind::Int2 => swapped(out, 2, "int2"),
+ WireKind::Int4 => swapped(out, 4, "int4"),
+ WireKind::Oid => swapped(out, 4, "oid"),
+ WireKind::Float4 => swapped(out, 4, "float4"),
+ WireKind::Date => swapped(out, 4, "date"),
+ WireKind::Int8 => swapped(out, 8, "int8"),
+ WireKind::Oid8 => swapped(out, 8, "oid8"),
+ WireKind::Float8 => swapped(out, 8, "float8"),
+ WireKind::Time => swapped(out, 8, "time"),
+ WireKind::Timestamp => swapped(out, 8, "timestamp"),
+ WireKind::TimestampTz => swapped(out, 8, "timestamptz"),
+ // Network byte order on disk too
+ WireKind::Uuid if raw.len() == 16 => {
+ out.extend_from_slice(raw);
+ Ok(())
+ }
+ WireKind::Uuid => Err(format!("uuid: want 16 bytes, got {}", raw.len())),
+ WireKind::Name if raw.len() < NAMEDATALEN => {
+ out.extend_from_slice(raw);
+ out.resize(out.len() + NAMEDATALEN - raw.len(), 0);
+ Ok(())
+ }
+ WireKind::Name => Err(format!("name: {} bytes reach NAMEDATALEN", raw.len())),
+ WireKind::Numeric => {
+ out.extend_from_slice(&numeric_from_send(raw)?);
+ Ok(())
+ }
+ WireKind::Bytea | WireKind::Text | WireKind::Json | WireKind::Cast | WireKind::Send => {
+ out.extend_from_slice(raw);
+ Ok(())
+ }
+ }
+}
+
+/// On-disk `attmissingval` from its `anyarray_send` form, `None` for types
+/// outside the wire matrix or an array other than one non-null element
+pub(crate) fn missing_from_send(type_oid: u32, send: &[u8]) -> Option> {
+ let kind = wire_kind(type_oid)?;
+ // array_send: ndim, hasnull, elemtype, (dim, lbound) per dim, (len, bytes) per element
+ let word = |i: usize| {
+ Some(i32::from_be_bytes(
+ send.get(4 * i..4 * i + 4)?.try_into().ok()?,
+ ))
+ };
+ if word(0)? != 1 || word(3)? != 1 {
+ return None;
+ }
+ let len = usize::try_from(word(5)?).ok()?;
+ let raw = send.get(24..)?;
+ if raw.len() != len {
+ return None;
+ }
+ let mut elem = Vec::new();
+ store_field(kind, raw, &mut elem).ok()?;
+ if matches!(
+ kind,
+ WireKind::Bytea | WireKind::Text | WireKind::Json | WireKind::Numeric
+ ) {
+ // 4-byte uncompressed varlena header, VARSIZE in upper 30 bits
+ let size = u32::try_from(elem.len() + 4).ok()?;
+ elem.splice(0..0, (size << 2).to_le_bytes());
+ }
+ Some(crate::decode::heap_decoder::raw_missing_array(
+ type_oid, &elem,
+ ))
+}
+
+/// PostgreSQL `NAMEDATALEN`, `name`'s stored width
+const NAMEDATALEN: usize = 64;
+
+/// `numeric_send` (`ndigits`, `weight`, `sign`, `dscale`, digits, all
+/// big-endian 16-bit) to the stored body
+fn numeric_from_send(raw: &[u8]) -> Result, String> {
+ use crate::decode::codecs::{Special, numeric_body, numeric_special_body};
+ let word = |i: usize| {
+ raw.get(2 * i..2 * i + 2)
+ .map(|b| u16::from_be_bytes([b[0], b[1]]))
+ .ok_or_else(|| format!("numeric: {} bytes truncate word {i}", raw.len()))
+ };
+ let ndigits = usize::from(word(0)?);
+ let weight = i32::from(word(1)? as i16);
+ let sign = word(2)?;
+ let dscale = word(3)?;
+ if raw.len() != 8 + 2 * ndigits {
+ return Err(format!(
+ "numeric: {ndigits} digits need {} bytes, got {}",
+ 8 + 2 * ndigits,
+ raw.len()
+ ));
+ }
+ let special = |s| Ok(numeric_special_body(s));
+ match sign {
+ 0xC000 => special(Special::NaN),
+ 0xD000 => special(Special::PInf),
+ 0xF000 => special(Special::NInf),
+ 0x0000 | 0x4000 => {
+ let digits = (0..ndigits)
+ .map(|i| word(4 + i).map(|d| d as i16))
+ .collect::, _>>()?;
+ Ok(numeric_body(sign == 0x4000, weight, dscale, &digits))
+ }
+ other => Err(format!("numeric: sign word {other:#06x}")),
+ }
}
// ---------------------------------------------------------------------------
@@ -2058,6 +2301,7 @@ struct CopyOutcome {
#[cfg(test)]
mod tests {
use super::*;
+ use crate::schema::JSONBOID;
use crate::schema::{RelAttr, RelName, ReplIdent};
use walrus::pg::walparser::RelFileNode;
@@ -2098,210 +2342,334 @@ mod tests {
}
}
- #[test]
- fn plan_casts_out_of_matrix_and_skips_dropped() {
- let d = desc(vec![
+ fn varlena(attnum: i16, name: &str, type_oid: u32) -> RelAttr {
+ RelAttr {
+ type_len: -1,
+ type_byval: false,
+ type_align: 'i',
+ type_storage: 'x',
+ ..attr(attnum, name, type_oid, false)
+ }
+ }
+
+ /// text[], aclitem, and a user enum: none has a local reader
+ const TEXTARRAYOID: u32 = 1009;
+ const ACLITEMOID: u32 = 1033;
+ const ENUM_OID: u32 = 16_500;
+
+ fn mixed_desc() -> RelDescriptor {
+ desc(vec![
attr(1, "id", INT8OID, false),
attr(2, "gone", TEXTOID, true),
- attr(3, "price", NUMERICOID, false),
- attr(4, "tags", 1009, false), // text[] — out of matrix
- attr(5, "doc", JSONBOID, false),
- ]);
- let plan = column_plan(&d);
+ varlena(3, "price", NUMERICOID),
+ varlena(4, "tags", TEXTARRAYOID),
+ varlena(5, "doc", JSONBOID),
+ RelAttr {
+ type_len: 12,
+ ..attr(6, "acl", ACLITEMOID, false)
+ },
+ RelAttr {
+ type_len: 4,
+ ..attr(7, "mood", ENUM_OID, false)
+ },
+ ])
+ }
+
+ #[test]
+ fn plan_casts_out_of_matrix_and_skips_dropped() {
+ let plan = column_plan(&mixed_desc(), &HashSet::new());
assert_eq!(
plan.select,
- "\"id\", \"price\"::text, \"tags\"::text, \"doc\"::text"
+ "\"id\", \"price\", \"tags\"::text, \"doc\"::text, \"acl\"::text, \"mood\"::text"
);
- assert_eq!(plan.natts, 5);
- assert_eq!(plan.cols.len(), 4, "dropped column not selected");
- assert_eq!(plan.cols[0].kind, WireKind::Int8);
- assert_eq!(plan.cols[1].kind, WireKind::NumericText);
- assert_eq!(plan.cols[2].kind, WireKind::CastText);
+ assert_eq!(plan.kinds.len(), 7);
+ assert_eq!(plan.cols.len(), 6, "dropped column not selected");
assert_eq!(plan.cols[2].attnum, 4);
+ assert_eq!(plan.kinds[0], WireKind::Int8);
+ assert_eq!(plan.kinds[2], WireKind::Numeric);
+ assert_eq!(plan.kinds[3], WireKind::Cast);
// jsonb_out text, the document shape the WAL path yields
- assert_eq!(plan.cols[3].kind, WireKind::Json);
+ assert_eq!(plan.kinds[4], WireKind::Cast);
}
+ /// Oracle-only types with binary I/O select bare; local-matrix types and
+ /// types lacking `typsend`/`typreceive` keep their text cast
#[test]
- fn decode_matches_wal_variants() {
- assert_eq!(
- decode_field(WireKind::Int8, 0, &7i64.to_be_bytes()).unwrap(),
- ColumnValue::Int8(7),
- );
- assert_eq!(
- decode_field(WireKind::Bool, 0, &[1]).unwrap(),
- ColumnValue::Bool(true),
- );
- assert_eq!(
- decode_field(WireKind::Date, 0, &8000i32.to_be_bytes()).unwrap(),
- ColumnValue::Date(8000),
- );
- assert_eq!(
- decode_field(WireKind::Text, 0, b"abc").unwrap(),
- ColumnValue::Text("abc".into()),
- );
- assert_eq!(
- decode_field(WireKind::Name, 0, b"orders").unwrap(),
- ColumnValue::Name("orders".into()),
- );
+ fn plan_keeps_send_form_of_binary_io_types() {
+ // jsonb has binary I/O but a local reader, aclitem lacks it
+ let binary = HashSet::from_iter([TEXTARRAYOID, ENUM_OID, JSONBOID]);
+ let plan = column_plan(&mixed_desc(), &binary);
assert_eq!(
- decode_field(WireKind::Json, 0, b"{\"a\": 1}").unwrap(),
- ColumnValue::Json("{\"a\": 1}".into()),
- );
- assert_eq!(
- decode_field(WireKind::NumericText, 0, b"12.50").unwrap(),
- ColumnValue::Numeric(NumericKind::Finite("12.50".into())),
+ plan.select,
+ "\"id\", \"price\", \"tags\", \"doc\"::text, \"acl\"::text, \"mood\""
);
assert_eq!(
- decode_field(WireKind::NumericText, 0, b"NaN").unwrap(),
- ColumnValue::Numeric(NumericKind::NaN),
+ plan.kinds[3..],
+ [
+ WireKind::Send,
+ WireKind::Cast,
+ WireKind::Cast,
+ WireKind::Send
+ ]
);
+ }
+
+ /// Send fields keep their bytes and mark, NULLs stay NULL, and the row
+ /// reads back the same from its spooled bytes
+ #[test]
+ fn send_fields_round_trip_as_send_datums() {
+ let d = mixed_desc();
+ let plan = column_plan(&d, &HashSet::from_iter([TEXTARRAYOID, ENUM_OID]));
+ let tags = b"\x00\x00\x00\x01array-send";
+ let fields: [Option<&[u8]>; 7] = [
+ Some(&7i64.to_be_bytes()),
+ None,
+ None,
+ Some(tags),
+ Some(br#"{"a": 1}"#),
+ Some(b"=r/postgres"),
+ None,
+ ];
+ let row = CopyRow::from_send(&plan.kinds, &fields).unwrap();
+ let row = CopyRow::from_bytes(row.bytes().into()).expect("spooled bytes parse");
+ let slots: Vec<_> = (0..7).map(|i| copy_slot(&row, &d, i)).collect();
assert_eq!(
- decode_field(WireKind::NumericText, 0, b"Infinity").unwrap(),
- ColumnValue::Numeric(NumericKind::PInf),
+ slots,
+ [
+ DatumRef::Fixed(&7i64.to_le_bytes()),
+ DatumRef::Null,
+ DatumRef::Null,
+ DatumRef::Send(tags),
+ DatumRef::Text(br#"{"a": 1}"#),
+ DatumRef::Text(b"=r/postgres"),
+ DatumRef::Null,
+ ]
);
+ assert_eq!(row.field(3), Some(&tags[..]), "mark stays off the bytes");
+ }
+
+ #[test]
+ fn missing_from_send_matches_stored_default() {
+ use crate::decode::datum::{Datum, build};
+ use crate::decode::heap_decoder::resolve;
+ use crate::schema::MissingDefault;
+ let hex = |s: &str| {
+ (0..s.len())
+ .step_by(2)
+ .map(|i| u8::from_str_radix(&s[i..i + 2], 16).unwrap())
+ .collect::>()
+ };
+ // anyarray_send(attmissingval) from PG 17 after ALTER TABLE ADD ... DEFAULT
+ let cases = [
+ (
+ NUMERICOID,
+ -1,
+ "0000000100000000000006a400000001000000010000000c0002000000000002000c1388",
+ build::numeric("12.50"),
+ ),
+ (
+ TEXTOID,
+ -1,
+ "0000000100000000000000190000000100000001000000026869",
+ Datum::Varlena(b"hi".to_vec()),
+ ),
+ (
+ NAMEOID,
+ 64,
+ "0000000100000000000000130000000100000001000000026e6d",
+ build::name("nm"),
+ ),
+ (
+ INT8OID,
+ 8,
+ "000000010000000000000014000000010000000100000008fffffffffffffffd",
+ Datum::fixed(&(-3i64).to_le_bytes()),
+ ),
+ ];
+ for (type_oid, type_len, send, want) in cases {
+ let mut att = attr(1, "c", type_oid, false);
+ att.type_len = type_len;
+ att.missing_default = missing_from_send(type_oid, &hex(send)).map(MissingDefault::Raw);
+ assert_eq!(
+ resolve(&att, DatumRef::Missing),
+ Some(want),
+ "type {type_oid}"
+ );
+ }
+ const TSVECTOROID: u32 = 3614;
+ let tsvector = hex("000000010000000000000e1e0000000100000001000000080000000178000000");
+ assert_eq!(missing_from_send(TSVECTOROID, &tsvector), None);
+ // Two elements, then a length word disagreeing with the bytes
+ let text = hex("0000000100000000000000190000000100000001000000026869");
+ let mut two = text.clone();
+ two[15] = 2;
+ assert_eq!(missing_from_send(TEXTOID, &two), None);
+ assert_eq!(missing_from_send(TEXTOID, &text[..text.len() - 1]), None);
+ }
+
+ #[test]
+ fn send_fields_store_as_wal_datums() {
+ use crate::decode::datum::{Datum, build};
+ let store = |kind, raw: &[u8]| {
+ let mut out = Vec::new();
+ store_field(kind, raw, &mut out).map(|()| out)
+ };
assert_eq!(
- decode_field(WireKind::NumericText, 0, b"-Infinity").unwrap(),
- ColumnValue::Numeric(NumericKind::NInf),
+ store(WireKind::Int8, &7i64.to_be_bytes()).unwrap(),
+ 7i64.to_le_bytes()
);
+ assert_eq!(store(WireKind::Bool, &[1]).unwrap(), [1]);
assert_eq!(
- decode_field(WireKind::CastText, 1007, b"{1,2}").unwrap(),
- ColumnValue::PgPendingText {
- type_oid: 1007,
- text: "{1,2}".into(),
- },
+ store(WireKind::Date, &8000i32.to_be_bytes()).unwrap(),
+ 8000i32.to_le_bytes()
);
+ assert_eq!(store(WireKind::Text, b"abc").unwrap(), b"abc");
+ let Datum::Fixed(name) = build::name("orders") else {
+ unreachable!()
+ };
+ assert_eq!(store(WireKind::Name, b"orders").unwrap(), name.as_slice());
+ // numeric_send of 12.50: 2 digits, weight 0, positive, dscale 2
+ let send: Vec = [2u16, 0, 0, 2, 12, 5000]
+ .iter()
+ .flat_map(|w| w.to_be_bytes())
+ .collect();
+ let Datum::Varlena(want) = build::numeric("12.50") else {
+ unreachable!()
+ };
+ assert_eq!(store(WireKind::Numeric, &send).unwrap(), want);
+ for (sign, text) in [
+ (0xC000u16, "NaN"),
+ (0xD000, "Infinity"),
+ (0xF000, "-Infinity"),
+ ] {
+ let send: Vec = [0u16, 0, sign, 0]
+ .iter()
+ .flat_map(|w| w.to_be_bytes())
+ .collect();
+ let Datum::Varlena(want) = build::numeric(text) else {
+ unreachable!()
+ };
+ assert_eq!(store(WireKind::Numeric, &send).unwrap(), want, "{text}");
+ }
// Wrong width is an error, not a silent misread
- assert!(decode_field(WireKind::Int4, 0, &[0, 1]).is_err());
- // Invalid UTF-8 degrades to Bytea like the heap decoder
- assert_eq!(
- decode_field(WireKind::Text, 0, &[0xFF, 0xFE]).unwrap(),
- ColumnValue::Bytea(vec![0xFF, 0xFE]),
- );
- assert_eq!(
- decode_field(WireKind::CastText, 1007, &[0xFF]).unwrap(),
- ColumnValue::Bytea(vec![0xFF]),
- );
- assert!(decode_field(WireKind::NumericText, 0, &[0xFF]).is_err());
+ assert!(store(WireKind::Int4, &[0, 1]).is_err());
+ assert!(store(WireKind::Numeric, &[0, 1, 0]).is_err());
+ // Cast columns store text output verbatim, invalid UTF-8 included
+ assert_eq!(store(WireKind::Cast, &[0xFF]).unwrap(), [0xFF]);
+ assert!(store(WireKind::Numeric, &[0, 1, 0, 0, 0, 0, 0, 0]).is_err());
+ assert!(store(WireKind::Numeric, &[0, 0, 0, 0, 0x12, 0x34, 0, 0]).is_err());
+ use WireKind::*;
+ for (kind, width) in [
+ (Char, 1),
+ (Int2, 2),
+ (Oid, 4),
+ (Float4, 4),
+ (Oid8, 8),
+ (Time, 8),
+ ] {
+ let raw: Vec = (1..=width).collect();
+ let want: Vec = raw.iter().rev().copied().collect();
+ assert_eq!(store(kind, &raw).unwrap(), want, "{kind:?}");
+ assert!(store(kind, &raw[1..]).is_err(), "{kind:?}");
+ }
+ let uuid: Vec = (0..16).collect();
+ assert_eq!(store(Uuid, &uuid).unwrap(), uuid);
+ assert!(store(Uuid, &uuid[1..]).is_err());
+ assert!(store(Name, &[b'n'; NAMEDATALEN]).is_err());
}
/// PostgreSQL binary COPY uses each type's `typsend` format
#[test]
- fn every_native_kind_decodes_its_send_bytes() {
+ fn every_native_kind_stores_its_send_bytes() {
+ let store = |kind, raw: &[u8]| {
+ let mut out = Vec::new();
+ store_field(kind, raw, &mut out).map(|()| out)
+ };
let uuid = *b"0123456789abcdef";
+ let name = [b"rel".as_slice(), &[0; NAMEDATALEN - 3]].concat();
for (oid, kind, raw, want) in [
- (BOOLOID, WireKind::Bool, vec![0], ColumnValue::Bool(false)),
- (
- CHAROID,
- WireKind::Char,
- vec![b'x'],
- ColumnValue::Char(b'x' as i8),
- ),
+ (BOOLOID, WireKind::Bool, vec![0], vec![0]),
+ (CHAROID, WireKind::Char, vec![b'x'], vec![b'x']),
(
INT2OID,
WireKind::Int2,
(-2i16).to_be_bytes().to_vec(),
- ColumnValue::Int2(-2),
+ (-2i16).to_le_bytes().to_vec(),
),
(
INT4OID,
WireKind::Int4,
5i32.to_be_bytes().to_vec(),
- ColumnValue::Int4(5),
+ 5i32.to_le_bytes().to_vec(),
),
(
MONEYOID,
WireKind::Int8,
1234i64.to_be_bytes().to_vec(),
- ColumnValue::Int8(1234),
+ 1234i64.to_le_bytes().to_vec(),
),
(
XIDOID,
WireKind::Oid,
77u32.to_be_bytes().to_vec(),
- ColumnValue::Oid(77),
+ 77u32.to_le_bytes().to_vec(),
),
(
PG_LSNOID,
WireKind::Oid8,
(1u64 << 40).to_be_bytes().to_vec(),
- ColumnValue::Oid8(1 << 40),
+ (1u64 << 40).to_le_bytes().to_vec(),
),
(
FLOAT4OID,
WireKind::Float4,
1.5f32.to_be_bytes().to_vec(),
- ColumnValue::Float4(1.5),
+ 1.5f32.to_le_bytes().to_vec(),
),
(
FLOAT8OID,
WireKind::Float8,
(-0.25f64).to_be_bytes().to_vec(),
- ColumnValue::Float8(-0.25),
+ (-0.25f64).to_le_bytes().to_vec(),
),
(
DATEOID,
WireKind::Date,
(-1i32).to_be_bytes().to_vec(),
- ColumnValue::Date(-1),
+ (-1i32).to_le_bytes().to_vec(),
),
(
TIMEOID,
WireKind::Time,
3_600_000_000i64.to_be_bytes().to_vec(),
- ColumnValue::Time(3_600_000_000),
+ 3_600_000_000i64.to_le_bytes().to_vec(),
),
(
TIMESTAMPOID,
WireKind::Timestamp,
9i64.to_be_bytes().to_vec(),
- ColumnValue::Timestamp(9),
+ 9i64.to_le_bytes().to_vec(),
),
(
TIMESTAMPTZOID,
WireKind::TimestampTz,
10i64.to_be_bytes().to_vec(),
- ColumnValue::TimestampTz(10),
- ),
- (
- UUIDOID,
- WireKind::Uuid,
- uuid.to_vec(),
- ColumnValue::Uuid(uuid),
- ),
- (
- BYTEAOID,
- WireKind::Bytea,
- vec![0, 0xFF],
- ColumnValue::Bytea(vec![0, 0xFF]),
- ),
- (
- BPCHAROID,
- WireKind::Text,
- b"ab ".to_vec(),
- ColumnValue::Text("ab ".into()),
- ),
- (
- NAMEOID,
- WireKind::Name,
- b"rel".to_vec(),
- ColumnValue::Name("rel".into()),
- ),
- (
- JSONOID,
- WireKind::Json,
- b"[]".to_vec(),
- ColumnValue::Json("[]".into()),
+ 10i64.to_le_bytes().to_vec(),
),
+ (UUIDOID, WireKind::Uuid, uuid.to_vec(), uuid.to_vec()),
+ (BYTEAOID, WireKind::Bytea, vec![0, 0xFF], vec![0, 0xFF]),
+ (BPCHAROID, WireKind::Text, b"ab ".to_vec(), b"ab ".to_vec()),
+ (NAMEOID, WireKind::Name, b"rel".to_vec(), name),
+ (JSONOID, WireKind::Json, b"[]".to_vec(), b"[]".to_vec()),
] {
assert_eq!(wire_kind(oid), Some(kind), "oid {oid}");
- assert_eq!(decode_field(kind, oid, &raw).unwrap(), want, "oid {oid}");
+ assert_eq!(store(kind, &raw).unwrap(), want, "oid {oid}");
let variable = matches!(
kind,
WireKind::Bytea | WireKind::Text | WireKind::Name | WireKind::Json
);
assert_eq!(
- decode_field(kind, oid, &[0; 17]).is_ok(),
+ store(kind, &[0; 17]).is_ok(),
variable,
"oid {oid}: only fixed-width kinds reject a wrong width"
);
@@ -2312,7 +2680,7 @@ mod tests {
for oid in [OID8OID, XID8OID] {
assert_eq!(wire_kind(oid), Some(WireKind::Oid8));
}
- assert_eq!(wire_kind(NUMERICOID), None);
+ assert_eq!(wire_kind(NUMERICOID), Some(WireKind::Numeric));
}
#[tokio::test]
diff --git a/src/backfill/spool.rs b/src/backfill/spool.rs
index b01adcd3..9b02f872 100644
--- a/src/backfill/spool.rs
+++ b/src/backfill/spool.rs
@@ -21,15 +21,17 @@ use std::path::PathBuf;
use tokio::fs::OpenOptions;
use tokio::io::AsyncSeekExt;
-use crate::backfill::backup_page_walk::BackfillTuple;
+use crate::backfill::backup_page_walk::{BackfillTuple, TupleData};
+use crate::backfill::copy_backfill::CopyRow;
+use crate::decode::visibility::PendingXids;
use crate::xact::spill::{
- ChunkReader, ChunkWriter, Cursor, IO_CHUNK, SpillError, decode_value, encode_value, push_u8,
- push_u16, push_u32, push_u64,
+ ChunkReader, ChunkWriter, Cursor, IO_CHUNK, SpillError, push_u8, push_u16, push_u32, push_u64,
};
use walrus::pg::walparser::RelFileNode;
const SPOOL_MAGIC: [u8; 2] = *b"WD";
-const SPOOL_VERSION: u16 = 2;
+/// v3 carries tuple bytes instead of decoded cells
+const SPOOL_VERSION: u16 = 3;
/// Default in-memory prefix budget before records spill to file
pub const DEFERRED_SPOOL_MEM_MAX: usize = 8 << 20;
@@ -468,15 +470,12 @@ fn truncated() -> SpillError {
}
pub(crate) fn approx_bytes(value: &BackfillTuple) -> usize {
- std::mem::size_of::()
- + value
- .columns
- .iter()
- .flatten()
- .map(crate::decode::heap_decoder::ColumnValue::approx_bytes)
- .sum::()
+ std::mem::size_of::() + value.data.len()
}
+const DATA_HEAP: u8 = 0;
+const DATA_COPY: u8 = 1;
+
fn encode_record(value: &BackfillTuple, out: &mut Vec) {
push_u32(out, value.rfn.spc_node);
push_u32(out, value.rfn.db_node);
@@ -488,16 +487,21 @@ fn encode_record(value: &BackfillTuple, out: &mut Vec) {
push_u64(out, value.page_lsn);
push_u32(out, value.blkno);
push_u16(out, value.offnum);
- push_u32(out, value.columns.len() as u32);
- for col in &value.columns {
- match col {
- None => push_u8(out, 0),
- Some(v) => {
- push_u8(out, 1);
- encode_value(out, v);
- }
+ match value.pending {
+ None => push_u8(out, 0),
+ Some(p) => {
+ push_u8(out, 1);
+ push_u32(out, p.insert);
+ push_u32(out, p.delete);
}
}
+ let (tag, bytes) = match &value.data {
+ TupleData::Heap(b) => (DATA_HEAP, &b[..]),
+ TupleData::Copy(r) => (DATA_COPY, r.bytes()),
+ };
+ push_u8(out, tag);
+ push_u32(out, bytes.len() as u32);
+ out.extend_from_slice(bytes);
}
fn decode_record(buf: &[u8]) -> Result {
@@ -514,14 +518,33 @@ fn decode_record(buf: &[u8]) -> Result {
let page_lsn = c.u64()?;
let blkno = c.u32()?;
let offnum = c.u16()?;
- let ncols = c.u32()? as usize;
- let mut columns = Vec::with_capacity(ncols);
- for _ in 0..ncols {
- columns.push(match c.u8()? {
- 0 => None,
- _ => Some(decode_value(c)?),
- });
- }
+ let pending = match c.u8()? {
+ 0 => None,
+ _ => Some(PendingXids {
+ insert: c.u32()?,
+ delete: c.u32()?,
+ }),
+ };
+ let tag = c.u8()?;
+ let len = c.u32()? as usize;
+ let bytes: Box<[u8]> = c.need(len)?.into();
+ let data = match tag {
+ DATA_HEAP => TupleData::Heap(bytes),
+ DATA_COPY => {
+ TupleData::Copy(
+ CopyRow::from_bytes(bytes).ok_or_else(|| SpillError::Format {
+ offset: 0,
+ detail: "malformed COPY row".into(),
+ })?,
+ )
+ }
+ tag => {
+ return Err(SpillError::Format {
+ offset: 0,
+ detail: format!("unknown tuple data tag {tag}"),
+ });
+ }
+ };
Ok(BackfillTuple {
rfn,
xid,
@@ -531,14 +554,15 @@ fn decode_record(buf: &[u8]) -> Result {
page_lsn,
blkno,
offnum,
- columns,
+ data,
+ pending,
})
}
#[cfg(test)]
mod tests {
use super::*;
- use crate::decode::heap_decoder::ColumnValue;
+ use crate::decode::datum::build;
use tempfile::tempdir;
use tokio::io::AsyncWriteExt;
@@ -635,14 +659,62 @@ mod tests {
page_lsn: lsn + 0x100,
blkno: 3,
offnum: 7,
- columns: vec![
- Some(ColumnValue::Int4(1)),
- None,
- Some(ColumnValue::Bytea(payload.to_vec())),
- ],
+ data: TupleData::from_cells(&[Some(build::int4(1)), None, Some(build::bytes(payload))]),
+ pending: None,
}
}
+ /// COPY rows and pending xids survive the file round trip
+ #[tokio::test]
+ async fn file_records_keep_copy_rows_and_pending_xids() {
+ let dir = tempdir().unwrap();
+ let mut spool = DeferredSpool::new(dir.path().join("copy.bin"), 0);
+ let row = CopyRow::new(&[Some(&7i32.to_be_bytes()[..]), None, Some(b"abc")]);
+ let mut copy = tuple(1, b"");
+ copy.data = TupleData::Copy(row.clone());
+ let mut pending = tuple(2, b"xyz");
+ pending.pending = Some(PendingXids {
+ insert: 11,
+ delete: 12,
+ });
+ spool.push(copy).await.unwrap();
+ spool.push(pending.clone()).await.unwrap();
+ assert_eq!(spool.resident_bytes(), 0);
+ let out = drain_all(spool).await;
+ assert_eq!(out[0].data, TupleData::Copy(row.clone()));
+ assert_eq!(out[0].pending, None);
+ assert_eq!(out[1].data, pending.data);
+ assert_eq!(out[1].pending, pending.pending);
+ assert_eq!(row.field(0), Some(&7i32.to_be_bytes()[..]));
+ assert_eq!(
+ (row.field(1), row.field(2), row.field(3)),
+ (None, Some(&b"abc"[..]), None)
+ );
+ }
+
+ /// `typsend` fields keep their mark across the file, so they still read
+ /// as oracle binary input
+ #[tokio::test]
+ async fn file_records_keep_send_fields() {
+ use crate::backfill::copy_backfill::{WireKind, copy_slot};
+ use crate::decode::datum::DatumRef;
+ let dir = tempdir().unwrap();
+ let mut spool = DeferredSpool::new(dir.path().join("send.bin"), 0);
+ let kinds = [WireKind::Int4, WireKind::Send, WireKind::Send];
+ let fields: [Option<&[u8]>; 3] = [Some(&7i32.to_be_bytes()), Some(b"\x01send"), None];
+ let mut copy = tuple(1, b"");
+ copy.data = TupleData::Copy(CopyRow::from_send(&kinds, &fields).unwrap());
+ spool.push(copy.clone()).await.unwrap();
+ let out = drain_all(spool).await;
+ assert_eq!(out[0].data, copy.data);
+ let TupleData::Copy(row) = &out[0].data else {
+ unreachable!("COPY row")
+ };
+ let rel = crate::schema::int4_rel(16400, "t", 'r', &[(1, "id"), (2, "e"), (3, "f")]);
+ assert_eq!(copy_slot(row, &rel, 1), DatumRef::Send(b"\x01send"));
+ assert_eq!(copy_slot(row, &rel, 2), DatumRef::Null);
+ }
+
#[tokio::test]
async fn checkpoint_then_reopen_keeps_every_record() {
let dir = tempdir().unwrap();
@@ -781,7 +853,7 @@ mod tests {
let out = drain_all(spool).await;
assert_eq!(out.len(), 10);
assert_eq!(out[3].source_lsn, 0x1003);
- assert_eq!(out[3].columns[2], Some(ColumnValue::Bytea(b"abc".to_vec())));
+ assert_eq!(out[3].data, tuple(0, b"abc").data);
}
#[tokio::test(flavor = "current_thread")]
diff --git a/src/backfill/visibility_gate.rs b/src/backfill/visibility_gate.rs
index 88b82eb8..480b3596 100644
--- a/src/backfill/visibility_gate.rs
+++ b/src/backfill/visibility_gate.rs
@@ -540,12 +540,13 @@ pub async fn resolve_greenfield(
mod tests {
use super::*;
use crate::backfill::backup_page_walk::{
- PAGE_BYTES, PageWalkSink, make_rel, make_rel_named, synth_single_tuple_page,
+ PAGE_BYTES, PageWalkSink, TupleData, make_rel, make_rel_named, synth_single_tuple_page,
};
use crate::backfill::backup_source::testing::expect_tap;
use crate::backfill::backup_source::{BackupSink, FileMeta, StartInfo};
use crate::backfill::spool::DEFERRED_SPOOL_MEM_MAX;
use crate::backfill::spool::SpoolMark;
+ use crate::decode::datum::{Datum, build};
use crate::decode::visibility::{
HEAP_XMAX_COMMITTED, HEAP_XMAX_INVALID, HEAP_XMAX_IS_MULTI, HEAP_XMAX_LOCK_ONLY,
HEAP_XMIN_COMMITTED, HEAP_XMIN_INVALID,
@@ -573,7 +574,8 @@ mod tests {
page_lsn: 0,
blkno: 0,
offnum: 0,
- columns: Vec::new(),
+ data: TupleData::Heap(Box::default()),
+ pending: None,
}
}
@@ -819,7 +821,6 @@ mod tests {
/// route to two tails at once, each resuming its own seq space
#[tokio::test]
async fn deferred_lanes_replay_on_their_own_tails() {
- use crate::decode::heap_decoder::ColumnValue;
use crate::mapping::{ColumnMapping, TableMapping, TableTarget};
let tmp = tempfile::tempdir().unwrap();
@@ -852,7 +853,7 @@ mod tests {
};
let row = |id| BackfillTuple {
- columns: vec![Some(ColumnValue::Int4(id))],
+ data: TupleData::from_cells(&[Some(build::int4(id))]),
..tuple(100, 0, HEAP_XMIN_COMMITTED)
};
let mut tails = Vec::new();
@@ -883,8 +884,11 @@ mod tests {
for (want_seq, (ack, ack_task, mut msg_rx)) in [3u64, 7].into_iter().zip(tails) {
let mut seqs = Vec::new();
while let Some(msg) = msg_rx.recv().await {
- let BatcherMsg::Rows(c) = msg else { panic!() };
- seqs.extend(c.rows.iter().map(|r| r.seq));
+ let BatcherMsg::Segments(segments) = msg else {
+ panic!()
+ };
+ let rows = crate::emit::pipeline::batcher::segment_rows(&segments);
+ seqs.extend(rows.iter().map(|r| r.0));
}
assert_eq!(seqs, vec![want_seq; 2], "rows rode their own lane's tail");
drop(ack);
@@ -958,7 +962,7 @@ mod tests {
/// them for chunk-store resolution
#[tokio::test]
async fn mapped_external_values_stream_untouched() {
- use crate::decode::heap_decoder::{ColumnValue, ToastPointer};
+ use crate::decode::heap_decoder::ToastPointer;
use crate::mapping::{ColumnMapping, TableMapping, TableTarget};
use crate::schema::RelName;
@@ -974,7 +978,7 @@ mod tests {
d.attributes.push(body);
let mut catalog = CatalogMap::new();
catalog.insert(desc.clone());
- let external = ColumnValue::ExternalToast(ToastPointer {
+ let external = Datum::External(ToastPointer {
va_rawsize: 8004,
va_extinfo: 8000,
va_valueid: 100,
@@ -1001,13 +1005,20 @@ mod tests {
tees: Vec::new(),
};
let row = |value, infomask| BackfillTuple {
- columns: vec![Some(ColumnValue::Int4(1)), Some(value)],
+ data: TupleData::from_cells(&[Some(build::int4(1)), Some(value)]),
..tuple(100, 0, infomask)
};
+ let has_mapped_external = |t: &BackfillTuple| {
+ let cells = t.cells(&desc).unwrap();
+ mapping
+ .columns
+ .iter()
+ .any(|c| matches!(cells[c.src_attnum as usize - 1], Some(Datum::External(_))))
+ };
let (stats, passed) = run_stream_phase(
&catalog,
vec![
- row(ColumnValue::Text("inline".into()), HEAP_XMIN_COMMITTED),
+ row(build::text("inline"), HEAP_XMIN_COMMITTED),
row(external.clone(), HEAP_XMIN_INVALID),
row(external.clone(), HEAP_XMIN_COMMITTED),
],
@@ -1015,8 +1026,8 @@ mod tests {
.await;
assert_eq!(stats.gated, 1);
assert_eq!(passed.len(), 2);
- assert!(!passed[0].has_mapped_external(&mapping));
- assert_eq!(passed[1].has_mapped_external(&mapping), body_mapped);
+ assert!(!has_mapped_external(&passed[0]));
+ assert_eq!(has_mapped_external(&passed[1]), body_mapped);
}
}
diff --git a/src/backfill/visibility_pending.rs b/src/backfill/visibility_pending.rs
index 1415bd1d..968cbfa5 100644
--- a/src/backfill/visibility_pending.rs
+++ b/src/backfill/visibility_pending.rs
@@ -27,7 +27,7 @@ use crate::backfill::backup_page_walk::{BOOTSTRAP_TUPLE_CHANNEL_CAP, BackfillTup
use crate::backfill::spool::{DEFERRED_SPOOL_MEM_MAX, DeferredSpool};
use crate::ch::quote_ident;
use crate::config::ResolvedConfig;
-use crate::decode::heap_decoder::ColumnValue;
+use crate::decode::datum::Datum;
use crate::decode::visibility::{PendingXids, PgXactView, XidStatus};
use crate::emit::ch_emitter::EmitterStats;
use crate::emit::pipeline::tail::OwnedTail;
@@ -159,17 +159,22 @@ fn pending_table_mapping(
}
}
-fn inject_metadata(tuple: &mut BackfillTuple, rel: &RelDescriptor, pending: PendingXids) {
+/// Values a pending row adds past `rel`'s attributes, see
+/// [`pending_table_mapping`]
+pub(crate) fn metadata_patch(
+ tuple: &BackfillTuple,
+ rel: &RelDescriptor,
+ out: &mut Vec<(usize, Datum)>,
+) {
+ let Some(pending) = tuple.pending else {
+ return;
+ };
let base = metadata_base(rel);
- let infomask = tuple.infomask as i16;
- debug_assert!(
- tuple.columns.len() <= base,
- "walk indexes columns by attnum-1 against this descriptor",
- );
- tuple.columns.resize(base, None);
- tuple.columns.push(Some(ColumnValue::Oid(pending.insert)));
- tuple.columns.push(Some(ColumnValue::Oid(pending.delete)));
- tuple.columns.push(Some(ColumnValue::Int2(infomask)));
+ out.extend([
+ (base, Datum::fixed(&pending.insert.to_le_bytes())),
+ (base + 1, Datum::fixed(&pending.delete.to_le_bytes())),
+ (base + 2, Datum::fixed(&tuple.infomask.to_le_bytes())),
+ ]);
}
#[derive(Debug)]
@@ -223,7 +228,7 @@ impl PendingSpool {
}
tally.start_lsn = tally.start_lsn.min(tuple.source_lsn);
tally.rows += 1;
- inject_metadata(&mut tuple, &rel, pending);
+ tuple.pending = Some(pending);
self.spool
.push(tuple)
.await
@@ -936,7 +941,8 @@ fn xid_list(xids: &[u32]) -> String {
#[cfg(test)]
mod tests {
use super::*;
- use crate::backfill::backup_page_walk::make_rel_named;
+ use crate::backfill::backup_page_walk::{TupleData, make_rel_named};
+ use crate::decode::datum::build;
fn manifest(rel: &str, xids: Vec) -> PendingManifest {
PendingManifest {
@@ -986,7 +992,7 @@ mod tests {
let pending = pending_table_mapping(&m, &desc, &rel);
assert_eq!(pending.target.table, "t__wspending");
- let mut tuple = BackfillTuple {
+ let tuple = BackfillTuple {
rfn: desc.rfn,
xid: 100,
xmax: 200,
@@ -995,26 +1001,40 @@ mod tests {
page_lsn: 0,
blkno: 0,
offnum: 0,
- columns: vec![Some(ColumnValue::Int4(1))],
- };
- inject_metadata(
- &mut tuple,
- &desc,
- PendingXids {
+ data: TupleData::from_cells(&[Some(build::int4(1))]),
+ pending: Some(PendingXids {
insert: 100,
delete: 200,
- },
+ }),
+ };
+ let mut patch = Vec::new();
+ metadata_patch(&tuple, &desc, &mut patch);
+ let route = RouteSnapshot::freeze(Arc::new(pending), Arc::default(), Default::default());
+ let view = tuple.view(&desc).unwrap().unwrap();
+ let stamp = crate::emit::ch_emitter::RowStamp {
+ source_lsn: tuple.source_lsn,
+ xid: tuple.xid,
+ commit_ts: 0,
+ op_code: crate::emit::ch_emitter::OP_INSERT,
+ leaf: None,
+ };
+ let mut chunks = crate::emit::pipeline::plan_spool::ChunkBuilder::new(1, usize::MAX);
+ chunks
+ .append((), &desc, &route, |enc| {
+ enc.append_view(&desc, &view.datums, &patch, &stamp)
+ })
+ .unwrap();
+ let chunks = chunks.take();
+ // UInt32 xids and UInt16 infomask read back as same-width ints
+ assert_eq!(
+ chunks[0].1.cells(),
+ [[
+ build::int4(1),
+ build::int4(100),
+ build::int4(200),
+ build::int2(0x0100),
+ ]],
);
- for c in &pending.columns[1..] {
- let idx = (c.src_attnum - 1) as usize;
- let got = tuple.columns[idx].as_ref().expect("metadata value");
- let want = match c.target_name.as_str() {
- XMIN_COLUMN => ColumnValue::Oid(100),
- XMAX_COLUMN => ColumnValue::Oid(200),
- _ => ColumnValue::Int2(0x0100),
- };
- assert_eq!(got, &want, "{}", c.target_name);
- }
}
#[tokio::test]
diff --git a/src/backfill/wal_replay.rs b/src/backfill/wal_replay.rs
index 8e863adb..07804656 100644
--- a/src/backfill/wal_replay.rs
+++ b/src/backfill/wal_replay.rs
@@ -15,9 +15,10 @@ use walrus::pg::walparser::{Oid, RelFileNode, RmId};
use crate::budget::{MemoryBudget, MemoryPermit, acquire_opt};
use crate::catalog::desc_log::{DescriptorLog, DescriptorLogs};
use crate::config::ResolvedConfig;
+use crate::decode::datum::Datum;
use crate::decode::heap_decoder::{
- CommittedTuple, XLH_DELETE_IS_SUPER, XLOG_HEAP_DELETE, XLOG_HEAP_HOT_UPDATE, XLOG_HEAP_INSERT,
- XLOG_HEAP_OPMASK, XLOG_HEAP_UPDATE, take_key_change_tombstone,
+ HeapOp, XLH_DELETE_IS_SUPER, XLOG_HEAP_DELETE, XLOG_HEAP_HOT_UPDATE, XLOG_HEAP_INSERT,
+ XLOG_HEAP_OPMASK, XLOG_HEAP_UPDATE, view_heap_record,
};
use crate::decode::visibility::PgXactPatch;
use crate::decode::wal_xact::{
@@ -26,7 +27,10 @@ use crate::decode::wal_xact::{
};
use crate::emit::ch_emitter::EmitterStats;
use crate::emit::pipeline::ack::{AckHandle, BufferedRowSender, OpenSeq, Publish, SeqAlloc};
-use crate::emit::pipeline::batcher::{BatcherClosed, BatcherMsg, RowBuffer};
+use crate::emit::pipeline::batcher::{
+ BatcherClosed, BatcherMsg, DECODE_CHUNK_BYTES, PlacedSegment,
+};
+use crate::emit::pipeline::plan_spool::{ChunkBuilder, ViewChange};
use crate::emit::route::{RouteSnapshot, freeze_routes};
use crate::mapping::MappingSnapshot;
use crate::pos::Pos;
@@ -34,10 +38,10 @@ use crate::record::{Record, RecordSink, SegmentSink, SinkError, WAL_SEG_SIZE};
use crate::schema::{FIRST_NORMAL_OBJECT_ID, RelDescriptor};
use crate::source::wal_stream::WalStream;
use crate::toast::{ChunkRefMap, ToastResolver, ToastRow, ToastRowRef};
-use crate::xact::spill::BodySpoolFile;
+use crate::xact::spill::{BodySpoolFile, DescribedInput};
use crate::xact::xact_buffer::{
BufferingDecoderSink, DrainEntry, DrainedBatch, SubxactTracker, WalkStep, XactBuffer,
- detoast_heap, heap_reads_toast, resolve_stash,
+ detoast_sites, external_sites, resolve_stash,
};
use foldhash::{HashMap, HashSet};
@@ -182,8 +186,11 @@ pub struct WalReplaySink {
batch_rows: usize,
batch_bytes: usize,
msg_tx: mpsc::Sender,
- /// Rows not yet on the batcher channel, shipped with held seq events
- row_buffer: RowBuffer,
+ /// Open commit's encoded rows, keyed by descriptor identity
+ chunks: ChunkBuilder,
+ /// Segments not yet on the batcher channel, shipped with held seq events
+ staged: Vec,
+ staged_rows: usize,
patch: Option>>,
seqs: SeqAlloc,
/// Current seq and its routed rows, registered on first row
@@ -212,6 +219,21 @@ fn tail_closed(_: BatcherClosed) -> SinkError {
SinkError::Other("wal_replay: tail closed".into())
}
+fn encode_err(e: crate::ch::EmitterError) -> SinkError {
+ SinkError::Other(format!("wal_replay: encode: {e}"))
+}
+
+/// Rows staged before a channel hop
+const STAGED_ROWS: usize = 65536;
+
+/// Commit a drained batch belongs to, with its TOAST sources
+struct Commit<'a> {
+ ts: i64,
+ lsn: u64,
+ spool: Option<&'a BodySpoolFile>,
+ ref_maps: &'a [&'a ChunkRefMap],
+}
+
impl WalReplaySink {
pub fn new(inputs: WalReplayInputs) -> Self {
let pending_cap = inputs
@@ -257,7 +279,9 @@ impl WalReplaySink {
batch_rows: inputs.batch_rows,
batch_bytes: inputs.batch_bytes,
msg_tx: inputs.msg_tx,
- row_buffer: RowBuffer::default(),
+ chunks: ChunkBuilder::new(STAGED_ROWS, DECODE_CHUNK_BYTES),
+ staged: Vec::new(),
+ staged_rows: 0,
patch: inputs.patch,
seqs: inputs.ack.seqs(inputs.next_seq).buffer_events(),
open: None,
@@ -299,7 +323,7 @@ impl WalReplaySink {
}
/// Transactions needing replay below `lsn` to rebuild buffered prefix
- pub async fn xacts_opened_below(&self, lsn: u64) -> Vec<(u32, u64)> {
+ pub async fn xacts_opened_below(&mut self, lsn: u64) -> Vec<(u32, u64)> {
self.buffer
.lock()
.await
@@ -327,7 +351,7 @@ impl WalReplaySink {
/// Lowest first-record LSN still buffered. A resume above it would drop
/// the prefix of a transaction that commits later
- pub async fn oldest_inflight(&self) -> Option {
+ pub async fn oldest_inflight(&mut self) -> Option {
self.buffer
.lock()
.await
@@ -344,19 +368,48 @@ impl WalReplaySink {
Ok(self.stats())
}
- async fn emit(
+ /// Seq of the open commit, registered on its first routed row; counts
+ /// `rows` against it
+ fn place_rows(&mut self, commit_lsn: u64, rows: usize) -> u64 {
+ let (open, placed) = self
+ .open
+ .get_or_insert_with(|| (self.seqs.open(commit_lsn, Publish::Commit), 0));
+ *placed += rows as u64;
+ open.seq()
+ }
+
+ /// Stage encoded rows as a segment of `seq` once due, or at once when a
+ /// leaf permit rides them: ship it before another acquisition can
+ /// wait on it
+ async fn encoded(
&mut self,
seq: u64,
- table: (&Arc, &Arc),
- committed: CommittedTuple,
- value_permit: Option>,
+ permit: Option,
) -> std::result::Result<(), SinkError> {
- if self.row_buffer.stage(seq, table, committed, value_permit) {
+ if permit.is_none() && !self.chunks.due() {
+ return Ok(());
+ }
+ let ship = permit.is_some();
+ self.stage(seq, permit.map(Arc::new));
+ if ship || self.staged_rows >= STAGED_ROWS {
self.flush_pending().await?;
}
Ok(())
}
+ fn stage(&mut self, seq: u64, permit: Option>) {
+ let chunks: Vec<_> = self.chunks.take().into_iter().map(|(_, c)| c).collect();
+ if chunks.is_empty() {
+ return;
+ }
+ self.staged_rows += chunks.iter().map(|c| c.rows()).sum::();
+ self.staged.push(PlacedSegment {
+ seq,
+ chunks,
+ permit,
+ });
+ }
+
/// Mirror all chunks from restored TOAST page image. Bypass transaction
/// stash because neighboring tuples may belong to other transactions.
///
@@ -538,7 +591,11 @@ impl WalReplaySink {
}
drain.finish().await.map_err(SinkError::from)?;
if let Some((seq, rows)) = self.open.take() {
+ self.stage(seq.seq(), None);
self.seqs.place(seq, rows);
+ if self.staged_rows >= STAGED_ROWS {
+ self.flush_pending().await?;
+ }
}
self.subxact_tracker.forget_tree(xid);
Ok(())
@@ -581,75 +638,108 @@ impl WalReplaySink {
.await
.map_err(SinkError::ctx("toast rewrite barrier"))?;
}
- WalkStep::Heap(mut heap) => {
- let rfn = heap.decoded.rfn;
+ WalkStep::Heap(input) => {
+ let rfn = input.descriptor.rfn;
+ let rows = u64::from(input.input.tuples);
// Decode TOAST chunks, route through parent row
let Some(target) = targets.get(&(rfn.db_node, rfn.rel_node)) else {
continue;
};
if commit_lsn <= self.from_lsn {
// Walked pages cover commits through from_lsn
- self.replay.commits_below_from += 1;
+ self.replay.commits_below_from += rows;
continue;
}
if commit_lsn > target.through {
- self.replay.commits_past_through += 1;
+ self.replay.commits_past_through += rows;
continue;
}
let Some(route) = &target.route else {
self.stats
.unsupported_relations
- .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ .fetch_add(rows, std::sync::atomic::Ordering::Relaxed);
continue;
};
- // Skip deletes for append-only destinations
- if route.drops_deletes()
- && matches!(heap.decoded.op, crate::decode::heap_decoder::HeapOp::Delete)
- {
- self.stats
- .deletes_discarded
- .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
- continue;
- }
- let rel = &target.rel;
- // Make earlier commits available before resolving values.
- // Inline-only heaps read nothing: flushing for them
- // shreds the batch on a row-heavy segment. Gate stays
- // as wide as detoast's leaf acquire: narrowing it to
- // store-bound pointers would hold pending leaf permits
- // across that acquire, which can wait on them
- if heap_reads_toast(&heap) {
- self.flush_rows().await?;
- }
- let value_permit = detoast_heap(&mut heap, spool, &ref_maps, &self.resolver)
+ let at = Commit {
+ ts: commit_ts,
+ lsn: commit_lsn,
+ spool,
+ ref_maps: &ref_maps,
+ };
+ self.replay_views(&input, route, &at).await?;
+ }
+ }
+ }
+ Ok(())
+ }
+
+ /// Whether `op` lands; append-only destinations drop deletes
+ fn lands(&self, route: &RouteSnapshot, op: HeapOp) -> bool {
+ let dropped = route.drops_deletes() && op == HeapOp::Delete;
+ if dropped {
+ self.stats
+ .deletes_discarded
+ .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ }
+ !dropped
+ }
+
+ /// Encode an input's tuple images in place, external values resolved
+ /// into per-image patches
+ async fn replay_views(
+ &mut self,
+ input: &DescribedInput,
+ route: &Arc,
+ at: &Commit<'_>,
+ ) -> std::result::Result<(), SinkError> {
+ let rel = &input.descriptor;
+ let lsn = input.input.source_lsn;
+ let images = view_heap_record(input.input.record_bytes(), rel)
+ .map_err(|e| SinkError::Other(format!("wal_replay: decode at {lsn:#X}: {e}")))?;
+ for h in &images {
+ if h.new.as_ref().is_some_and(|t| t.partial) {
+ return Err(SinkError::Other(format!(
+ "wal_replay: partial update at {lsn:#X}"
+ )));
+ }
+ if !self.lands(route, h.op) {
+ continue;
+ }
+ let sites = external_sites([h.new.as_ref(), h.old.as_ref()]);
+ let mut patches: [Vec<(usize, Datum)>; 2] = Default::default();
+ let mut permit = None;
+ if !sites.is_empty() {
+ // Make earlier commits available before resolving values.
+ // Inline-only heaps read nothing: flushing for them shreds
+ // the batch on a row-heavy segment
+ self.flush_rows().await?;
+ let db = rel.rfn.db_node;
+ let (values, leaf) =
+ detoast_sites(&sites, db, lsn, at.spool, at.ref_maps, &self.resolver)
.await
.map_err(SinkError::from)?;
- let (open, rows) = self
- .open
- .get_or_insert_with(|| (self.seqs.open(commit_lsn, Publish::Commit), 0));
- *rows += 1;
- let seq = open.seq();
- if !route.drops_deletes()
- && let Some(tomb) = take_key_change_tombstone(&mut heap.decoded, rel)
- {
- *rows += 1;
- let tomb = CommittedTuple {
- decoded: tomb,
- commit_ts,
- commit_lsn,
- };
- self.emit(seq, (rel, route), tomb, None).await?;
- }
- let committed = CommittedTuple {
- decoded: heap.decoded,
- commit_ts,
- commit_lsn,
- };
- self.emit(seq, (rel, route), committed, value_permit.map(Arc::new))
- .await?;
- self.replay.rows_replayed += 1;
+ for (site, v) in sites.iter().zip(values) {
+ patches[site.image].push((site.idx, v));
}
+ permit = leaf;
}
+ let change = ViewChange {
+ op: h.op,
+ xid: input.input.xid,
+ source_lsn: lsn,
+ new: h.new.as_ref(),
+ old: h.old.as_ref(),
+ new_patch: &patches[0],
+ old_patch: &patches[1],
+ };
+ let key = Arc::as_ptr(rel) as usize;
+ let rows = self
+ .chunks
+ .push_views(key, rel, route, change, at.ts)
+ .map_err(encode_err)?;
+ let seq = self.place_rows(at.lsn, rows);
+ self.replay.rows_replayed += 1;
+ self.encoded(seq, permit).await?;
}
Ok(())
}
@@ -727,10 +817,15 @@ impl BufferedRowSender for WalReplaySink {
}
async fn send_rows(&mut self) -> std::result::Result<(), SinkError> {
- self.row_buffer
- .flush(&self.msg_tx)
+ if self.staged.is_empty() {
+ return Ok(());
+ }
+ self.staged_rows = 0;
+ let staged = std::mem::take(&mut self.staged);
+ self.msg_tx
+ .send(BatcherMsg::Segments(staged))
.await
- .map_err(tail_closed)
+ .map_err(|_| tail_closed(BatcherClosed))
}
}
@@ -744,6 +839,7 @@ impl RecordSink for WalReplaySink {
return Ok(());
};
// Failed commit's seq stays unplaced, pinning ack below it
+ let _ = self.chunks.take();
self.flush_pending().await?;
Err(e)
})
@@ -842,6 +938,7 @@ pub async fn pump_segments_through(
mod tests {
use super::*;
use crate::catalog::desc_log::DescLogIdentity;
+ use crate::decode::datum::build;
use crate::decode::heap_decoder::SIZE_OF_HEAP_UPDATE;
use crate::decode::visibility::{PgXactAccum, PgXactView, SealedPatch, XidStatus};
use crate::decode::wal_xact::{
@@ -1535,9 +1632,7 @@ mod tests {
#[tokio::test]
async fn error_between_commits_ships_committed_rows() {
- use crate::decode::heap_decoder::{
- ColumnValue, DecodedHeap, DecodedTuple, DescribedHeap, HeapOp,
- };
+ use crate::decode::heap_decoder::{DecodedHeap, DecodedTuple, DescribedHeap, HeapOp};
use crate::mapping::{ColumnMapping, TableMapping, TableTarget};
use crate::schema::{RelName, ReplIdent};
use walrus::pg::walparser::RelFileNode;
@@ -1587,7 +1682,7 @@ mod tests {
source_lsn,
op: HeapOp::Insert,
new: Some(DecodedTuple {
- columns: vec![Some(ColumnValue::Int4(1))],
+ columns: vec![Some(build::int4(1))],
partial: false,
}),
old: None,
@@ -1605,7 +1700,10 @@ mod tests {
abort.parsed.main_data = std::borrow::Cow::Owned(vec![0u8; 10]);
assert!(sink.on_record(&abort).await.is_err());
let shipped = msg_rx.try_recv();
- assert!(matches!(&shipped, Ok(BatcherMsg::Rows(chunk)) if chunk.rows.len() == 1));
+ assert!(matches!(
+ &shipped,
+ Ok(BatcherMsg::Segments(s)) if crate::emit::pipeline::batcher::segment_rows(s).len() == 1
+ ));
// Rows staged past a closed tail fail the replay
buffer.lock().await.on_heap(insert(903, 30)).await.unwrap();
diff --git a/src/bin/stream/bootstrap.rs b/src/bin/stream/bootstrap.rs
index 14f31ca7..83bbd9bf 100644
--- a/src/bin/stream/bootstrap.rs
+++ b/src/bin/stream/bootstrap.rs
@@ -18,7 +18,7 @@ use walshadow::backfill::visibility_gate::{
DeferredLane, GateStats, GreenfieldSink, PendingGate, resolve_greenfield, stream_phase,
};
use walshadow::backfill_bootstrap::{
- BootstrapConfig, BootstrapOutcome, BootstrapProgress, drain_backfill, seed_in_snapshot,
+ BootstrapConfig, BootstrapOutcome, BootstrapProgress, seed_in_snapshot,
spawn_greenfield_bootstrap,
};
use walshadow::backup_source::BackupSource;
@@ -27,7 +27,6 @@ use walshadow::backup_source_object_store::ObjectStoreSource;
use walshadow::bootstrap_marker::{self, BootstrapMarker};
use walshadow::ch_emitter::{BootstrapMode, EmitterConfig, EmitterStats};
use walshadow::config::{CliOverrides, ConfigResolver, cli_over_toml};
-use walshadow::decoder_sink::MetricsTupleObserver;
use walshadow::emit::route::routing;
use walshadow::mapping::MappingHandle;
use walshadow::metrics::{MetricsRegistry, MetricsSnapshot};
@@ -128,7 +127,7 @@ fn refuse_unowned_cluster(previous: Option<&BootstrapMarker>, dir: &Path) -> Res
/// (synthetic INSERT `_lsn = start_lsn`, `_commit_ts = 0`, `_is_deleted = 0`).
/// `wait_through(K)` proves every bootstrap seq durable on CH before
/// teardown, so the WAL pump resumes against a fully-shipped baseline.
-/// `None`: rows drain to a metrics-only observer via `drain_backfill`.
+/// `None`: rows drain counted only.
pub(crate) async fn run_bootstrap(
src_cfg: &PgConfig,
feed: &mut SourceFeed,
@@ -981,10 +980,17 @@ pub(crate) async fn run_bootstrap(
});
(rows_routed, outcome, window, landing_tracker)
} else {
- // Metrics-only skips destination convergence
- let mut observer = MetricsTupleObserver::default();
- let (drain_res, pump_res) = tokio::join!(drain_backfill(rx, &mut observer), pump);
- let shipped = drain_res.context("bootstrap drain")?;
+ // Metrics-only skips destination convergence; the walk already
+ // validated every row against its descriptor
+ let count = async move {
+ let mut rx = rx;
+ let mut shipped = 0u64;
+ while let Some(slab) = rx.recv().await {
+ shipped += slab.len() as u64;
+ }
+ shipped
+ };
+ let (shipped, pump_res) = tokio::join!(count, pump);
let outcome: BootstrapOutcome = pump_res
.context("bootstrap pump join")?
.context("bootstrap pump")?;
diff --git a/src/bin/stream/metrics_publish.rs b/src/bin/stream/metrics_publish.rs
index ce291798..1197b6f0 100644
--- a/src/bin/stream/metrics_publish.rs
+++ b/src/bin/stream/metrics_publish.rs
@@ -235,8 +235,6 @@ pub(crate) struct DrainResident {
pub(crate) chunks: u64,
pub(crate) rows: u64,
pub(crate) spool: u64,
- pub(crate) raw_pending_rows: u64,
- pub(crate) raw_pending_bytes: u64,
}
impl DrainResident {
@@ -246,8 +244,6 @@ impl DrainResident {
chunks: b.drain_chunk_resident_bytes(),
rows: b.drain_row_resident_bytes(),
spool: b.toast_spool_bytes(),
- raw_pending_rows: b.raw_pending_rows(),
- raw_pending_bytes: b.raw_pending_bytes(),
}
}
}
@@ -431,8 +427,6 @@ pub(crate) async fn populate_pipeline_metrics(
xact_stats.raw_stash_bytes_mem,
xact_stats.raw_stash_bytes_spill,
],
- raw_pending_rows: drain_resident.raw_pending_rows,
- raw_pending_bytes: drain_resident.raw_pending_bytes,
pump_queue_depth,
queue_records_out_total,
catalog_boundary_holds_total: boundary_hold.holds.load(Ordering::Relaxed),
@@ -592,6 +586,7 @@ pub(crate) fn stage_gauges_on(v: &StageCounters<'_>, base: MetricsSnapshot) -> M
|s| &s.plan_failures_detoast,
|s| &s.plan_failures_partial_update,
|s| &s.plan_failures_view,
+ |s| &s.plan_failures_encode,
|s| &s.plan_failures_drain,
],
),
@@ -613,7 +608,11 @@ pub(crate) fn stage_gauges_on(v: &StageCounters<'_>, base: MetricsSnapshot) -> M
insertbatch_batches_out_total: emitter(|s| &s.insertbatch_batches_out),
inserter_batches_in_total: emitter(|s| &s.inserter_batches_in),
inserter_ch_seconds_total: emitter_seconds(|s| &s.inserter_ch_nanos),
- inserter_encode_seconds_total: emitter_seconds(|s| &s.inserter_encode_nanos),
+ resident_seal_seconds_total: emitter_seconds(|s| &s.resident_seal_nanos),
+ resident_raw_bytes_total: emitter(|s| &s.resident_raw_bytes),
+ resident_stored_bytes_total: emitter(|s| &s.resident_stored_bytes),
+ output_resident_bytes: emitter(|s| &s.output_resident_bytes),
+ batcher_open_bytes: emitter(|s| &s.batcher_open_bytes),
oracle_resolve_seconds_total: emitter_seconds(|s| &s.oracle_resolve_nanos),
process_cpu_seconds_total: proc_cpu,
process_resident_memory_bytes: proc_rss,
@@ -752,8 +751,6 @@ mod tests {
chunks: 0,
rows: n * 10,
spool: 0,
- raw_pending_rows: n,
- raw_pending_bytes: n * 10,
},
budget: None,
decoder_stats: &decoder,
@@ -777,7 +774,6 @@ mod tests {
assert_eq!(snap.pump_queue_depth, n);
assert_eq!(snap.queue_records_out_total, n * 3);
assert_eq!(snap.drain_resident_bytes, n * 10);
- assert_eq!(snap.raw_pending_rows, n);
assert_eq!(snap.uptime_seconds, n);
assert_eq!(snap.archive_restore_active, 1);
assert_eq!(snap.archive_wal_segments_total, 42);
diff --git a/src/bin/stream/session.rs b/src/bin/stream/session.rs
index 814455b6..fc1ec7ce 100644
--- a/src/bin/stream/session.rs
+++ b/src/bin/stream/session.rs
@@ -1235,7 +1235,7 @@ async fn session_body(
emitter_cfg.row_budget = rc.row_budget;
emitter_cfg.byte_budget = rc.byte_budget;
emitter_cfg.flush_timeout = rc.flush_timeout;
- emitter_cfg.conn.compression = rc.conn.compression;
+ emitter_cfg.compression = rc.compression;
emitter_cfg.retry.max_attempts = rc.retry_max_attempts;
}
applicators.insert(conn.oid, built.applicator);
@@ -1283,7 +1283,7 @@ async fn session_body(
} else {
// Metrics-only (no CH): the identical pipeline with a null tail —
// zero CH connections, no DDL applicator, no oracle (nothing ships,
- // PgPending stays raw). The empty mapping routes nothing, so seqs
+ // out-of-matrix values stay stored bytes). The empty mapping routes nothing, so seqs
// complete at placement and the watermark + slot advance move as in
// a CH run. Emitter stats stay unexported (`emitter_stats_handle`
// None), matching the old serial surface.
@@ -1342,7 +1342,7 @@ async fn session_body(
)
.await?;
let mirror_resolver = reorder_sink.planning_resolver();
- let predecode = Arc::new(decoder.predecoder(reorder_sink.planning_resolver()));
+ let prescan = Arc::new(decoder.prescan(reorder_sink.planning_resolver()));
let decoder_xact = QueueingRecordSink::spawn(
DecoderXactPair {
decoder,
@@ -1352,7 +1352,7 @@ async fn session_body(
decoder_queue_capacity,
span_registry.clone(),
)
- .with_predecode(predecode);
+ .with_prescan(prescan);
let boundary_gate = CatalogBoundaryGate::new(
shadow_state.clone(),
BoundaryGateConfig {
diff --git a/src/bin/stream/sinks.rs b/src/bin/stream/sinks.rs
index 413623f9..e9184898 100644
--- a/src/bin/stream/sinks.rs
+++ b/src/bin/stream/sinks.rs
@@ -6,7 +6,6 @@ use anyhow::Result;
use std::future::Future;
use std::pin::Pin;
use walshadow::boundary_hold::BoundaryHoldSink;
-use walshadow::heap_decoder::Predecoded;
use walshadow::record::{MetricsRecordSink, Record, RecordSink, SinkError};
use walshadow::xact_buffer::BufferingDecoderSink;
@@ -30,7 +29,7 @@ impl RecordSink for DecoderXactPair {
&'a mut self,
record: &'a Record<'a>,
) -> Pin> + Send + 'a>> {
- self.on_records(std::slice::from_ref(record), &mut [])
+ self.on_records(std::slice::from_ref(record))
}
/// Absorbs every heap record before draining any commit in the batch:
@@ -38,11 +37,10 @@ impl RecordSink for DecoderXactPair {
fn on_records<'a>(
&'a mut self,
batch: &'a [Record<'a>],
- pre: &'a mut [Option],
) -> Pin> + Send + 'a>> {
Box::pin(async move {
- self.decoder.on_records(batch, pre).await?;
- self.xact_drain.on_records(batch, &mut []).await?;
+ self.decoder.on_records(batch).await?;
+ self.xact_drain.on_records(batch).await?;
Ok(())
})
}
diff --git a/src/bin/stream/source_db.rs b/src/bin/stream/source_db.rs
index 6f3977b6..b419333b 100644
--- a/src/bin/stream/source_db.rs
+++ b/src/bin/stream/source_db.rs
@@ -124,7 +124,7 @@ pub(crate) async fn build_source_db(input: SourceDbInputs<'_>) -> anyhow::Result
cfg.row_budget = rc.row_budget;
cfg.byte_budget = rc.byte_budget;
cfg.flush_timeout = rc.flush_timeout;
- cfg.conn.compression = rc.conn.compression;
+ cfg.compression = rc.compression;
cfg.retry.max_attempts = rc.retry_max_attempts;
}
// DDL applicator owned by the reorder coordinator so ALTER /
diff --git a/src/budget.rs b/src/budget.rs
index a09723e2..f1c4e4d1 100644
--- a/src/budget.rs
+++ b/src/budget.rs
@@ -156,6 +156,8 @@ struct Inner {
/// (mixed pool), the whole pool otherwise (leaf-only pool)
leaf_max: usize,
cur: AtomicU64,
+ /// Process-wide sum over pools of one kind, eg every tail's output pool
+ gauge: Option>,
peak: AtomicU64,
waits: AtomicU64,
overshoots: AtomicU64,
@@ -178,6 +180,15 @@ impl MemoryBudget {
/// `leaf_reserve` bytes stay out of admission's reach so per-value
/// leaf acquires under held admission permits always complete
pub fn with_leaf_reserve(total: usize, leaf_reserve: usize) -> Self {
+ Self::build(total, leaf_reserve, None)
+ }
+
+ /// [`Self::with_leaf_reserve`] also mirroring held bytes into `gauge`
+ pub fn gauged(total: usize, leaf_reserve: usize, gauge: Arc) -> Self {
+ Self::build(total, leaf_reserve, Some(gauge))
+ }
+
+ fn build(total: usize, leaf_reserve: usize, gauge: Option>) -> Self {
let total = total.min(MAX_BYTES);
let admission_max = total.saturating_sub(leaf_reserve);
Self {
@@ -194,6 +205,7 @@ impl MemoryBudget {
leaf_reserve.min(total)
},
cur: AtomicU64::new(0),
+ gauge,
peak: AtomicU64::new(0),
waits: AtomicU64::new(0),
overshoots: AtomicU64::new(0),
@@ -208,6 +220,11 @@ impl MemoryBudget {
self.inner.total
}
+ /// Bytes admission can hold, `total` minus leaf reserve
+ pub fn admission_max(&self) -> usize {
+ self.inner.admission_max
+ }
+
/// Share a single leaf acquire waits on: the reserve on a mixed pool,
/// the whole pool on a leaf-only one. A holder staying inside it
/// never withholds bytes admission needs
@@ -369,6 +386,16 @@ impl Inner {
fn meter(&self, bytes: u64) {
let now = self.cur.fetch_add(bytes, Ordering::Relaxed) + bytes;
self.peak.fetch_max(now, Ordering::Relaxed);
+ if let Some(g) = &self.gauge {
+ g.fetch_add(bytes, Ordering::Relaxed);
+ }
+ }
+
+ fn unmeter(&self, bytes: u64) {
+ self.cur.fetch_sub(bytes, Ordering::Relaxed);
+ if let Some(g) = &self.gauge {
+ g.fetch_sub(bytes, Ordering::Relaxed);
+ }
}
/// Atomically reserve `want` if capacity is available and queues in
@@ -574,16 +601,27 @@ impl MemoryPermit {
freed |= BIG_FREE;
}
self.inner.release(freed);
- self.inner
- .cur
- .fetch_sub(self.bytes - bytes, Ordering::Relaxed);
+ self.inner.unmeter(self.bytes - bytes);
+ self.bytes = bytes;
+ }
+
+ /// Set held bytes to `bytes` once known: shrink, or account growth past
+ /// what was reserved as overshoot. Growth never waits, so an owner can
+ /// always finish output it already started
+ pub fn settle(&mut self, bytes: u64) {
+ if bytes <= self.bytes {
+ self.shrink(bytes);
+ return;
+ }
+ self.inner.overshoots.fetch_add(1, Ordering::Relaxed);
+ self.inner.meter(bytes - self.bytes);
self.bytes = bytes;
}
}
impl Drop for MemoryPermit {
fn drop(&mut self) {
- self.inner.cur.fetch_sub(self.bytes, Ordering::Relaxed);
+ self.inner.unmeter(self.bytes);
let big = if self.big_leaf { BIG_FREE } else { 0 };
self.inner
.release(word(self.units, self.admission_units) | big);
@@ -948,6 +986,26 @@ mod tests {
assert_eq!(b.resident_bytes(), 0);
}
+ #[tokio::test(flavor = "current_thread")]
+ async fn settle_grows_unmetered_and_gauge_sums_pools() {
+ let gauge = Arc::new(AtomicU64::new(0));
+ let a = MemoryBudget::gauged(64 << 10, 16 << 10, gauge.clone());
+ let b = MemoryBudget::gauged(64 << 10, 0, gauge.clone());
+ let mut p = a.admit(32 << 10).await;
+ let q = b.acquire(8 << 10).await;
+ assert_eq!(gauge.load(Ordering::Relaxed), 40 << 10);
+ // Growth past admission keeps admission units free for others
+ p.settle(48 << 10);
+ assert_eq!(a.overshoots_total(), 1);
+ assert_eq!(a.resident_bytes(), 48 << 10);
+ assert!(a.try_admit(16 << 10).is_some());
+ p.settle(4 << 10);
+ assert_eq!(a.resident_bytes(), 4 << 10);
+ assert!(a.try_admit(44 << 10).is_some());
+ drop((p, q));
+ assert_eq!(gauge.load(Ordering::Relaxed), 0);
+ }
+
/// Allow only one large value above reserve at a time
#[tokio::test(flavor = "current_thread")]
async fn oversized_leaves_serialize() {
diff --git a/src/catalog/shadow_catalog.rs b/src/catalog/shadow_catalog.rs
index bd54c89a..e4d23450 100644
--- a/src/catalog/shadow_catalog.rs
+++ b/src/catalog/shadow_catalog.rs
@@ -1084,6 +1084,7 @@ fn descriptor_from_rows(
type_align: attr.attalign.to_string(),
type_storage: attr.attstorage.to_string(),
missing: None,
+ missing_send: None,
};
let mut rel_attr = raw.build().map_err(CatalogError::Parse)?;
if let Some(hex) = attr.attmissingval.as_deref()
diff --git a/src/catalog/type_bridge.rs b/src/catalog/type_bridge.rs
index 3da91b4e..3de45b33 100644
--- a/src/catalog/type_bridge.rs
+++ b/src/catalog/type_bridge.rs
@@ -37,22 +37,21 @@
//!
//! ## Default expression
//!
-//! `attmissingval` arrives as on-disk PG bytes (pinned worker) or element
-//! text (SQL path). Resolve tier-3 bytes through shadow PG before rendering
-//! `ADD COLUMN` defaults with [`column_value_to_sql_literal`]
+//! `attmissingval` arrives as on-disk PG bytes (pinned worker, SQL path for
+//! types with a wire codec) or element text (other SQL-path types). Tier-3
+//! bytes resolve to text through shadow PG first.
+//! The default encodes into its column as a row would, and
+//! `column_literal` renders that cell as the `ADD COLUMN` default
use crate::ascii_buf::{AsciiArray, AsciiBuf};
use crate::ch::types;
-use crate::decode::codecs::{
- format_time_us, macaddr_to_text, macaddr8_to_text, tid_to_text, timetz_to_text,
-};
-use crate::decode::heap_decoder::{self, ColumnValue};
-use crate::emit::ch_emitter::DATE32_PG_EPOCH_DAYS;
+use crate::decode::codecs::format_time_us;
+use crate::emit::ch_emitter::{ColumnBuf, decimal_le_to_text, encode_default};
use crate::schema::{
BOOLOID, BPCHAROID, BYTEAOID, CHAROID, CIDOID, CIDROID, DATEOID, FLOAT4OID, FLOAT8OID, INETOID,
- INT2OID, INT4OID, INT8OID, INTERVALOID, JSONBOID, JSONOID, MONEYOID, NAMEOID, NUMERICOID,
- OID8OID, OIDOID, PG_LSNOID, RelAttr, TEXTOID, TIMEOID, TIMESTAMPOID, TIMESTAMPTZOID, TIMETZOID,
- UUIDOID, VARCHAROID, XID8OID, XIDOID,
+ INT2OID, INT4OID, INT8OID, INTERVALOID, JSONBOID, JSONOID, MONEYOID, MissingDefault, NAMEOID,
+ NUMERICOID, OID8OID, OIDOID, PG_LSNOID, RelAttr, TEXTOID, TIMEOID, TIMESTAMPOID,
+ TIMESTAMPTZOID, TIMETZOID, UUIDOID, VARCHAROID, XID8OID, XIDOID,
};
use clickhouse_c::{Allocator, Kind, TypeAst};
use walrus::time::civil_from_days;
@@ -155,12 +154,13 @@ fn array_ch_type(type_name: &str) -> Option {
/// `ALTER TABLE ... ADD COLUMN ... DEFAULT k`). `None` when no fast-path
/// default or value not cleanly expressible as CH literal
fn render_default(att: &RelAttr, ch_inner: &str) -> Option {
- let _ = att.missing_default.as_ref()?;
- let value = heap_decoder::missing_value_for(att);
- if matches!(value, ColumnValue::Null) {
- return None;
+ if let Some(buf) = encode_default(att, ch_inner) {
+ return column_literal(&buf, ch_inner);
}
- column_value_to_sql_literal(&value, ch_inner)
+ let Some(MissingDefault::Text(text)) = &att.missing_default else {
+ return None;
+ };
+ pending_text_literal(text, ch_inner)
}
/// Decode `pg_attribute.atttypmod` for `numeric(p, s)`. Falls back to
@@ -200,79 +200,78 @@ fn temporal_precision(typmod: i32) -> i32 {
}
}
-/// Render PG-decoded value as CH SQL literal for after `DEFAULT `. `None`
-/// for shapes not cleanly expressible (e.g. `Unsupported.raw` needing
-/// typmod-aware reconstruction). `ch_inner` (non-nullable, from
-/// [`base_type_for`]) drives timestamp formatting + cast shape
-pub fn column_value_to_sql_literal(v: &ColumnValue, ch_inner: &str) -> Option {
- match v {
- ColumnValue::Null => Some("NULL".into()),
- ColumnValue::Bool(b) => Some(if *b { "true".into() } else { "false".into() }),
- ColumnValue::Char(i) => Some(i.to_string()),
- ColumnValue::Int2(i) => Some(i.to_string()),
- ColumnValue::Int4(i) => Some(i.to_string()),
- ColumnValue::Int8(i) => Some(i.to_string()),
- ColumnValue::Oid(i) => Some(i.to_string()),
- ColumnValue::Oid8(i) => Some(i.to_string()),
- ColumnValue::Float4(f) => Some(format_float(*f as f64)),
- ColumnValue::Float8(f) => Some(format_float(*f)),
- ColumnValue::Numeric(n) => {
- use crate::decode::codecs::NumericKind;
- match n {
- NumericKind::Finite(s) => {
- if matches!(
- types::kind(ch_inner),
- Some(
- Kind::Decimal32 | Kind::Decimal64 | Kind::Decimal128 | Kind::Decimal256
- )
- ) {
- Some(s.clone())
- } else {
- Some(sql_string_literal(s))
- }
- }
- NumericKind::NaN => Some("nan".into()),
- NumericKind::PInf => Some("inf".into()),
- NumericKind::NInf => Some("-inf".into()),
- }
- }
- ColumnValue::Text(s) | ColumnValue::Name(s) | ColumnValue::Json(s) => {
- Some(sql_string_literal(s))
+/// First cell of `buf`, a non-nullable `ch_inner` column, as a CH SQL
+/// literal for after `DEFAULT `. `None` for shapes a literal can't spell
+pub(crate) fn column_literal(buf: &ColumnBuf, ch_inner: &str) -> Option {
+ let bytes = match buf {
+ ColumnBuf::String { offsets, data, .. } => {
+ let cell = &data[..usize::try_from(*offsets.first()?).ok()?];
+ return Some(sql_bytes_literal(cell));
}
- ColumnValue::Bytea(b) => Some(sql_bytes_literal(b)),
- // PG counts days from 2000-01-01, `toDate32` from 1970-01-01
- ColumnValue::Date(days) => Some(format!(
+ ColumnBuf::Fixed { width, bytes } => bytes.get(..*width)?,
+ _ => return None,
+ };
+ let ast = TypeAst::parse(ch_inner, Allocator::global(&mimalloc::MiMalloc)).ok()?;
+ let view = ast.view();
+ let int = |n: i128| Some(n.to_string());
+ let le = |n: usize| -> Option<[u8; 16]> {
+ let mut wide = [0u8; 16];
+ wide[..n].copy_from_slice(bytes.get(..n)?);
+ Some(wide)
+ };
+ let signed = |n: usize| {
+ let shift = 128 - 8 * n as u32;
+ le(n).map(|w| (i128::from_le_bytes(w) << shift) >> shift)
+ };
+ let unsigned = |n: usize| le(n).map(|w| u128::from_le_bytes(w) as i128);
+ // Ticks at `scale` digits to microseconds
+ let micros = |ticks: i64, scale: u32| match scale.checked_sub(6) {
+ Some(down) => ticks / 10i64.pow(down),
+ None => ticks.saturating_mul(10i64.pow(6 - scale)),
+ };
+ match view.kind()? {
+ Kind::Bool => Some(if bytes[0] != 0 { "true" } else { "false" }.into()),
+ Kind::Int8 => int(signed(1)?),
+ Kind::Int16 => int(signed(2)?),
+ Kind::Int32 => int(signed(4)?),
+ Kind::Int64 => int(signed(8)?),
+ Kind::UInt8 => int(unsigned(1)?),
+ Kind::UInt16 => int(unsigned(2)?),
+ Kind::UInt32 => int(unsigned(4)?),
+ Kind::UInt64 => int(unsigned(8)?),
+ Kind::Float32 => Some(format_float(f64::from(f32::from_le_bytes(
+ bytes.try_into().ok()?,
+ )))),
+ Kind::Float64 => Some(format_float(f64::from_le_bytes(bytes.try_into().ok()?))),
+ Kind::Decimal32 | Kind::Decimal64 | Kind::Decimal128 | Kind::Decimal256 => Some(
+ decimal_le_to_text(bytes, u8::try_from(view.decimal_scale()?).ok()?),
+ ),
+ // `toDate32` counts days from 1970-01-01
+ Kind::Date32 => Some(format!(
"toDate32({})",
- i64::from(*days) + i64::from(DATE32_PG_EPOCH_DAYS)
+ i32::from_le_bytes(bytes.try_into().ok()?)
)),
- ColumnValue::Time(micros) => {
- let txt = format_time_us(*micros);
- Some(sql_string_literal(&txt))
- }
- ColumnValue::TimeTz { micros, tz_seconds } => {
- let txt = timetz_to_text(*micros, *tz_seconds);
+ Kind::Time64 => {
+ let ticks = i64::from_le_bytes(bytes.try_into().ok()?);
+ let txt = format_time_us(micros(ticks, view.datetime64_scale()?));
Some(sql_string_literal(&txt))
}
- ColumnValue::Timestamp(micros) | ColumnValue::TimestampTz(micros) => {
- let txt = render_pg_timestamp(*micros);
- let prec = datetime64_scale(ch_inner).unwrap_or(6);
+ Kind::DateTime64 => {
+ let prec = view.datetime64_scale()?;
+ let unix = micros(i64::from_le_bytes(bytes.try_into().ok()?), prec);
+ let txt = render_pg_timestamp(unix - walrus::pg::replication::PG_EPOCH_USEC);
Some(format!(
"toDateTime64({}, {prec}, 'UTC')",
sql_string_literal(&txt)
))
}
- ColumnValue::Interval(v) => Some(sql_string_literal(&v.to_text())),
- ColumnValue::Inet(v) => Some(sql_string_literal(&v.to_text())),
- ColumnValue::MacAddr(b) => Some(sql_string_literal(&macaddr_to_text(b))),
- ColumnValue::MacAddr8(b) => Some(sql_string_literal(&macaddr8_to_text(b))),
- ColumnValue::Tid { block, offset } => {
- Some(sql_string_literal(&tid_to_text(*block, *offset)))
+ Kind::Uuid => {
+ let mut pg: [u8; 16] = bytes.try_into().ok()?;
+ pg[..8].reverse();
+ pg[8..].reverse();
+ Some(format!("toUUID({})", sql_string_literal(&format_uuid(&pg))))
}
- ColumnValue::Uuid(b) => Some(format!("toUUID({})", sql_string_literal(&format_uuid(b)))),
- ColumnValue::PgPendingText { text, .. } => pending_text_literal(text, ch_inner),
- ColumnValue::PgPending { .. }
- | ColumnValue::ExternalToast(_)
- | ColumnValue::Unsupported { .. } => None,
+ _ => None,
}
}
@@ -338,16 +337,28 @@ fn format_uuid(b: &[u8; 16]) -> AsciiArray<36> {
AsciiArray::new(out)
}
-/// CH SQL string-literal escaping. Double backslash + single-quote; other
-/// control chars stay literal, CH parser accepts raw UTF-8 inside `'…'`
+/// CH SQL string literal. Doubles single quotes and backslashes; raw UTF-8
+/// stays literal inside `'…'`
fn sql_string_literal(s: &str) -> String {
- let mut out = String::with_capacity(s.len() + 2);
+ sql_bytes_literal(s.as_bytes())
+}
+
+/// [`sql_string_literal`] of arbitrary bytes, those outside UTF-8 spelled
+/// `\xHH`, which CH string literals decode to the raw byte
+fn sql_bytes_literal(b: &[u8]) -> String {
+ use std::fmt::Write as _;
+ let mut out = String::with_capacity(b.len() + 2);
out.push('\'');
- for ch in s.chars() {
- match ch {
- '\'' => out.push_str("''"),
- '\\' => out.push_str("\\\\"),
- other => out.push(other),
+ for chunk in b.utf8_chunks() {
+ for ch in chunk.valid().chars() {
+ match ch {
+ '\'' => out.push_str("''"),
+ '\\' => out.push_str("\\\\"),
+ other => out.push(other),
+ }
+ }
+ for byte in chunk.invalid() {
+ write!(out, "\\x{byte:02x}").expect("String write");
}
}
out.push('\'');
@@ -424,22 +435,28 @@ fn pg_array_items(text: &str) -> Option>> {
}
}
-/// Raw bytes as CH binary literal via `unhex('…')`; round-trip safe for
-/// non-UTF-8 payloads into CH `String` columns
-fn sql_bytes_literal(b: &[u8]) -> String {
- let mut hex = String::with_capacity(b.len() * 2);
- for byte in b {
- hex.push_str(&format!("{byte:02x}"));
- }
- format!("unhex('{hex}')")
-}
-
#[cfg(test)]
mod tests {
use super::*;
- use crate::decode::codecs::{IntervalValue, NumericKind};
+ use crate::decode::datum::{Datum, build};
+ use crate::decode::heap_decoder::{raw_missing_array, short_varlena};
use crate::schema::RelAttr;
+ /// Literal of `v` stored as a `type_oid` fast default, for `ch_inner`
+ fn literal(type_oid: u32, type_len: i16, v: &Datum, ch_inner: &str) -> Option {
+ let elem = match v {
+ Datum::Fixed(b) => b.to_vec(),
+ Datum::Varlena(b) => short_varlena(b),
+ other => panic!("no stored default for {other:?}"),
+ };
+ let mut att = attr(type_oid, -1, true, None);
+ att.type_len = type_len;
+ att.missing_default = Some(crate::schema::MissingDefault::Raw(raw_missing_array(
+ type_oid, &elem,
+ )));
+ render_default(&att, ch_inner)
+ }
+
fn attr(oid: u32, typmod: i32, not_null: bool, missing: Option<&str>) -> RelAttr {
RelAttr {
attnum: 1,
@@ -469,7 +486,7 @@ mod tests {
];
for (micros, expected) in cases {
assert_eq!(
- column_value_to_sql_literal(&ColumnValue::Time(micros), "Time64(6)"),
+ literal(TIMEOID, 8, &build::micros(micros), "Time64(6)"),
Some(sql_string_literal(expected)),
"micros={micros}"
);
@@ -489,13 +506,7 @@ mod tests {
];
for (micros, tz, expected) in cases {
assert_eq!(
- column_value_to_sql_literal(
- &ColumnValue::TimeTz {
- micros,
- tz_seconds: tz,
- },
- "String"
- ),
+ literal(TIMETZOID, 12, &build::timetz(micros, tz), "String"),
Some(sql_string_literal(expected)),
"micros={micros} tz={tz}"
);
@@ -719,12 +730,9 @@ mod tests {
#[test]
fn unresolved_disk_default_never_renders_as_string() {
- let value = ColumnValue::PgPending {
- type_oid: TSVECTOROID,
- raw: vec![0, 0, 0, 0x20],
- };
+ let value = build::bytes([0, 0, 0, 0x20]);
for ch_type in ["JSON", "String", "Array(Nullable(Int32))"] {
- assert_eq!(column_value_to_sql_literal(&value, ch_type), None);
+ assert_eq!(literal(TSVECTOROID, -1, &value, ch_type), None);
}
}
@@ -737,40 +745,45 @@ mod tests {
#[test]
fn column_value_text_escapes_singles() {
- let lit = column_value_to_sql_literal(&ColumnValue::Text("a'b".into()), "String").unwrap();
+ let lit = literal(TEXTOID, -1, &build::text("a'b"), "String").unwrap();
assert_eq!(lit, "'a''b'");
}
#[test]
fn column_value_int_round_trips() {
- let lit = column_value_to_sql_literal(&ColumnValue::Int4(-42), "Int32").unwrap();
+ let lit = literal(INT4OID, 4, &build::int4(-42), "Int32").unwrap();
assert_eq!(lit, "-42");
}
#[test]
fn column_value_numeric_to_decimal_drops_quotes() {
- let lit = column_value_to_sql_literal(
- &ColumnValue::Numeric(NumericKind::Finite("3.14".into())),
- "Decimal(10, 2)",
- )
- .unwrap();
- assert_eq!(lit, "3.14");
- let lit = column_value_to_sql_literal(
- &ColumnValue::Numeric(NumericKind::Finite("3.14".into())),
- "String",
- )
- .unwrap();
- assert_eq!(lit, "'3.14'");
+ let pi = build::numeric("3.14");
+ assert_eq!(
+ literal(NUMERICOID, -1, &pi, "Decimal(10, 2)").unwrap(),
+ "3.14"
+ );
+ assert_eq!(
+ literal(NUMERICOID, -1, &pi, "Decimal(10, 4)").unwrap(),
+ "3.1400"
+ );
+ assert_eq!(
+ literal(NUMERICOID, -1, &build::numeric("-0.05"), "Decimal(10, 2)").unwrap(),
+ "-0.05"
+ );
+ assert_eq!(literal(NUMERICOID, -1, &pi, "String").unwrap(), "'3.14'");
+ assert_eq!(
+ literal(NUMERICOID, -1, &build::numeric("NaN"), "Decimal(10, 2)"),
+ None
+ );
}
#[test]
fn column_value_interval_renders_pg_text() {
- let v = ColumnValue::Interval(IntervalValue {
- months: 0,
- days: 1,
- micros: 0,
- });
- let lit = column_value_to_sql_literal(&v, "String").unwrap();
+ // micros, days, months
+ let mut v = 0i64.to_le_bytes().to_vec();
+ v.extend_from_slice(&1i32.to_le_bytes());
+ v.extend_from_slice(&0i32.to_le_bytes());
+ let lit = literal(INTERVALOID, 16, &Datum::fixed(&v), "String").unwrap();
assert!(lit.starts_with('\''), "{lit}");
assert!(lit.contains("day"), "{lit}");
}
@@ -781,7 +794,7 @@ mod tests {
0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0xfe, 0xdc, 0xba, 0x98, 0x76, 0x54,
0x32, 0x10,
];
- let lit = column_value_to_sql_literal(&ColumnValue::Uuid(uuid), "UUID").unwrap();
+ let lit = literal(UUIDOID, 16, &build::uuid(uuid), "UUID").unwrap();
assert_eq!(
lit, "toUUID('01234567-89ab-cdef-fedc-ba9876543210')",
"uuid format mismatch: {lit}"
@@ -790,8 +803,7 @@ mod tests {
#[test]
fn column_value_timestamp_routes_through_to_datetime64() {
- let lit = column_value_to_sql_literal(&ColumnValue::TimestampTz(0), "DateTime64(6, 'UTC')")
- .unwrap();
+ let lit = literal(TIMESTAMPTZOID, 8, &build::micros(0), "DateTime64(6, 'UTC')").unwrap();
assert!(
lit.starts_with("toDateTime64('2000-01-01"),
"PG epoch should render as 2000-01-01: {lit}"
@@ -802,35 +814,26 @@ mod tests {
#[test]
fn column_value_date_shifts_to_unix_epoch() {
// 2024-01-01 is day 8766 after 2000-01-01, day 19723 after 1970-01-01
- let lit = column_value_to_sql_literal(&ColumnValue::Date(8766), "Date32").unwrap();
+ let lit = literal(DATEOID, 4, &build::date(8766), "Date32").unwrap();
assert_eq!(lit, "toDate32(19723)");
- let lit = column_value_to_sql_literal(&ColumnValue::Date(0), "Date32").unwrap();
+ let lit = literal(DATEOID, 4, &build::date(0), "Date32").unwrap();
assert_eq!(lit, "toDate32(10957)");
}
#[test]
- fn column_value_bytea_renders_unhex_literal() {
- let lit =
- column_value_to_sql_literal(&ColumnValue::Bytea(vec![0xDE, 0xAD]), "String").unwrap();
- assert_eq!(lit, "unhex('dead')");
+ fn bytea_default_escapes_invalid_utf8() {
+ let lit = literal(BYTEAOID, -1, &build::bytes([0xFF, b'\'', 0xFE]), "String").unwrap();
+ assert_eq!(lit, r"'\xff''\xfe'");
+ let lit = literal(BYTEAOID, -1, &build::bytes(b"ok"), "String").unwrap();
+ assert_eq!(lit, "'ok'");
}
#[test]
- fn column_value_null_is_explicit_null() {
- let lit = column_value_to_sql_literal(&ColumnValue::Null, "Int32").unwrap();
- assert_eq!(lit, "NULL");
- }
-
- #[test]
- fn external_toast_does_not_render() {
- use crate::decode::heap_decoder::ToastPointer;
- let v = ColumnValue::ExternalToast(ToastPointer {
- va_rawsize: 1,
- va_extinfo: 0,
- va_valueid: 0,
- va_toastrelid: 0,
- });
- assert!(column_value_to_sql_literal(&v, "String").is_none());
+ fn absent_default_renders_nothing() {
+ assert_eq!(
+ render_default(&attr(INT4OID, -1, true, None), "Int32"),
+ None
+ );
}
#[test]
@@ -913,11 +916,7 @@ mod tests {
for type_name in ["vector", "halfvec"] {
let r = map(&array_attr(type_name, "[1,2,3]"), false);
assert_eq!(r.ch_type, "Array(Float32)");
- assert_eq!(
- r.default_sql.as_deref(),
- Some("unhex('5b312c322c335d')"),
- "{type_name}"
- );
+ assert_eq!(r.default_sql.as_deref(), Some("'[1,2,3]'"), "{type_name}");
}
}
@@ -931,7 +930,7 @@ mod tests {
#[test]
fn other_pending_text_defaults_cast_from_their_text() {
let r = map(&attr(NUMERICOID, -1, true, Some("3.14")), false);
- assert_eq!(r.default_sql.as_deref(), Some("unhex('332e3134')"));
+ assert_eq!(r.default_sql.as_deref(), Some("'3.14'"));
}
#[test]
@@ -954,50 +953,92 @@ mod tests {
#[test]
fn scalar_values_render_as_ch_literals() {
- use crate::decode::codecs::InetValue;
- let mut addr = [0u8; 16];
- addr[..4].copy_from_slice(&[10, 0, 0, 1]);
+ use crate::decode::codecs::encode_tid;
+ use crate::schema::{
+ CHAROID, FLOAT4OID, FLOAT8OID, INETOID, INT2OID, INT8OID, MACADDR8OID, MACADDROID,
+ OID8OID, OIDOID, TIDOID,
+ };
let cases = [
- (ColumnValue::Char(-3), "-3"),
- (ColumnValue::Int2(7), "7"),
- (ColumnValue::Int8(-1 << 40), "-1099511627776"),
- (ColumnValue::Oid(4_000_000_000), "4000000000"),
- (ColumnValue::Oid8(u64::MAX), "18446744073709551615"),
- (ColumnValue::Float4(2.0), "2.0"),
- (ColumnValue::Float8(f64::NEG_INFINITY), "-inf"),
- (ColumnValue::Numeric(NumericKind::NaN), "nan"),
- (ColumnValue::Numeric(NumericKind::PInf), "inf"),
- (ColumnValue::Numeric(NumericKind::NInf), "-inf"),
(
- ColumnValue::Inet(InetValue {
- family: 2,
- bits: 32,
- is_cidr: false,
- addr,
- }),
+ CHAROID,
+ 1,
+ Datum::fixed(&(-3i8).to_le_bytes()),
+ "Int8",
+ "-3",
+ ),
+ (INT2OID, 2, build::int2(7), "Int16", "7"),
+ (INT8OID, 8, build::int8(-1 << 40), "Int64", "-1099511627776"),
+ (OIDOID, 4, build::oid(4_000_000_000), "UInt32", "4000000000"),
+ (
+ OID8OID,
+ 8,
+ build::oid8(u64::MAX),
+ "UInt64",
+ "18446744073709551615",
+ ),
+ (
+ FLOAT4OID,
+ 4,
+ Datum::fixed(&2.0f32.to_le_bytes()),
+ "Float32",
+ "2.0",
+ ),
+ (
+ FLOAT8OID,
+ 8,
+ Datum::fixed(&f64::NEG_INFINITY.to_le_bytes()),
+ "Float64",
+ "-inf",
+ ),
+ (NUMERICOID, -1, build::numeric("NaN"), "String", "'NaN'"),
+ (
+ NUMERICOID,
+ -1,
+ build::numeric("Infinity"),
+ "String",
+ "'Infinity'",
+ ),
+ (
+ NUMERICOID,
+ -1,
+ build::numeric("-Infinity"),
+ "String",
+ "'-Infinity'",
+ ),
+ (
+ INETOID,
+ -1,
+ build::bytes([2, 32, 10, 0, 0, 1]),
+ "String",
"'10.0.0.1'",
),
(
- ColumnValue::MacAddr([0x08, 0, 0x2b, 1, 2, 3]),
+ MACADDROID,
+ 6,
+ Datum::fixed(&[0x08, 0, 0x2b, 1, 2, 3]),
+ "String",
"'08:00:2b:01:02:03'",
),
(
- ColumnValue::MacAddr8([0x08, 0, 0x2b, 1, 2, 3, 4, 5]),
+ MACADDR8OID,
+ 8,
+ Datum::fixed(&[0x08, 0, 0x2b, 1, 2, 3, 4, 5]),
+ "String",
"'08:00:2b:01:02:03:04:05'",
),
(
- ColumnValue::Tid {
- block: 12,
- offset: 3,
- },
+ TIDOID,
+ 6,
+ Datum::fixed(&encode_tid(12, 3)),
+ "String",
"'(12,3)'",
),
];
- for (value, want) in cases {
+ for (type_oid, type_len, value, ch_inner, want) in cases {
assert_eq!(
- column_value_to_sql_literal(&value, "String").as_deref(),
+ literal(type_oid, type_len, &value, ch_inner).as_deref(),
Some(want),
- "{value:?}"
+ "{type_oid} {value:?}"
);
}
}
@@ -1007,4 +1048,32 @@ mod tests {
assert_eq!(pg_array_items(r#"{"a"x}"#), None);
assert_eq!(pending_text_literal(r#"{"a"x}"#, "Array(String)"), None);
}
+
+ #[test]
+ fn fixed_cells_render_by_target_width() {
+ let fixed = |bytes: &[u8]| ColumnBuf::Fixed {
+ width: bytes.len(),
+ bytes: bytes.to_vec(),
+ };
+ let cases: [(&[u8], &str, &str); 10] = [
+ (&[0xFE], "Int8", "-2"),
+ (&(-3i16).to_le_bytes(), "Int16", "-3"),
+ (&(-5i64).to_le_bytes(), "Int64", "-5"),
+ (&[0xFE], "UInt8", "254"),
+ (&u16::MAX.to_le_bytes(), "UInt16", "65535"),
+ (&u32::MAX.to_le_bytes(), "UInt32", "4294967295"),
+ (&u64::MAX.to_le_bytes(), "UInt64", "18446744073709551615"),
+ (&1.5f32.to_le_bytes(), "Float32", "1.5"),
+ (&(-0.25f64).to_le_bytes(), "Float64", "-0.25"),
+ (&7i64.to_le_bytes(), "Time64(3)", "'00:00:00.007'"),
+ ];
+ for (bytes, ch, want) in cases {
+ assert_eq!(
+ column_literal(&fixed(bytes), ch).as_deref(),
+ Some(want),
+ "{ch}"
+ );
+ }
+ assert_eq!(column_literal(&fixed(&[1]), "IPv4"), None);
+ }
}
diff --git a/src/ch.rs b/src/ch.rs
index e6fbd3da..258bb628 100644
--- a/src/ch.rs
+++ b/src/ch.rs
@@ -77,7 +77,7 @@ impl std::str::FromStr for CompressionChoice {
}
impl CompressionChoice {
- fn to_wire(self) -> Compression {
+ pub(crate) fn to_wire(self) -> Compression {
match self {
Self::None => Compression::None,
Self::Lz4 => Compression::Lz4,
@@ -302,6 +302,7 @@ pub async fn connect_creating_database(
struct Connection {
client: Option,
dialed: Option>,
+ compression: CompressionChoice,
last_used: Instant,
}
@@ -310,6 +311,7 @@ impl Default for Connection {
Self {
client: None,
dialed: None,
+ compression: CompressionChoice::None,
last_used: Instant::now(),
}
}
@@ -319,6 +321,8 @@ impl Default for Connection {
pub struct ChConn {
instance: Option,
selected: Option>,
+ /// Dial with this instead of configured compression
+ compression: Option,
active: Connection,
parked: foldhash::HashMap, Connection>,
dest: Arc,
@@ -332,6 +336,7 @@ impl ChConn {
dest,
instance: None,
selected: None,
+ compression: None,
active: Connection::default(),
parked: Default::default(),
dials: 0,
@@ -366,6 +371,12 @@ impl ChConn {
Ok(())
}
+ /// Dial every destination with `compression`, so prepared blocks sealed
+ /// in it send as is
+ pub fn force_compression(&mut self, compression: CompressionChoice) {
+ self.compression = Some(compression);
+ }
+
/// Redial now, for settings fixed at connect (codec, host, credentials)
pub async fn dial(&mut self) -> Result<(), EmitterError> {
self.active.client = None;
@@ -446,23 +457,32 @@ impl ChConn {
self.selected = Some(self.dest.instance(name)?);
}
let config = self.config();
+ let compression = self.compression.unwrap_or(config.compression);
let moved = self
.active
.dialed
.as_ref()
.is_none_or(|dialed| !Arc::ptr_eq(dialed, &config));
- if moved || self.active.last_used.elapsed() >= config.idle_reconnect() {
+ if moved
+ || compression != self.active.compression
+ || self.active.last_used.elapsed() >= config.idle_reconnect()
+ {
self.active.client = None;
}
if let Some(client) = self.active.client.take() {
return Ok(client);
}
+ let mut dial = config.clone();
+ if compression != config.compression {
+ Arc::make_mut(&mut dial).compression = compression;
+ }
let client = if self.instance.is_some() {
- connect_creating_database(&config).await?.0
+ connect_creating_database(&dial).await?.0
} else {
- connect_client(&*config).await?
+ connect_client(&*dial).await?
};
self.active.dialed = Some(config);
+ self.active.compression = compression;
self.active.last_used = Instant::now();
self.dials += 1;
Ok(client)
diff --git a/src/ch/test_support.rs b/src/ch/test_support.rs
index e1563567..e1119bc9 100644
--- a/src/ch/test_support.rs
+++ b/src/ch/test_support.rs
@@ -7,20 +7,23 @@ use tokio::task::JoinHandle;
use crate::ch::CompressionChoice;
use crate::emit::ch_emitter::{EmitterConfig, InstanceConfig, RetryConfig};
+/// Server expecting `sql` then `blocks` one-row Data blocks per attempt.
+/// First attempt drops connection without reply, after `cut` blocks when set
pub(crate) async fn retry_server(
max_attempts: u32,
sql: &str,
- insert: bool,
+ blocks: usize,
idle: bool,
+ cut: Option,
) -> (EmitterConfig, JoinHandle<()>) {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let config = EmitterConfig {
conn: InstanceConfig {
host: "127.0.0.1".into(),
port: listener.local_addr().unwrap().port(),
- compression: CompressionChoice::None,
..Default::default()
},
+ compression: CompressionChoice::None,
retry: RetryConfig {
max_attempts,
initial_backoff: Duration::from_millis(1),
@@ -44,11 +47,16 @@ pub(crate) async fn retry_server(
const EMPTY_DATA: &[u8] = &[2, 0, 1, 0, 2, 0xff, 0xff, 0xff, 0xff, 0, 0, 0];
expected.push(0);
expected.extend_from_slice(EMPTY_DATA);
- if insert {
- // One UInt8 column x without custom serialization, one row 42
- expected.extend_from_slice(
- b"\x02\x00\x01\x00\x02\xff\xff\xff\xff\x00\x01\x01\x01x\x05UInt8\x00\x2a",
- );
+ // One UInt8 column x without custom serialization, one row 42
+ const BLOCK: &[u8] = b"\x02\x00\x01\x00\x02\xff\xff\xff\xff\x00\x01\x01\x01x\x05UInt8\x00\x2a";
+ let cut_at = cut.map(|n| {
+ assert!(n < blocks);
+ expected.len() + n * BLOCK.len()
+ });
+ for _ in 0..blocks {
+ expected.extend_from_slice(BLOCK);
+ }
+ if blocks > 0 {
expected.extend_from_slice(EMPTY_DATA);
}
let server = tokio::spawn(async move {
@@ -74,6 +82,12 @@ pub(crate) async fn retry_server(
if idle && connection == 0 {
continue;
}
+ if let Some(at) = cut_at.filter(|_| connection == 0) {
+ let mut request = vec![0; at];
+ stream.read_exact(&mut request).await.unwrap();
+ assert_eq!(request, expected[..at]);
+ continue;
+ }
let mut request = vec![0; expected.len()];
stream.read_exact(&mut request).await.unwrap();
assert_eq!(request, expected);
diff --git a/src/config.rs b/src/config.rs
index e382f4f1..949120bc 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -158,9 +158,10 @@ pub struct ResolvedConfig {
pub flush_timeout: Duration,
/// CH client retry budget (live: inserter reads per attempt loop)
pub retry_max_attempts: u32,
- /// CH connection (live: inserter + DDL applicator reconnect on change,
- /// inserter rebuilds its codec on compression change)
+ /// CH connection (live: inserter + DDL applicator reconnect on change)
pub conn: InstanceConfig,
+ /// Wire compression (live: resolvers seal later batches in it)
+ pub compression: CompressionChoice,
/// Source PostgreSQL connection and slot, live via config reload
pub source: SourceConn,
/// Columns-less `[table.*]` opt-in intents (live: reorder coordinator
@@ -183,6 +184,7 @@ impl ResolvedConfig {
pub fn dest_conn_eq(&self, cfg: &EmitterConfig) -> bool {
cfg.instances == self.instances
&& cfg.conn == self.conn
+ && cfg.compression == self.compression
&& cfg.retry.max_attempts == self.retry_max_attempts
}
@@ -190,6 +192,7 @@ impl ResolvedConfig {
let mut out = EmitterConfig {
instances: self.instances.clone(),
conn: self.conn.clone(),
+ compression: self.compression,
..base.clone()
};
out.retry.max_attempts = self.retry_max_attempts;
@@ -786,6 +789,7 @@ impl ConfigResolver {
flush_timeout: base.flush_timeout,
retry_max_attempts: base.retry.max_attempts,
conn: base.conn.clone(),
+ compression: base.compression,
source: base.source.clone(),
table_opt_ins: base.table_opt_ins.clone(),
paused: base.paused,
@@ -863,7 +867,7 @@ impl ConfigResolver {
// codec (e.g. zstd with the feature off) is rejected here, never
// surfaced as a fatal when the inserter reconnects.
match v.parse().map(|c: CompressionChoice| (c, c.build_codec())) {
- Ok((c, Ok(_))) => rc.conn.compression = c,
+ Ok((c, Ok(_))) => rc.compression = c,
_ => {
rejections += 1;
tracing::warn!(target: "walshadow::config", value = %v, "config_global.compression rejected");
@@ -1160,7 +1164,7 @@ mod tests {
assert_eq!(r.row_budget, 1000);
assert_eq!(r.byte_budget, 4096);
assert_eq!(r.flush_timeout, Duration::from_millis(250));
- assert_eq!(r.conn.compression, CompressionChoice::None);
+ assert_eq!(r.compression, CompressionChoice::None);
assert_eq!(r.retry_max_attempts, 9);
}
@@ -1206,7 +1210,7 @@ mod tests {
// Prior (TOML/base) values survive each rejection.
assert_eq!(r.drop_table_strategy, DropTableStrategy::Retain);
assert_eq!(r.detach_strategy, DropTableStrategy::Retain);
- assert_eq!(r.conn.compression, base.conn.compression);
+ assert_eq!(r.compression, base.compression);
assert_eq!(r.row_budget, base.row_budget);
assert_eq!(r.byte_budget, base.byte_budget);
assert_eq!(r.flush_timeout, base.flush_timeout);
diff --git a/src/crash_test.rs b/src/crash_test.rs
index 1bb34334..7b5d3d40 100644
--- a/src/crash_test.rs
+++ b/src/crash_test.rs
@@ -14,6 +14,8 @@ pub enum Point {
BeforeDdl,
AfterAlter,
AfterDdl,
+ /// One destination accepted rows, later destinations and ack pending
+ AfterInsert,
AfterAck,
AfterManifest,
BootstrapHandoff,
diff --git a/src/decode/codecs/fixed.rs b/src/decode/codecs/fixed.rs
index c7efbc70..8014abdd 100644
--- a/src/decode/codecs/fixed.rs
+++ b/src/decode/codecs/fixed.rs
@@ -42,6 +42,15 @@ pub(crate) fn decode_tid(b: &[u8; 6]) -> (u32, u16) {
)
}
+/// Inverse of [`decode_tid`]
+pub(crate) fn encode_tid(block: u32, offset: u16) -> [u8; 6] {
+ let mut out = [0u8; 6];
+ out[..2].copy_from_slice(&((block >> 16) as u16).to_le_bytes());
+ out[2..4].copy_from_slice(&(block as u16).to_le_bytes());
+ out[4..].copy_from_slice(&offset.to_le_bytes());
+ out
+}
+
/// `tid` as `tidout` prints it
pub(crate) fn tid_to_text(block: u32, offset: u16) -> AsciiBuf<18> {
let mut out = AsciiBuf::new();
@@ -78,6 +87,7 @@ mod tests {
#[test]
fn tid_joins_block_halves() {
assert_eq!(decode_tid(&[1, 0, 2, 0, 3, 0]), (65538, 3));
+ assert_eq!(encode_tid(65538, 3), [1, 0, 2, 0, 3, 0]);
let (block, offset) = decode_tid(&[0xff; 6]);
assert_eq!(tid_to_text(block, offset).as_str(), "(4294967295,65535)");
assert_eq!(parse_tid("(4294967295,65535)"), Some((block, offset)));
diff --git a/src/decode/codecs/inet.rs b/src/decode/codecs/inet.rs
index 9d2bbd0b..b1674450 100644
--- a/src/decode/codecs/inet.rs
+++ b/src/decode/codecs/inet.rs
@@ -18,7 +18,7 @@ use crate::ascii_buf::AsciiBuf;
pub const PGSQL_AF_INET: u8 = 2;
pub const PGSQL_AF_INET6: u8 = 3;
-#[derive(Debug, Clone, PartialEq, Eq)]
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct InetValue {
pub family: u8,
pub bits: u8,
diff --git a/src/decode/codecs/jsonb.rs b/src/decode/codecs/jsonb.rs
index 890a8d2d..aa7d968b 100644
--- a/src/decode/codecs/jsonb.rs
+++ b/src/decode/codecs/jsonb.rs
@@ -36,14 +36,14 @@ const JSONB_MAX_DEPTH: usize = 1000;
/// Decode `jsonb` varlena body to PG `jsonb_out` text (`JsonbToCString`)
pub fn decode_jsonb(body: &[u8]) -> Result {
- decode_jsonb_with_capacity(body, body.len())
+ let mut out = String::with_capacity(body.len());
+ render_jsonb_text(body, &mut out)?;
+ Ok(out)
}
-/// [`decode_jsonb`] into a buffer sized by [`jsonb_text_len`]
-pub fn decode_jsonb_with_capacity(body: &[u8], capacity: usize) -> Result {
- let mut out = String::with_capacity(capacity);
- render_jsonb_container(body, 0, &mut out)?;
- Ok(out)
+/// `jsonb_out` text appended to `out`
+pub fn render_jsonb_text(body: &[u8], out: &mut String) -> Result<(), CodecError> {
+ render_jsonb_container(body, 0, out)
}
/// [`decode_jsonb`] length without rendering. Text may outgrow its body,
@@ -453,7 +453,8 @@ mod tests {
for (hex, want) in JSONB_VECTORS {
let body = varlena_body(hex);
let len = jsonb_text_len(&body).expect(hex);
- let text = decode_jsonb_with_capacity(&body, len).expect(hex);
+ let mut text = String::with_capacity(len);
+ render_jsonb_text(&body, &mut text).expect(hex);
assert_eq!(
(text.as_str(), text.capacity()),
(*want, want.len()),
diff --git a/src/decode/codecs/mod.rs b/src/decode/codecs/mod.rs
index fe2fb6c3..e9b334f9 100644
--- a/src/decode/codecs/mod.rs
+++ b/src/decode/codecs/mod.rs
@@ -11,12 +11,17 @@ mod time;
mod uuid;
pub(crate) use fixed::{
- decode_tid, macaddr_to_text, macaddr8_to_text, parse_macaddr, parse_tid, tid_to_text,
+ decode_tid, encode_tid, macaddr_to_text, macaddr8_to_text, parse_macaddr, parse_tid,
+ tid_to_text,
};
pub use inet::{InetValue, PGSQL_AF_INET, PGSQL_AF_INET6, decode_inet};
pub use interval::{IntervalValue, decode_interval};
-pub use jsonb::{decode_jsonb, decode_jsonb_with_capacity, jsonb_text_len};
-pub use numeric::{NumericKind, decode_numeric, numeric_text_len};
+pub use jsonb::{decode_jsonb, jsonb_text_len, render_jsonb_text};
+pub(crate) use numeric::{Numeric, parse_numeric, scaled_groups};
+pub use numeric::{
+ Special, numeric_body, numeric_from_text, numeric_special_body, numeric_text_len,
+ render_numeric_text,
+};
pub use text_array::decode_text_array;
pub(crate) use time::{format_time_us, timetz_to_text};
pub(crate) use uuid::uuid_to_ch_wire;
diff --git a/src/decode/codecs/numeric.rs b/src/decode/codecs/numeric.rs
index 30613599..f899e9be 100644
--- a/src/decode/codecs/numeric.rs
+++ b/src/decode/codecs/numeric.rs
@@ -40,31 +40,37 @@ const NUMERIC_SHORT_WEIGHT_MASK: u16 = 0x003F;
const NUMERIC_DSCALE_MASK: u16 = 0x3FFF;
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub enum NumericKind {
+/// Non-finite `numeric`, a header with no digits
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Special {
NaN,
/// `+Infinity` (PG 14+)
PInf,
/// `-Infinity` (PG 14+)
NInf,
- /// PG-text form
- Finite(String),
}
-impl NumericKind {
- pub(crate) fn as_text(&self) -> &str {
+impl Special {
+ pub(crate) fn as_text(self) -> &'static str {
match self {
- Self::Finite(s) => s,
Self::NaN => "NaN",
Self::PInf => "Infinity",
Self::NInf => "-Infinity",
}
}
+
+ fn header(self) -> u16 {
+ match self {
+ Self::NaN => NUMERIC_NAN,
+ Self::PInf => NUMERIC_PINF,
+ Self::NInf => NUMERIC_NINF,
+ }
+ }
}
/// `numeric` header fields plus its base-10000 digits, still in the datum
-enum Numeric<'a> {
- Special(NumericKind),
+pub(crate) enum Numeric<'a> {
+ Special(Special),
Finite {
neg: bool,
weight: i32,
@@ -74,7 +80,7 @@ enum Numeric<'a> {
},
}
-fn parse_numeric(body: &[u8]) -> Result, CodecError> {
+pub(crate) fn parse_numeric(body: &[u8]) -> Result, CodecError> {
if body.len() < 2 {
return Err(CodecError::Truncated {
offset: 0,
@@ -89,10 +95,10 @@ fn parse_numeric(body: &[u8]) -> Result, CodecError> {
if is_special {
return Ok(Numeric::Special(match n_header {
- NUMERIC_NAN => NumericKind::NaN,
- NUMERIC_PINF => NumericKind::PInf,
- NUMERIC_NINF => NumericKind::NInf,
- _ => NumericKind::NaN, // unknown special, treat as NaN (lossy-safe)
+ NUMERIC_PINF => Special::PInf,
+ NUMERIC_NINF => Special::NInf,
+ // Unknown special reads as NaN, lossy but safe
+ _ => Special::NaN,
}));
}
@@ -141,26 +147,8 @@ fn parse_numeric(body: &[u8]) -> Result, CodecError> {
})
}
-/// Decode `numeric` varlena body to PG `numeric_out` text or special value
-pub fn decode_numeric(body: &[u8]) -> Result {
- match parse_numeric(body)? {
- Numeric::Special(kind) => Ok(kind),
- Numeric::Finite {
- neg,
- weight,
- dscale,
- digits,
- } => {
- let mut out = String::with_capacity(finite_text_len(neg, weight, dscale, digits));
- render_numeric(neg, weight, dscale, digits, &mut out);
- Ok(NumericKind::Finite(out))
- }
- }
-}
-
-/// `numeric_out` text into a caller's buffer, for callers holding one already:
-/// `jsonb`'s numeric children render straight into the enclosing document
-pub(crate) fn render_numeric_text(body: &[u8], out: &mut String) -> Result<(), CodecError> {
+/// `numeric_out` text appended to `out`
+pub fn render_numeric_text(body: &[u8], out: &mut String) -> Result<(), CodecError> {
match parse_numeric(body)? {
Numeric::Special(kind) => out.push_str(kind.as_text()),
Numeric::Finite {
@@ -173,7 +161,7 @@ pub(crate) fn render_numeric_text(body: &[u8], out: &mut String) -> Result<(), C
Ok(())
}
-/// [`decode_numeric`] text length without rendering
+/// [`render_numeric_text`] length without rendering
pub fn numeric_text_len(body: &[u8]) -> Result {
Ok(match parse_numeric(body)? {
Numeric::Special(kind) => kind.as_text().len(),
@@ -262,6 +250,121 @@ fn render_numeric(neg: bool, weight: i32, dscale: i32, digits: &[[u8; 2]], out:
}
}
+const NUMERIC_SHORT_DSCALE_MAX: i32 = 0x3F;
+const NUMERIC_SHORT_WEIGHT_MAX: i32 = 63;
+const NUMERIC_SHORT_WEIGHT_MIN: i32 = -64;
+const NBASE: u32 = 10_000;
+
+/// Body PostgreSQL stores for a non-finite value
+pub fn numeric_special_body(special: Special) -> Vec {
+ special.header().to_le_bytes().to_vec()
+}
+
+/// Body `make_result` stores: zero groups trimmed from both ends, short
+/// header when weight and dscale fit
+pub fn numeric_body(neg: bool, weight: i32, dscale: u16, digits: &[i16]) -> Vec {
+ let lead = digits.iter().take_while(|&&d| d == 0).count();
+ let digits = &digits[lead..];
+ let digits = &digits[..digits.len() - digits.iter().rev().take_while(|&&d| d == 0).count()];
+ let (neg, weight) = if digits.is_empty() {
+ (false, 0)
+ } else {
+ (neg, weight - lead as i32)
+ };
+ let dscale = i32::from(dscale & NUMERIC_DSCALE_MASK);
+ let mut out = Vec::with_capacity(4 + 2 * digits.len());
+ if dscale <= NUMERIC_SHORT_DSCALE_MAX
+ && (NUMERIC_SHORT_WEIGHT_MIN..=NUMERIC_SHORT_WEIGHT_MAX).contains(&weight)
+ {
+ let sign = if neg { NUMERIC_SHORT_SIGN_MASK } else { 0 };
+ let weight_sign = if weight < 0 {
+ NUMERIC_SHORT_WEIGHT_SIGN_MASK
+ } else {
+ 0
+ };
+ let header = NUMERIC_SHORT
+ | sign
+ | ((dscale as u16) << NUMERIC_SHORT_DSCALE_SHIFT)
+ | weight_sign
+ | (weight as u16 & NUMERIC_SHORT_WEIGHT_MASK);
+ out.extend_from_slice(&header.to_le_bytes());
+ } else {
+ let sign = if neg { NUMERIC_NEG } else { NUMERIC_POS };
+ out.extend_from_slice(&(sign | dscale as u16).to_le_bytes());
+ out.extend_from_slice(&(weight as i16).to_le_bytes());
+ }
+ for d in digits {
+ out.extend_from_slice(&d.to_le_bytes());
+ }
+ out
+}
+
+/// `numeric_in` for plain decimal text and the special spellings
+pub fn numeric_from_text(text: &str) -> Option> {
+ let text = text.trim();
+ match text.to_ascii_lowercase().as_str() {
+ "nan" => return Some(numeric_special_body(Special::NaN)),
+ "infinity" | "+infinity" | "inf" | "+inf" => {
+ return Some(numeric_special_body(Special::PInf));
+ }
+ "-infinity" | "-inf" => return Some(numeric_special_body(Special::NInf)),
+ _ => {}
+ }
+ let (neg, body) = match text.as_bytes().first()? {
+ b'-' => (true, &text[1..]),
+ b'+' => (false, &text[1..]),
+ _ => (false, text),
+ };
+ let (int, frac) = body.split_once('.').unwrap_or((body, ""));
+ let all_digits = |s: &str| s.bytes().all(|b| b.is_ascii_digit());
+ if (int.is_empty() && frac.is_empty()) || !all_digits(int) || !all_digits(frac) {
+ return None;
+ }
+ let dscale = u16::try_from(frac.len()).ok()?;
+ let int_pad = (4 - int.len() % 4) % 4;
+ let frac_pad = (4 - frac.len() % 4) % 4;
+ let decimal: Vec = std::iter::repeat_n(b'0', int_pad)
+ .chain(int.bytes())
+ .chain(frac.bytes())
+ .chain(std::iter::repeat_n(b'0', frac_pad))
+ .collect();
+ let digits: Vec = decimal
+ .chunks(4)
+ .map(|g| g.iter().fold(0i16, |acc, b| acc * 10 + i16::from(b - b'0')))
+ .collect();
+ let weight = i32::try_from((int_pad + int.len()) / 4).ok()? - 1;
+ Some(numeric_body(neg, weight, dscale, &digits))
+}
+
+/// Finite value times `10^scale`, fractional digits past `scale` dropped.
+/// `step(m, a)` sets the caller's accumulator to `acc * m + a`, `false` on
+/// overflow
+pub(crate) fn scaled_groups(
+ weight: i32,
+ digits: &[[u8; 2]],
+ scale: u32,
+ mut step: impl FnMut(u32, u32) -> bool,
+) -> bool {
+ let digit = |g: i32| {
+ usize::try_from(g)
+ .ok()
+ .and_then(|i| digits.get(i))
+ .map_or(0, |raw| i16::from_le_bytes(*raw).max(0) as u32)
+ };
+ let whole = (scale / 4) as i32;
+ let rem = scale % 4;
+ for e in (-whole..=weight.max(0)).rev() {
+ if !step(NBASE, digit(weight - e)) {
+ return false;
+ }
+ }
+ if rem == 0 {
+ return true;
+ }
+ let tail = digit(weight + whole + 1) / 10u32.pow(4 - rem);
+ step(10u32.pow(rem), tail)
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -294,86 +397,71 @@ mod tests {
out
}
+ fn text(body: &[u8]) -> Result {
+ let mut out = String::new();
+ render_numeric_text(body, &mut out)?;
+ Ok(out)
+ }
+
#[test]
fn numeric_short_one_digit() {
let body = short_numeric(false, 0, 0, &[42]);
- assert_eq!(
- decode_numeric(&body).unwrap(),
- NumericKind::Finite("42".into())
- );
+ assert_eq!(text(&body).unwrap(), "42");
}
#[test]
fn numeric_short_negative() {
let body = short_numeric(true, 0, 0, &[7]);
- assert_eq!(
- decode_numeric(&body).unwrap(),
- NumericKind::Finite("-7".into())
- );
+ assert_eq!(text(&body).unwrap(), "-7");
}
#[test]
fn numeric_short_with_scale() {
// 1.5: digits base-10000 → [1, 5000]
let body = short_numeric(false, 0, 1, &[1, 5000]);
- assert_eq!(
- decode_numeric(&body).unwrap(),
- NumericKind::Finite("1.5".into())
- );
+ assert_eq!(text(&body).unwrap(), "1.5");
}
#[test]
fn numeric_zero() {
let body = short_numeric(false, 0, 0, &[]);
- assert_eq!(
- decode_numeric(&body).unwrap(),
- NumericKind::Finite("0".into())
- );
+ assert_eq!(text(&body).unwrap(), "0");
}
#[test]
fn numeric_long_form_large() {
// 12345: base-10000 blocks [1, 2345], weight 1
let body = long_numeric(false, 1, 0, &[1, 2345]);
- assert_eq!(
- decode_numeric(&body).unwrap(),
- NumericKind::Finite("12345".into())
- );
+ assert_eq!(text(&body).unwrap(), "12345");
}
#[test]
fn numeric_specials() {
let nan = NUMERIC_NAN.to_le_bytes();
- assert_eq!(decode_numeric(&nan).unwrap(), NumericKind::NaN);
+ assert_eq!(text(&nan).unwrap(), "NaN");
let pinf = NUMERIC_PINF.to_le_bytes();
- assert_eq!(decode_numeric(&pinf).unwrap(), NumericKind::PInf);
+ assert_eq!(text(&pinf).unwrap(), "Infinity");
let ninf = NUMERIC_NINF.to_le_bytes();
- assert_eq!(decode_numeric(&ninf).unwrap(), NumericKind::NInf);
+ assert_eq!(text(&ninf).unwrap(), "-Infinity");
}
#[test]
fn numeric_truncated_returns_error() {
let one_byte = [0u8];
- assert!(matches!(
- decode_numeric(&one_byte),
- Err(CodecError::Truncated { .. })
- ));
+ assert!(matches!(text(&one_byte), Err(CodecError::Truncated { .. })));
}
#[test]
fn numeric_short_negative_weight_renders_leading_zero() {
let body = short_numeric(false, -1, 1, &[5000]);
- assert_eq!(
- decode_numeric(&body).unwrap(),
- NumericKind::Finite("0.5".into())
- );
+ assert_eq!(text(&body).unwrap(), "0.5");
}
#[test]
fn numeric_long_form_truncated_body() {
let body = NUMERIC_POS.to_le_bytes().to_vec();
assert_eq!(
- decode_numeric(&body),
+ text(&body),
Err(CodecError::Truncated {
offset: 2,
need: 4,
@@ -386,16 +474,13 @@ mod tests {
fn numeric_unknown_special_falls_back_to_nan() {
let header: u16 = NUMERIC_SPECIAL | 0x0001;
let body = header.to_le_bytes();
- assert_eq!(decode_numeric(&body).unwrap(), NumericKind::NaN);
+ assert_eq!(text(&body).unwrap(), "NaN");
}
#[test]
fn numeric_dscale_trailing_zero_pad() {
let body = short_numeric(false, 0, 4, &[5]);
- assert_eq!(
- decode_numeric(&body).unwrap(),
- NumericKind::Finite("5.0000".into())
- );
+ assert_eq!(text(&body).unwrap(), "5.0000");
}
#[test]
@@ -415,18 +500,71 @@ mod tests {
NUMERIC_NINF.to_le_bytes().to_vec(),
];
for body in bodies {
- let text = decode_numeric(&body).unwrap();
- assert_eq!(
- numeric_text_len(&body),
- Ok(text.as_text().len()),
- "{text:?}"
- );
- let mut rendered = String::new();
- render_numeric_text(&body, &mut rendered).unwrap();
- assert_eq!(rendered, text.as_text());
+ let rendered = text(&body).unwrap();
+ assert_eq!(numeric_text_len(&body), Ok(rendered.len()), "{rendered:?}");
}
}
+ #[test]
+ fn numeric_from_text_round_trips_canonically() {
+ for (input, want) in [
+ ("0", "0"),
+ ("-0.00", "0.00"),
+ ("12345", "12345"),
+ ("-12.340", "-12.340"),
+ ("0.0005", "0.0005"),
+ ("100000000", "100000000"),
+ (".5", "0.5"),
+ ("NaN", "NaN"),
+ ("-Infinity", "-Infinity"),
+ ] {
+ let body = numeric_from_text(input).unwrap();
+ assert_eq!(text(&body).unwrap(), want, "{input}");
+ }
+ assert_eq!(
+ numeric_from_text("12345").unwrap(),
+ short_numeric(false, 1, 0, &[1, 2345])
+ );
+ assert_eq!(
+ numeric_from_text("1.5").unwrap(),
+ short_numeric(false, 0, 1, &[1, 5000])
+ );
+ assert_eq!(
+ numeric_from_text("0").unwrap(),
+ short_numeric(false, 0, 0, &[])
+ );
+ let wide = format!("1.{}", "0".repeat(70));
+ assert_eq!(
+ numeric_from_text(&wide).unwrap(),
+ long_numeric(false, 0, 70, &[1])
+ );
+ for bad in ["", "-", ".", "1e5", "1.2.3", "abc"] {
+ assert_eq!(numeric_from_text(bad), None, "{bad}");
+ }
+ }
+
+ #[test]
+ fn scaled_groups_truncates_past_scale() {
+ let scaled = |input: &str, scale: u32| {
+ let body = numeric_from_text(input).unwrap();
+ let Numeric::Finite { weight, digits, .. } = parse_numeric(&body).unwrap() else {
+ panic!("finite");
+ };
+ let mut acc = 0u128;
+ assert!(scaled_groups(weight, digits, scale, |m, a| {
+ acc = acc * u128::from(m) + u128::from(a);
+ true
+ }));
+ acc
+ };
+ assert_eq!(scaled("12.34", 2), 1234);
+ assert_eq!(scaled("12.3456789", 3), 12345);
+ assert_eq!(scaled("12", 5), 1_200_000);
+ assert_eq!(scaled("0.0005", 4), 5);
+ assert_eq!(scaled("0.0005", 3), 0);
+ assert_eq!(scaled("123456789.5", 0), 123_456_789);
+ }
+
#[test]
fn numeric_rejects_oversized_digit_array() {
let mut body = NUMERIC_POS.to_le_bytes().to_vec();
@@ -435,9 +573,6 @@ mod tests {
0u8,
((NUMERIC_DSCALE_MASK as usize) + 5) * 2,
));
- assert!(matches!(
- decode_numeric(&body),
- Err(CodecError::BadNumeric { .. })
- ));
+ assert!(matches!(text(&body), Err(CodecError::BadNumeric { .. })));
}
}
diff --git a/src/decode/datum.rs b/src/decode/datum.rs
new file mode 100644
index 00000000..a31b1c80
--- /dev/null
+++ b/src/decode/datum.rs
@@ -0,0 +1,152 @@
+//! Tuple attributes as PostgreSQL stores or sends them. Encoders read source
+//! bytes in place; owned copies hold only bytes that outlive their record
+
+use crate::decode::heap_decoder::ToastPointer;
+use smallvec::SmallVec;
+
+/// One attribute as stored in a tuple image, storage framing stripped
+#[derive(Clone, Copy, Debug, PartialEq)]
+pub enum DatumRef<'a> {
+ /// Not in WAL image: prefix/suffix elision or truncated tail
+ Absent,
+ /// Bitmap bit clear
+ Null,
+ /// Past tuple `natts`, resolves through attribute's fast default
+ Missing,
+ Dropped,
+ /// `typlen > 0` bytes as stored
+ Fixed(&'a [u8]),
+ /// Uncompressed varlena body
+ Varlena(&'a [u8]),
+ /// Inline compressed varlena: 4-byte header, `va_tcinfo`, codec body
+ Compressed(&'a [u8]),
+ External(ToastPointer),
+ /// `typlen = -2` body without trailing NUL
+ Cstring(&'a [u8]),
+ /// Varlena tag PostgreSQL never writes to disk; rest of buffer
+ BadTag(&'a [u8]),
+ /// `typoutput` text: fast defaults read over SQL, COPY fields cast to text
+ Text(&'a [u8]),
+ /// `typsend` bytes of a COPY field, read back through `typreceive`
+ Send(&'a [u8]),
+}
+
+/// Owned [`DatumRef`] for values outliving their record: TOAST values,
+/// synthetic columns, COPY and test cells
+#[derive(Clone, Debug, PartialEq)]
+pub enum Datum {
+ Null,
+ Fixed(SmallVec<[u8; 16]>),
+ Varlena(Vec),
+ External(ToastPointer),
+ Cstring(Vec),
+ BadTag(Vec),
+ Text(Vec),
+ Send(Vec),
+}
+
+impl Datum {
+ pub fn view(&self) -> DatumRef<'_> {
+ match self {
+ Self::Null => DatumRef::Null,
+ Self::Fixed(b) => DatumRef::Fixed(b),
+ Self::Varlena(b) => DatumRef::Varlena(b),
+ Self::External(p) => DatumRef::External(*p),
+ Self::Cstring(b) => DatumRef::Cstring(b),
+ Self::BadTag(b) => DatumRef::BadTag(b),
+ Self::Text(b) => DatumRef::Text(b),
+ Self::Send(b) => DatumRef::Send(b),
+ }
+ }
+
+ /// Payload bytes held, for resident accounting
+ pub fn approx_bytes(&self) -> usize {
+ match self {
+ Self::Null => 0,
+ Self::Fixed(b) => b.len(),
+ Self::External(_) => std::mem::size_of::(),
+ Self::Varlena(b)
+ | Self::Cstring(b)
+ | Self::BadTag(b)
+ | Self::Text(b)
+ | Self::Send(b) => b.len(),
+ }
+ }
+
+ pub fn fixed(le: &[u8]) -> Self {
+ Self::Fixed(SmallVec::from_slice(le))
+ }
+}
+
+/// Cells in their on-disk form, for tests to state tuples
+#[cfg(any(test, feature = "test-support"))]
+pub mod build {
+ use super::Datum;
+
+ pub fn bool_(v: bool) -> Datum {
+ Datum::fixed(&[u8::from(v)])
+ }
+
+ pub fn int2(v: i16) -> Datum {
+ Datum::fixed(&v.to_le_bytes())
+ }
+
+ pub fn int4(v: i32) -> Datum {
+ Datum::fixed(&v.to_le_bytes())
+ }
+
+ pub fn int8(v: i64) -> Datum {
+ Datum::fixed(&v.to_le_bytes())
+ }
+
+ pub fn oid(v: u32) -> Datum {
+ Datum::fixed(&v.to_le_bytes())
+ }
+
+ pub fn oid8(v: u64) -> Datum {
+ Datum::fixed(&v.to_le_bytes())
+ }
+
+ /// Days since 2000-01-01
+ pub fn date(v: i32) -> Datum {
+ Datum::fixed(&v.to_le_bytes())
+ }
+
+ /// `time` or `timestamp[tz]` microseconds
+ pub fn micros(v: i64) -> Datum {
+ Datum::fixed(&v.to_le_bytes())
+ }
+
+ pub fn timetz(micros: i64, tz_seconds: i32) -> Datum {
+ let mut b = micros.to_le_bytes().to_vec();
+ b.extend_from_slice(&tz_seconds.to_le_bytes());
+ Datum::fixed(&b)
+ }
+
+ pub fn uuid(v: [u8; 16]) -> Datum {
+ Datum::fixed(&v)
+ }
+
+ pub fn name(v: impl AsRef) -> Datum {
+ let mut b = v.as_ref().as_bytes().to_vec();
+ b.resize(64, 0);
+ Datum::fixed(&b)
+ }
+
+ pub fn text(v: impl AsRef) -> Datum {
+ Datum::Varlena(v.as_ref().as_bytes().to_vec())
+ }
+
+ pub fn bytes(v: impl AsRef<[u8]>) -> Datum {
+ Datum::Varlena(v.as_ref().to_vec())
+ }
+
+ pub fn numeric(v: impl AsRef) -> Datum {
+ Datum::Varlena(crate::decode::codecs::numeric_from_text(v.as_ref()).expect("numeric text"))
+ }
+
+ /// `typoutput` text
+ pub fn out_text(v: impl AsRef) -> Datum {
+ Datum::Text(v.as_ref().as_bytes().to_vec())
+ }
+}
diff --git a/src/decode/decoder_sink.rs b/src/decode/decoder_sink.rs
index 321d09ac..9d494c89 100644
--- a/src/decode/decoder_sink.rs
+++ b/src/decode/decoder_sink.rs
@@ -1,20 +1,15 @@
-//! Shared types for the heap-tuple decoder fan-out: [`TupleObserver`],
-//! [`DecoderStats`], [`DecoderSinkError`].
-//!
-//! WAL commits drain through the parallel [`pipeline`](crate::emit::pipeline);
-//! [`TupleObserver`] is the greenfield-bootstrap page-walk drain destination
-//! (`drain_backfill`). Catalog gate timeouts poison the stream, so no
+//! Shared types for the heap-tuple decoder fan-out: [`DecoderStats`],
+//! [`DecoderSinkError`]. WAL commits drain through the parallel
+//! [`pipeline`](crate::emit::pipeline). Catalog gate timeouts poison the stream, so no
//! `replay_timeout` counter: silent loss is impossible by construction.
-use std::future::Future;
-use std::pin::Pin;
use std::sync::atomic::{AtomicU64, Ordering};
use thiserror::Error;
use crate::catalog::shadow_catalog::CatalogError;
use crate::ch::EmitterError;
-use crate::decode::heap_decoder::{CommittedTuple, DecodeError};
+use crate::decode::heap_decoder::DecodeError;
use crate::record::SinkError;
use walrus::pg::walparser::{RmId, XLogRecordHeader};
@@ -40,28 +35,6 @@ impl From for DecoderSinkError {
}
}
-/// Destination of decoded + committed heap events. Wire type is
-/// [`CommittedTuple`] not [`DecodedHeap`](crate::decode::heap_decoder::DecodedHeap):
-/// CH emitter needs `commit_ts` for its `_commit_ts` synthetic column.
-///
-/// `on_xact_end` fires after every tuple in a committed xact is delivered.
-/// Returns highest commit_lsn now durable on the observer. Callers advance
-/// their ack ceiling from the return value, NOT `commit_lsn`. Default
-/// returns `commit_lsn` (instant ack).
-pub trait TupleObserver: Send {
- fn on_tuple<'a>(
- &'a mut self,
- committed: &'a CommittedTuple,
- ) -> Pin> + Send + 'a>>;
-
- fn on_xact_end<'a>(
- &'a mut self,
- commit_lsn: u64,
- ) -> Pin> + Send + 'a>> {
- Box::pin(async move { Ok(commit_lsn) })
- }
-}
-
crate::atomic_stats! {
pub struct DecoderStats {
pub decoded,
@@ -168,56 +141,18 @@ pub struct SkipKey {
pub type SkipCounts = crate::emit::pipeline::row_ledger::KeyedCounts;
-#[derive(Debug, Default)]
-pub struct MetricsTupleObserver {
- pub stats: DecoderStats,
-}
-
-impl TupleObserver for MetricsTupleObserver {
- fn on_tuple<'a>(
- &'a mut self,
- committed: &'a CommittedTuple,
- ) -> Pin> + Send + 'a>> {
- Box::pin(async move {
- self.stats.record(&committed.decoded);
- Ok(())
- })
- }
-}
-
-/// In-memory test observer; retains full committed tuples for assertions.
-#[derive(Debug, Default)]
-pub struct CollectingTupleObserver {
- pub tuples: Vec,
-}
-
-impl TupleObserver for CollectingTupleObserver {
- fn on_tuple<'a>(
- &'a mut self,
- committed: &'a CommittedTuple,
- ) -> Pin> + Send + 'a>> {
- Box::pin(async move {
- self.tuples.push(committed.clone());
- Ok(())
- })
- }
-}
-
impl DecoderStats {
- /// Single source of truth for the `HeapOp → counter` mapping; all decoder
- /// dispatch sites route through here.
- pub fn record(&self, decoded: &crate::decode::heap_decoder::DecodedHeap) {
+ pub fn record_op(&self, op: crate::decode::heap_decoder::HeapOp, rows: u16, partial: bool) {
use crate::decode::heap_decoder::HeapOp;
- self.decoded.fetch_add(1, Ordering::Relaxed);
- match decoded.op {
- HeapOp::Insert => self.inserts.fetch_add(1, Ordering::Relaxed),
- HeapOp::Update => self.updates.fetch_add(1, Ordering::Relaxed),
- HeapOp::HotUpdate => self.hot_updates.fetch_add(1, Ordering::Relaxed),
- HeapOp::Delete => self.deletes.fetch_add(1, Ordering::Relaxed),
- HeapOp::Truncate => self.truncates.fetch_add(1, Ordering::Relaxed),
+ let rows = u64::from(rows);
+ self.decoded.fetch_add(rows, Ordering::Relaxed);
+ match op {
+ HeapOp::Insert => self.inserts.fetch_add(rows, Ordering::Relaxed),
+ HeapOp::Update => self.updates.fetch_add(rows, Ordering::Relaxed),
+ HeapOp::HotUpdate => self.hot_updates.fetch_add(rows, Ordering::Relaxed),
+ HeapOp::Delete => self.deletes.fetch_add(rows, Ordering::Relaxed),
+ HeapOp::Truncate => self.truncates.fetch_add(rows, Ordering::Relaxed),
};
- let partial = decoded.new.as_ref().is_some_and(|t| t.partial)
- || decoded.old.as_ref().is_some_and(|t| t.partial);
if partial {
self.partial.fetch_add(1, Ordering::Relaxed);
}
@@ -334,31 +269,11 @@ impl DecoderStats {
#[cfg(test)]
mod tests {
use super::*;
- use crate::decode::heap_decoder::{ColumnValue, DecodedHeap, HeapOp};
- use walrus::pg::walparser::RelFileNode;
-
- fn wrap(decoded: DecodedHeap) -> CommittedTuple {
- CommittedTuple {
- decoded,
- commit_ts: 0,
- commit_lsn: 0,
- }
- }
+ use crate::decode::heap_decoder::HeapOp;
- #[tokio::test(flavor = "current_thread")]
- async fn metrics_observer_buckets_by_op() {
- let mut obs = MetricsTupleObserver::default();
- let mk = |op: HeapOp, partial: bool| DecodedHeap {
- rfn: RelFileNode::default(),
- xid: 0,
- source_lsn: 0,
- op,
- new: Some(crate::decode::heap_decoder::DecodedTuple {
- columns: vec![Some(ColumnValue::Null)],
- partial,
- }),
- old: None,
- };
+ #[test]
+ fn record_op_buckets_by_op() {
+ let s = DecoderStats::default();
for op in [
HeapOp::Insert,
HeapOp::Insert,
@@ -366,10 +281,9 @@ mod tests {
HeapOp::HotUpdate,
HeapOp::Delete,
] {
- obs.on_tuple(&wrap(mk(op, false))).await.unwrap();
+ s.record_op(op, 1, false);
}
- obs.on_tuple(&wrap(mk(HeapOp::Update, true))).await.unwrap();
- let s = &obs.stats;
+ s.record_op(HeapOp::Update, 1, true);
let ld = |a: &AtomicU64| a.load(Ordering::Relaxed);
assert_eq!(ld(&s.decoded), 6);
assert_eq!(ld(&s.inserts), 2);
@@ -379,36 +293,6 @@ mod tests {
assert_eq!(ld(&s.partial), 1);
}
- #[tokio::test(flavor = "current_thread")]
- async fn collecting_observer_keeps_full_clone() {
- let mut obs = CollectingTupleObserver::default();
- let d = DecodedHeap {
- rfn: RelFileNode {
- spc_node: 1663,
- db_node: 5,
- rel_node: 16385,
- },
- xid: 42,
- source_lsn: 0x1234,
- op: HeapOp::Insert,
- new: Some(crate::decode::heap_decoder::DecodedTuple {
- columns: vec![Some(ColumnValue::Int4(7))],
- partial: false,
- }),
- old: None,
- };
- let c = CommittedTuple {
- decoded: d,
- commit_ts: 9_876,
- commit_lsn: 0,
- };
- obs.on_tuple(&c).await.unwrap();
- assert_eq!(obs.tuples.len(), 1);
- assert_eq!(obs.tuples[0].decoded.xid, 42);
- assert_eq!(obs.tuples[0].decoded.source_lsn, 0x1234);
- assert_eq!(obs.tuples[0].commit_ts, 9_876);
- }
-
#[test]
fn stats_summary_skips_zero_buckets() {
let s = DecoderStats::default();
diff --git a/src/decode/heap_decoder.rs b/src/decode/heap_decoder.rs
index 9681f54f..d188f571 100644
--- a/src/decode/heap_decoder.rs
+++ b/src/decode/heap_decoder.rs
@@ -41,7 +41,7 @@
//! | `relreplident` | old payload | this module's behaviour |
//! |---|---|---|
//! | `Full` (`'f'`) | every non-dropped column | decode every column; `old` populated |
-//! | `UsingIndex` (`'i'`) | indexed columns only, others written as NULL via bitmap | decode all; non-indexed columns surface as `Some(ColumnValue::Null)` |
+//! | `UsingIndex` (`'i'`) | indexed columns only, others written as NULL via bitmap | decode all; non-indexed columns surface as `Some(Datum::Null)` |
//! | `Default` (`'d'`) with PK | PK columns when `XLH_UPDATE_CONTAINS_OLD_KEY` set | same as `UsingIndex` shape, with PK attnums populated |
//! | `Default` (`'d'`) no PK | nothing (PG never sets OLD_KEY) | `old = None` |
//! | `Nothing` (`'n'`) | empty | `old = None`; partial = true noted in stats |
@@ -63,11 +63,9 @@
//! downstream by design. Every output stamped with `xid` so xact buffer
//! can key on it.
-use std::borrow::Cow;
-use std::string::FromUtf8Error;
-use std::sync::Arc;
-
use smallvec::{SmallVec, smallvec};
+
+use crate::decode::datum::{Datum, DatumRef};
use thiserror::Error;
use walrus::pg::walparser::{RelFileNode, RmId, XLogRecord};
@@ -83,32 +81,6 @@ use crate::schema::{
/// only MULTI_INSERT with ntuples > 1 spills.
pub type DecodedHeaps = SmallVec<[DecodedHeap; 1]>;
-/// Tuples of one record decoded ahead of the serial worker against the
-/// descriptor resolved then. Decoding is pure in (record, descriptor), so
-/// they stand in for the worker's own decode when its lookup lands on the
-/// same descriptor version
-#[derive(Debug)]
-pub struct Predecoded {
- rel: Arc,
- valid_from: u64,
- set: DecodedHeaps,
-}
-
-impl Predecoded {
- pub fn new(rel: Arc, valid_from: u64, set: DecodedHeaps) -> Self {
- Self {
- rel,
- valid_from,
- set,
- }
- }
-
- /// Tuples, if decoded against `rel` valid from `valid_from`
- pub fn take_if(self, rel: &Arc, valid_from: u64) -> Option {
- (Arc::ptr_eq(&self.rel, rel) && self.valid_from == valid_from).then_some(self.set)
- }
-}
-
/// PG `SizeOfHeapHeader`.
pub const SIZE_OF_HEAP_HEADER: usize = 5;
/// PG `SizeofHeapTupleHeader`, stable at 23 since PG 7.x.
@@ -230,124 +202,6 @@ pub enum DecodeError {
BadPrefixSuffix { need: usize, have: usize },
}
-/// Tier 1/2 decoded value space. Out-of-matrix types surface as
-/// [`ColumnValue::Unsupported`] or [`ColumnValue::ExternalToast`].
-#[derive(Debug, Clone, PartialEq)]
-pub enum ColumnValue {
- Null,
- Bool(bool),
- /// `char` (typname='char'). PG stores as `int8`; surface `i8` to
- /// distinguish from `int2`.
- Char(i8),
- Int2(i16),
- Int4(i32),
- Int8(i64),
- Float4(f32),
- Float8(f64),
- /// `oid`, `xid`, `cid`
- Oid(u32),
- /// `oid8`, `xid8`, `pg_lsn`
- Oid8(u64),
- MacAddr([u8; 6]),
- MacAddr8([u8; 8]),
- Tid {
- block: u32,
- offset: u16,
- },
- /// `date`, days since PG epoch 2000-01-01, negative for pre-epoch.
- Date(i32),
- /// `time`, microseconds since midnight. Legacy float8 storage
- /// (`--disable-integer-datetimes`) removed in PG 10.
- Time(i64),
- /// `timestamp`, microseconds since PG epoch 2000-01-01 00:00:00 UTC.
- Timestamp(i64),
- /// `timestamptz`, same storage as `timestamp`; tz is presentation-only
- /// in PG, bytes identical.
- TimestampTz(i64),
- /// `timetz`, `time` storage + 4-byte UTC offset (seconds, negative for
- /// east of UTC per PG sign convention).
- TimeTz {
- micros: i64,
- tz_seconds: i32,
- },
- /// `uuid`, 16 raw bytes network byte order on disk (PG `uuid_send`, memcpy).
- Uuid([u8; 16]),
- /// `name`, NUL-padded NAMEDATALEN(64) C string, no varlena header;
- /// surface trimmed.
- Name(String),
- Bytea(Vec),
- /// `text`/`varchar`/`bpchar`, varlena + UTF-8 decode. Invalid UTF-8
- /// surfaces as [`ColumnValue::Bytea`].
- Text(String),
- /// `numeric` Tier 2. PG-text form for finite; NaN/Infinity carry their
- /// flag. Emitter maps to CH `String` (PG precision is per-row, no fixed
- /// CH `Decimal` fits without operator config).
- Numeric(crate::decode::codecs::NumericKind),
- /// `inet`/`cidr` Tier 2. Emit via [`crate::decode::codecs::InetValue::to_text`].
- Inet(crate::decode::codecs::InetValue),
- /// `interval` Tier 2, 16-byte fixed-width (months, days, micros).
- Interval(crate::decode::codecs::IntervalValue),
- /// JSON document text: `json` is varlena text on disk and passes
- /// through unchanged, `jsonb` renders through
- /// [`crate::decode::codecs::decode_jsonb`].
- Json(String),
- /// Unhandled on-disk Datum body, varlena header stripped
- PgPending {
- type_oid: u32,
- raw: Vec,
- },
- /// Attribute default text requiring `typinput`, not Datum reconstruction
- PgPendingText {
- type_oid: u32,
- text: String,
- },
- /// On-disk TOAST pointer; xact buffer's TOAST reassembly dereferences.
- ExternalToast(ToastPointer),
- /// Type OID outside decoder matrix, carries raw bytes.
- Unsupported {
- type_oid: u32,
- raw: Vec,
- },
-}
-
-impl ColumnValue {
- /// Approximate in-memory payload bytes, to bound the decode pool's
- /// coalescing buffer. Detoast has already run by route time so
- /// `ExternalToast` pointers are resolved into `Text`/`Bytea`. Exhaustive
- /// so a new variant forces a width.
- pub fn approx_bytes(&self) -> usize {
- use crate::decode::codecs::NumericKind;
- match self {
- ColumnValue::Null => 0,
- ColumnValue::Bool(_) | ColumnValue::Char(_) => 1,
- ColumnValue::Int2(_) => 2,
- ColumnValue::MacAddr(_) | ColumnValue::Tid { .. } => 6,
- ColumnValue::Int4(_)
- | ColumnValue::Float4(_)
- | ColumnValue::Oid(_)
- | ColumnValue::Date(_) => 4,
- ColumnValue::Int8(_)
- | ColumnValue::Float8(_)
- | ColumnValue::Oid8(_)
- | ColumnValue::MacAddr8(_)
- | ColumnValue::Time(_)
- | ColumnValue::Timestamp(_)
- | ColumnValue::TimestampTz(_) => 8,
- ColumnValue::TimeTz { .. } => 12,
- ColumnValue::Uuid(_) => 16,
- ColumnValue::Interval(_) => 16,
- ColumnValue::Inet(_) => 19,
- ColumnValue::Numeric(NumericKind::Finite(s)) => s.len(),
- ColumnValue::Numeric(_) => 0,
- ColumnValue::Name(s) | ColumnValue::Text(s) | ColumnValue::Json(s) => s.len(),
- ColumnValue::Bytea(b) => b.len(),
- ColumnValue::PgPending { raw, .. } | ColumnValue::Unsupported { raw, .. } => raw.len(),
- ColumnValue::PgPendingText { text, .. } => text.len(),
- ColumnValue::ExternalToast(_) => 16,
- }
- }
-}
-
/// On-disk TOAST pointer (PG `varatt.h` `struct varatt_external`), 16 bytes,
/// unaligned in source tuple.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -370,6 +224,7 @@ pub struct DecodedHeap {
pub old: Option,
}
+#[cfg(any(test, feature = "test-support"))]
impl DecodedHeap {
/// Sum across both images (REPLICA IDENTITY FULL keeps a fat `old` too).
pub fn approx_bytes(&self) -> usize {
@@ -378,7 +233,7 @@ impl DecodedHeap {
t.columns
.iter()
.flatten()
- .map(ColumnValue::approx_bytes)
+ .map(Datum::approx_bytes)
.sum::()
})
};
@@ -392,6 +247,7 @@ impl DecodedHeap {
/// between decode and drain can't reinterpret the row. Spill serializes
/// `descriptor_valid_from` instead of the `Arc`; rehydrate re-obtains the
/// descriptor by key and fails closed on span mismatch
+#[cfg(any(test, feature = "test-support"))]
#[derive(Debug, Clone, PartialEq)]
pub struct DescribedHeap {
pub decoded: DecodedHeap,
@@ -400,6 +256,7 @@ pub struct DescribedHeap {
pub descriptor_valid_from: u64,
}
+#[cfg(any(test, feature = "test-support"))]
impl DescribedHeap {
/// Resident accounting counts the retained `Arc` reference (contents
/// shared, not per-row) plus the span key
@@ -424,6 +281,7 @@ pub enum HeapOp {
Truncate,
}
+#[cfg(any(test, feature = "test-support"))]
/// One drained tuple, fully reassembled. `commit_ts` is the commit-record
/// `xact_time` carried through
/// [`crate::xact::xact_buffer::XactBuffer::drain_committed`].
@@ -435,7 +293,7 @@ pub struct CommittedTuple {
/// (decoder unbuffered path).
pub commit_ts: i64,
/// Source LSN of matching `XLOG_XACT_COMMIT`. CH emitter snapshots onto
- /// its ack-LSN gauge once the xact's `send_data(None)` finishes; daemon
+ /// its ack-LSN gauge once the xact's `send_data_end()` finishes; daemon
/// status loop writes it to cursor file `emitter_ack_lsn`. 0 for
/// decoder unbuffered path.
pub commit_lsn: u64,
@@ -445,13 +303,106 @@ pub struct CommittedTuple {
pub struct DecodedTuple {
/// Indexed by attnum-1. `None` = absent from WAL (PG UPDATE prefix/suffix
/// compression skipped it, or this `relreplident` shape never carries it).
- /// `Some(ColumnValue::Null)` is explicit NULL (bitmap bit clear).
- pub columns: Vec>,
+ /// `Some(Datum::Null)` is explicit NULL (bitmap bit clear).
+ pub columns: Vec >,
/// True iff a column was elided by PG prefix/suffix compression; xact
/// buffer backfills from previous tuple image.
pub partial: bool,
}
+/// Builds one tuple image from walked attributes: owned datums for
+/// [`DecodedTuple`], borrowed for [`TupleView`]
+pub trait TupleBuild<'a>: Sized {
+ fn with_capacity(n: usize) -> Self;
+ fn push(&mut self, att: &RelAttr, datum: DatumRef<'a>);
+ fn set_partial(&mut self);
+}
+
+impl<'a> TupleBuild<'a> for DecodedTuple {
+ fn with_capacity(n: usize) -> Self {
+ Self {
+ columns: Vec::with_capacity(n),
+ partial: false,
+ }
+ }
+
+ #[inline(always)]
+ fn push(&mut self, att: &RelAttr, datum: DatumRef<'a>) {
+ self.columns.push(resolve(att, datum));
+ }
+
+ fn set_partial(&mut self) {
+ self.partial = true;
+ }
+}
+
+/// Tuple image borrowing its record. Indexed by attnum-1 like
+/// [`DecodedTuple::columns`]
+#[derive(Clone, Debug, Default, PartialEq)]
+pub struct TupleView<'a> {
+ pub datums: Vec>,
+ pub partial: bool,
+}
+
+impl<'a> TupleBuild<'a> for TupleView<'a> {
+ fn with_capacity(n: usize) -> Self {
+ Self {
+ datums: Vec::with_capacity(n),
+ partial: false,
+ }
+ }
+
+ fn push(&mut self, _: &RelAttr, datum: DatumRef<'a>) {
+ self.datums.push(datum);
+ }
+
+ fn set_partial(&mut self) {
+ self.partial = true;
+ }
+}
+
+#[cfg(test)]
+impl TupleView<'_> {
+ /// Owned datums, as [`decode_heap_record`] builds them
+ pub fn to_tuple(&self, rel: &RelDescriptor) -> DecodedTuple {
+ DecodedTuple {
+ columns: self
+ .datums
+ .iter()
+ .zip(&rel.attributes)
+ .map(|(d, att)| resolve(att, *d))
+ .collect(),
+ partial: self.partial,
+ }
+ }
+}
+
+/// Tuple walked for validity and shape only, building nothing
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
+pub struct TupleShape {
+ pub partial: bool,
+}
+
+impl TupleBuild<'_> for TupleShape {
+ fn with_capacity(_: usize) -> Self {
+ Self::default()
+ }
+
+ fn push(&mut self, _: &RelAttr, _: DatumRef<'_>) {}
+
+ fn set_partial(&mut self) {
+ self.partial = true;
+ }
+}
+
+/// Images of one heap tuple change, before identity and position attach
+#[derive(Clone, Debug, PartialEq)]
+pub struct HeapImages {
+ pub op: HeapOp,
+ pub new: Option,
+ pub old: Option,
+}
+
/// Top-level decode entry point. One-element [`DecodedHeaps`] for
/// INSERT/UPDATE/HOT_UPDATE/DELETE, N-element for `XLOG_HEAP2_MULTI_INSERT`,
/// empty for ops with no tuple payload (LOCK, INPLACE, TRUNCATE, other
@@ -465,26 +416,127 @@ pub fn decode_heap_record(
source_lsn: u64,
rel: &RelDescriptor,
) -> Result {
- let rm = record.header.resource_manager_id;
- let info_op = record.header.info & XLOG_HEAP_OPMASK;
let rfn = record
.blocks
.first()
.map(|b| b.header.location.rel)
.unwrap_or_default();
let xid = record.header.xact_id;
+ Ok(parse_heap_record::(record.into(), rel)?
+ .into_iter()
+ .map(|images| DecodedHeap {
+ rfn,
+ xid,
+ source_lsn,
+ op: images.op,
+ new: images.new,
+ old: images.old,
+ })
+ .collect())
+}
+/// [`decode_heap_record`] borrowing attribute bytes from `record`
+pub fn view_heap_record<'a>(
+ record: impl Into>,
+ rel: &RelDescriptor,
+) -> Result>; 1]>, DecodeError> {
+ parse_heap_record(record.into(), rel)
+}
+
+/// Record fields heap decode reads
+#[derive(Clone, Copy, Debug)]
+pub struct HeapRecordBytes<'a> {
+ pub rm: u8,
+ pub info: u8,
+ pub main_data: &'a [u8],
+ /// Block 0 registered data
+ pub block: Option<&'a [u8]>,
+}
+
+impl<'a> From<&'a XLogRecord<'_>> for HeapRecordBytes<'a> {
+ fn from(record: &'a XLogRecord<'_>) -> Self {
+ Self {
+ rm: record.header.resource_manager_id,
+ info: record.header.info,
+ main_data: &record.main_data,
+ block: record.blocks.first().map(|b| &b.data[..]),
+ }
+ }
+}
+
+/// Heap record summary read from its headers, tuple bytes unchecked
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct RecordShape {
+ pub op: HeapOp,
+ pub tuples: u16,
+ /// PG elided an update's prefix or suffix
+ pub partial: bool,
+}
+
+/// Shape of a record [`view_heap_record`] yields images for, `None` when it
+/// yields none
+pub fn record_shape<'a>(
+ record: impl Into>,
+) -> Result, DecodeError> {
+ let record = record.into();
+ let md = record.main_data;
+ let heap2 = record.rm == RmId::Heap2 as u8;
+ let (op, need) = match record.info & XLOG_HEAP_OPMASK {
+ XLOG_HEAP2_MULTI_INSERT if heap2 => (HeapOp::Insert, SIZE_OF_HEAP_MULTI_INSERT),
+ _ if heap2 || record.rm != RmId::Heap as u8 => return Ok(None),
+ XLOG_HEAP_INSERT => (HeapOp::Insert, 0),
+ XLOG_HEAP_UPDATE => (HeapOp::Update, SIZE_OF_HEAP_UPDATE),
+ XLOG_HEAP_HOT_UPDATE => (HeapOp::HotUpdate, SIZE_OF_HEAP_UPDATE),
+ XLOG_HEAP_DELETE => (HeapOp::Delete, SIZE_OF_HEAP_DELETE),
+ _ => return Ok(None),
+ };
+ if md.len() < need {
+ return Err(DecodeError::Truncated {
+ offset: 0,
+ need,
+ have: md.len(),
+ });
+ }
+ let mut shape = RecordShape {
+ op,
+ tuples: 1,
+ partial: false,
+ };
+ if heap2 {
+ shape.tuples = u16::from_le_bytes([md[2], md[3]]);
+ if shape.tuples == 0 {
+ return Err(DecodeError::Truncated {
+ offset: 0,
+ need: 1,
+ have: 0,
+ });
+ }
+ if md[0] & XLH_INSERT_CONTAINS_NEW_TUPLE == 0 {
+ return Ok(None);
+ }
+ } else if matches!(op, HeapOp::Update | HeapOp::HotUpdate) {
+ shape.partial = md[7] & (XLH_UPDATE_PREFIX_FROM_OLD | XLH_UPDATE_SUFFIX_FROM_OLD) != 0;
+ }
+ Ok(Some(shape))
+}
+
+fn parse_heap_record<'a, T: TupleBuild<'a>>(
+ record: HeapRecordBytes<'a>,
+ rel: &RelDescriptor,
+) -> Result; 1]>, DecodeError> {
+ let rm = record.rm;
+ let info_op = record.info & XLOG_HEAP_OPMASK;
if rm == RmId::Heap2 as u8 && info_op == XLOG_HEAP2_MULTI_INSERT {
- return decode_multi_insert(record, source_lsn, rfn, xid, rel);
+ return decode_multi_insert(record, rel);
}
if rm != RmId::Heap as u8 {
return Ok(SmallVec::new());
}
let op = match info_op {
- XLOG_HEAP_INSERT => decode_insert(record, source_lsn, rfn, xid, rel)?,
- XLOG_HEAP_UPDATE => decode_update(record, source_lsn, rfn, xid, rel, false)?,
- XLOG_HEAP_HOT_UPDATE => decode_update(record, source_lsn, rfn, xid, rel, true)?,
- XLOG_HEAP_DELETE => decode_delete(record, source_lsn, rfn, xid, rel)?,
+ XLOG_HEAP_INSERT => decode_insert(record, rel)?,
+ XLOG_HEAP_UPDATE => decode_update(record, rel, false)?,
+ XLOG_HEAP_HOT_UPDATE => decode_update(record, rel, true)?,
+ XLOG_HEAP_DELETE => decode_delete(record, rel)?,
_ => return Ok(SmallVec::new()),
};
Ok(smallvec![op])
@@ -501,14 +553,11 @@ pub const SIZE_OF_MULTI_INSERT_TUPLE: usize = 7;
/// sequential `FirstOffsetNumber + i`), PG `heapam.c::heap_multi_insert`.
pub const XLOG_HEAP_INIT_PAGE: u8 = 0x80;
-fn decode_multi_insert(
- record: &XLogRecord,
- source_lsn: u64,
- rfn: RelFileNode,
- xid: u32,
+fn decode_multi_insert<'a, T: TupleBuild<'a>>(
+ record: HeapRecordBytes<'a>,
rel: &RelDescriptor,
-) -> Result {
- let md = &record.main_data;
+) -> Result; 1]>, DecodeError> {
+ let md = record.main_data;
if md.len() < SIZE_OF_HEAP_MULTI_INSERT {
return Err(DecodeError::Truncated {
offset: 0,
@@ -532,7 +581,7 @@ fn decode_multi_insert(
if flags & XLH_INSERT_CONTAINS_NEW_TUPLE == 0 {
return Ok(SmallVec::new());
}
- let init_page = record.header.info & XLOG_HEAP_INIT_PAGE != 0;
+ let init_page = record.info & XLOG_HEAP_INIT_PAGE != 0;
// Non-init_page: main_data continues with offsets[ntuples] of u16.
// Values unneeded (block 0 carries tuples in order); length-check only
let expected_md_min = if init_page {
@@ -548,12 +597,11 @@ fn decode_multi_insert(
});
}
- let Some(block) = record.blocks.first() else {
+ let Some(data) = record.block else {
return Ok(SmallVec::new());
};
- let data = &block.data;
let mut cursor = 0usize;
- let mut out: DecodedHeaps = SmallVec::with_capacity(ntuples);
+ let mut out = SmallVec::with_capacity(ntuples);
for _ in 0..ntuples {
// Each xl_multi_insert_tuple SHORTALIGN'd (2-byte) at write time,
// PG `heap_xlog_multi_insert`
@@ -582,12 +630,10 @@ fn decode_multi_insert(
have: data.len().saturating_sub(cursor),
});
}
- let tup = walk_tuple(&data[..cursor + datalen], header, cursor, rel, 0)?;
+ let mut tup = TupleBuild::with_capacity(rel.attributes.len());
+ walk_tuple(&data[..cursor + datalen], header, cursor, rel, 0, &mut tup)?;
cursor += datalen;
- out.push(DecodedHeap {
- rfn,
- xid,
- source_lsn,
+ out.push(HeapImages {
op: HeapOp::Insert,
new: Some(tup),
old: None,
@@ -596,44 +642,28 @@ fn decode_multi_insert(
Ok(out)
}
-fn decode_insert(
- record: &XLogRecord,
- source_lsn: u64,
- rfn: RelFileNode,
- xid: u32,
+fn decode_insert<'a, T: TupleBuild<'a>>(
+ record: HeapRecordBytes<'a>,
rel: &RelDescriptor,
-) -> Result {
- let Some(block) = record.blocks.first() else {
- // PG always references target page; emit empty new rather than
- // fail the stream
- return Ok(DecodedHeap {
- rfn,
- xid,
- source_lsn,
- op: HeapOp::Insert,
- new: None,
- old: None,
- });
+) -> Result, DecodeError> {
+ // PG always references target page; emit empty new rather than fail
+ // the stream
+ let new = match record.block {
+ Some(data) => Some(decode_tuple_payload(data, 0, rel, 0)?),
+ None => None,
};
- let new = decode_new_tuple_block(&block.data, 0, rel)?;
- Ok(DecodedHeap {
- rfn,
- xid,
- source_lsn,
+ Ok(HeapImages {
op: HeapOp::Insert,
- new: Some(new),
+ new,
old: None,
})
}
-fn decode_update(
- record: &XLogRecord,
- source_lsn: u64,
- rfn: RelFileNode,
- xid: u32,
+fn decode_update<'a, T: TupleBuild<'a>>(
+ record: HeapRecordBytes<'a>,
rel: &RelDescriptor,
hot: bool,
-) -> Result {
+) -> Result, DecodeError> {
if record.main_data.len() < SIZE_OF_HEAP_UPDATE {
return Err(DecodeError::Truncated {
offset: 0,
@@ -647,20 +677,20 @@ fn decode_update(
let has_old_tuple = flags & XLH_UPDATE_CONTAINS_OLD_TUPLE != 0;
let has_old_key = flags & XLH_UPDATE_CONTAINS_OLD_KEY != 0;
- let new = if let Some(b) = record.blocks.first() {
+ let new = if let Some(data) = record.block {
let mut cursor = 0;
let prefixlen = if has_prefix {
- read_u16(&b.data, &mut cursor)? as usize
+ read_u16(data, &mut cursor)? as usize
} else {
0
};
- // Suffix bytes lie past WAL slice, so its end already bounds the walk
+ // Suffix bytes are past the WAL slice, so its length goes unread
if has_suffix {
- read_u16(&b.data, &mut cursor)?;
+ read_u16(data, &mut cursor)?;
}
- let mut t = decode_tuple_payload(&b.data, cursor, rel, prefixlen)?;
+ let mut t: T = decode_tuple_payload(data, cursor, rel, prefixlen)?;
if has_prefix || has_suffix {
- t.partial = true;
+ t.set_partial();
}
Some(t)
} else {
@@ -668,7 +698,7 @@ fn decode_update(
};
let old = if has_old_tuple || has_old_key {
- decode_old_tuple_from_main_data(&record.main_data, SIZE_OF_HEAP_UPDATE, rel)?
+ decode_old_tuple_from_main_data(record.main_data, SIZE_OF_HEAP_UPDATE, rel)?
} else {
None
};
@@ -678,23 +708,13 @@ fn decode_update(
} else {
HeapOp::Update
};
- Ok(DecodedHeap {
- rfn,
- xid,
- source_lsn,
- op,
- new,
- old,
- })
+ Ok(HeapImages { op, new, old })
}
-fn decode_delete(
- record: &XLogRecord,
- source_lsn: u64,
- rfn: RelFileNode,
- xid: u32,
+fn decode_delete<'a, T: TupleBuild<'a>>(
+ record: HeapRecordBytes<'a>,
rel: &RelDescriptor,
-) -> Result {
+) -> Result, DecodeError> {
if record.main_data.len() < SIZE_OF_HEAP_DELETE {
return Err(DecodeError::Truncated {
offset: 0,
@@ -705,48 +725,27 @@ fn decode_delete(
let flags = record.main_data[7]; // xl_heap_delete.flags (after xmax:4 + offnum:2 + infobits:1)
let has_old = flags & (XLH_DELETE_CONTAINS_OLD_TUPLE | XLH_DELETE_CONTAINS_OLD_KEY) != 0;
let old = if has_old {
- decode_old_tuple_from_main_data(&record.main_data, SIZE_OF_HEAP_DELETE, rel)?
+ decode_old_tuple_from_main_data(record.main_data, SIZE_OF_HEAP_DELETE, rel)?
} else {
None
};
- Ok(DecodedHeap {
- rfn,
- xid,
- source_lsn,
+ Ok(HeapImages {
op: HeapOp::Delete,
new: None,
old,
})
}
-/// Block-0 data shape: `xl_heap_header (5)` + bitmap[+pad] + col data.
-fn decode_new_tuple_block(
- data: &[u8],
- data_start: usize,
- rel: &RelDescriptor,
-) -> Result {
- decode_tuple_payload(data, data_start, rel, 0)
-}
-
-fn decode_old_tuple_from_main_data(
- main_data: &[u8],
+fn decode_old_tuple_from_main_data<'a, T: TupleBuild<'a>>(
+ main_data: &'a [u8],
header_off: usize,
rel: &RelDescriptor,
-) -> Result, DecodeError> {
- if main_data.len() < header_off + SIZE_OF_HEAP_HEADER {
- return Err(DecodeError::Truncated {
- offset: header_off,
- need: SIZE_OF_HEAP_HEADER,
- have: main_data.len() - header_off,
- });
- }
+) -> Result , DecodeError> {
Ok(Some(decode_tuple_payload(main_data, header_off, rel, 0)?))
}
-/// Decode a `block.data`-shaped tuple for the bootstrap page-walk sink,
-/// which reshapes on-disk `HeapTupleHeaderData`-prefixed tuples into this
-/// form. `prefixlen=0`: WAL prefix/suffix-from-old compression
-/// doesn't apply at backup time.
+/// Owned cells of a `block.data`-shaped tuple
+#[cfg(any(test, feature = "test-support"))]
pub(crate) fn decode_block_data(
block_data: &[u8],
rel: &RelDescriptor,
@@ -754,6 +753,96 @@ pub(crate) fn decode_block_data(
decode_tuple_payload(block_data, 0, rel, 0)
}
+/// Tuple of a `block.data`-shaped buffer, borrowing attribute bytes
+pub fn view_block_data<'a>(
+ block_data: &'a [u8],
+ rel: &RelDescriptor,
+) -> Result, DecodeError> {
+ decode_tuple_payload(block_data, 0, rel, 0)
+}
+
+/// [`view_block_data`] refilling `out`, sparing a bulk loop the return copy
+pub fn view_block_data_into<'a>(
+ block_data: &'a [u8],
+ rel: &RelDescriptor,
+ out: &mut TupleView<'a>,
+) -> Result<(), DecodeError> {
+ out.datums.clear();
+ out.partial = false;
+ decode_tuple_payload_into(block_data, 0, rel, 0, out)
+}
+
+/// Validate a `block.data`-shaped buffer without building attributes
+pub fn scan_block_data(block_data: &[u8], rel: &RelDescriptor) -> Result {
+ decode_tuple_payload(block_data, 0, rel, 0)
+}
+
+/// `block.data` image of `values` as `heap_fill_tuple` lays them out,
+/// aligned per `attrs`. Values past them align by width
+#[cfg(any(test, feature = "test-support"))]
+pub fn form_block_data(attrs: &[RelAttr], values: &[Option]) -> Vec {
+ let natts = values.len();
+ let has_null = values.iter().any(|v| matches!(v, None | Some(Datum::Null)));
+ let bitmap_len = if has_null { natts.div_ceil(8) } else { 0 };
+ let hoff = (SIZE_OF_HEAP_TUPLE_HEADER + bitmap_len).next_multiple_of(8);
+ let mut bitmap = vec![0u8; bitmap_len];
+ let mut data = Vec::new();
+ for (i, v) in values.iter().enumerate() {
+ let align_of = |len: usize| {
+ attrs
+ .get(i)
+ .map_or(if len.is_power_of_two() { len.min(8) } else { 1 }, |a| {
+ align_bytes(a.type_align)
+ })
+ };
+ let (bytes, align) = match v {
+ None | Some(Datum::Null) => continue,
+ Some(Datum::Fixed(b)) => (b.to_vec(), align_of(b.len())),
+ Some(Datum::Varlena(b) | Datum::BadTag(b) | Datum::Text(b) | Datum::Send(b))
+ if b.len() < 127 =>
+ {
+ let mut v = vec![(((b.len() + 1) << 1) | 1) as u8];
+ v.extend_from_slice(b);
+ (v, 1)
+ }
+ Some(Datum::Varlena(b) | Datum::BadTag(b) | Datum::Text(b) | Datum::Send(b)) => {
+ let mut v = (((b.len() + 4) as u32) << 2).to_le_bytes().to_vec();
+ v.extend_from_slice(b);
+ (v, align_of(4).max(4))
+ }
+ Some(Datum::Cstring(b)) => {
+ let mut v = b.clone();
+ v.push(0);
+ (v, 1)
+ }
+ Some(Datum::External(p)) => {
+ let mut v = vec![0x01, 18];
+ for word in [
+ p.va_rawsize as u32,
+ p.va_extinfo,
+ p.va_valueid,
+ p.va_toastrelid,
+ ] {
+ v.extend_from_slice(&word.to_le_bytes());
+ }
+ (v, 1)
+ }
+ };
+ if has_null {
+ bitmap[i / 8] |= 1 << (i % 8);
+ }
+ data.resize(data.len().next_multiple_of(align), 0);
+ data.extend_from_slice(&bytes);
+ }
+ let mut out = (natts as u16).to_le_bytes().to_vec();
+ out.extend_from_slice(&if has_null { HEAP_HASNULL } else { 0 }.to_le_bytes());
+ out.push(hoff as u8);
+ out.extend_from_slice(&bitmap);
+ out.resize(SIZE_OF_HEAP_HEADER + hoff - SIZE_OF_HEAP_TUPLE_HEADER, 0);
+ out.extend_from_slice(&data);
+ out
+}
+
/// Fields PG strips into `xl_heap_header` or `xl_multi_insert_tuple`
#[derive(Clone, Copy)]
struct TupleHeader {
@@ -767,15 +856,28 @@ struct TupleHeader {
/// Walk a tuple payload (`xl_heap_header` at `header_off`, then bitmap,
/// padding, col data). `prefixlen` is PG `XLH_UPDATE_PREFIX_FROM_OLD` byte
/// count elided from front of logical col-data region; columns wholly inside
-/// it or past end of `buf` surface `None` and caller flips `partial`.
+/// it or past end of `buf` surface absent and caller flips `partial`.
/// Bitmap + xl_heap_header precede prefix elision so are always present at
/// `header_off`.
-fn decode_tuple_payload(
- buf: &[u8],
+#[inline]
+fn decode_tuple_payload<'a, T: TupleBuild<'a>>(
+ buf: &'a [u8],
header_off: usize,
rel: &RelDescriptor,
prefixlen: usize,
-) -> Result {
+) -> Result {
+ let mut out = T::with_capacity(rel.attributes.len());
+ decode_tuple_payload_into(buf, header_off, rel, prefixlen, &mut out)?;
+ Ok(out)
+}
+
+fn decode_tuple_payload_into<'a, T: TupleBuild<'a>>(
+ buf: &'a [u8],
+ header_off: usize,
+ rel: &RelDescriptor,
+ prefixlen: usize,
+ out: &mut T,
+) -> Result<(), DecodeError> {
if buf.len() < header_off + SIZE_OF_HEAP_HEADER {
return Err(DecodeError::Truncated {
offset: header_off,
@@ -795,18 +897,20 @@ fn decode_tuple_payload(
header_off + SIZE_OF_HEAP_HEADER,
rel,
prefixlen,
+ out,
)
}
-/// Walk bitmap, padding and col data starting at `bitmap_off`, one column per
+/// Walk bitmap, padding and col data starting at `bitmap_off`, one datum per
/// attribute of `rel`. `buf` ends where tuple ends
-fn walk_tuple(
- buf: &[u8],
+fn walk_tuple<'a, T: TupleBuild<'a>>(
+ buf: &'a [u8],
header: TupleHeader,
bitmap_off: usize,
rel: &RelDescriptor,
prefixlen: usize,
-) -> Result {
+ out: &mut T,
+) -> Result<(), DecodeError> {
let t_hoff = usize::from(header.hoff);
if t_hoff < SIZE_OF_HEAP_TUPLE_HEADER {
return Err(DecodeError::BadHoff {
@@ -833,13 +937,19 @@ fn walk_tuple(
// carried verbatim in WAL
let col_data_off = bitmap_off + (t_hoff - SIZE_OF_HEAP_TUPLE_HEADER);
- let mut columns = Vec::with_capacity(rel.attributes.len());
+ let attrs = &rel.attributes;
+ // Mark this + every later col absent
+ let absent_from = |out: &mut T, idx: usize| {
+ for att in &attrs[idx..] {
+ out.push(att, DatumRef::Absent);
+ }
+ out.set_partial();
+ };
// `cur` is offset within the *logical* col-data region (relative to
// t_hoff); align matches PG `att_align_nominal` against this offset. t_hoff
// is MAXALIGN'd(8) so col 1 starts clean. WAL offset of logical `cur` is
// `col_data_off + (cur - prefixlen)` when `cur >= prefixlen`
let mut cur: usize = 0;
- let attrs = effective_attrs(rel, natts);
let wal_col_data_avail = buf.len().saturating_sub(col_data_off);
let logical_end = wal_col_data_avail + prefixlen;
for (idx, att) in attrs.iter().enumerate() {
@@ -847,11 +957,11 @@ fn walk_tuple(
// wal natts < catalog natts: trailing cols added via ALTER ADD
// COLUMN after the record. Mirror PG `getmissingattr`
// (heaptuple.c) `attmissingval[1]` fast-path default
- columns.push(Some(missing_value_for(att)));
+ out.push(att, DatumRef::Missing);
continue;
}
if has_null && !bitmap_is_set(bitmap, idx) {
- columns.push(Some(ColumnValue::Null));
+ out.push(att, DatumRef::Null);
continue;
}
let col_size_hint = att.type_len;
@@ -871,46 +981,16 @@ fn walk_tuple(
cur = align_for(cur, att.type_align);
}
- if cur < prefixlen {
- let len = decoded_size_or_skip(att);
- match len {
- Some(n) => {
- columns.push(None);
- cur += n;
- }
- None => {
- // Varlena in prefix: can't advance cur without the bytes.
- // Mark this + every later col absent, bail; xact buffer
- // reconstructs
- columns.push(None);
- for _ in (idx + 1)..attrs.len() {
- columns.push(None);
- }
- return Ok(DecodedTuple {
- columns,
- partial: true,
- });
- }
- }
- continue;
- }
-
- if cur >= logical_end {
- // In suffix / past EOF
- columns.push(None);
- // Advance cur for fixed-width accounting only; all emit None
- if let Some(n) = decoded_size_or_skip(att) {
- cur += n;
- } else {
- // Varlena tail; remaining cols stay absent
- for _ in (idx + 1)..attrs.len() {
- columns.push(None);
- }
- return Ok(DecodedTuple {
- columns,
- partial: true,
- });
- }
+ if cur < prefixlen || cur >= logical_end {
+ // In prefix, or suffix / past EOF. Advance cur for fixed-width
+ // accounting only; varlena width needs its bytes so later cols
+ // stay absent too
+ let Some(n) = decoded_size_or_skip(att) else {
+ absent_from(out, idx);
+ return Ok(());
+ };
+ out.push(att, DatumRef::Absent);
+ cur += n;
continue;
}
@@ -918,135 +998,139 @@ fn walk_tuple(
if att.type_len > 0 && !att.dropped {
let len = att.type_len as usize;
if let Some(body) = buf.get(abs..abs + len) {
- columns.push(Some(fixed_value(att, body)));
+ out.push(att, DatumRef::Fixed(body));
cur += len;
continue;
}
}
- let Ok((value, consumed)) = decode_one_value(att, buf, abs) else {
- columns.push(None);
- for _ in (idx + 1)..attrs.len() {
- columns.push(None);
- }
- return Ok(DecodedTuple {
- columns,
- partial: true,
- });
+ let Ok((datum, consumed)) = read_datum(att, buf, abs) else {
+ absent_from(out, idx);
+ return Ok(());
};
- columns.push(Some(value));
+ out.push(att, datum);
cur += consumed;
}
- Ok(DecodedTuple {
- columns,
- partial: false,
- })
+ Ok(())
}
-/// Resolve a column's fast default ([`RelAttr::missing_default`]) to a value;
-/// `Null` when there is none. Tier-3 stays a pending value the oracle resolves.
-pub fn missing_value_for(att: &RelAttr) -> ColumnValue {
+/// Column's fast default ([`RelAttr::missing_default`]) as stored or as
+/// `typoutput` text; `Null` when there is none
+pub fn missing_datum(att: &RelAttr) -> DatumRef<'_> {
match &att.missing_default {
- Some(MissingDefault::Raw(arr)) => missing_value_from_array(att, arr),
- Some(MissingDefault::Text(text)) => missing_value_from_text(att.type_oid, text),
- None => ColumnValue::Null,
+ Some(MissingDefault::Raw(arr)) => missing_array_element(att, arr),
+ Some(MissingDefault::Text(text)) => DatumRef::Text(text.as_bytes()),
+ None => DatumRef::Null,
}
}
-/// Raw and text spell one default differently but resolve to one value, so
-/// compare by value; tier-3 stays pending, so equal source type counts as equal.
+/// Raw and text spell one default differently, so compare parsed datums.
+/// Text no local parser reads stays pending; equal source type counts as equal
pub fn missing_defaults_equivalent(a: &RelAttr, b: &RelAttr) -> bool {
- use ColumnValue::{PgPending, PgPendingText};
- match (missing_value_for(a), missing_value_for(b)) {
- (
- PgPending { type_oid: x, .. } | PgPendingText { type_oid: x, .. },
- PgPending { type_oid: y, .. } | PgPendingText { type_oid: y, .. },
- ) => x == y,
- (l, r) => l == r,
- }
+ #[derive(PartialEq)]
+ enum Image {
+ Value(Option),
+ Pending(u32),
+ }
+ let image = |att: &RelAttr| match missing_datum(att) {
+ DatumRef::Text(text) => match text_datum(att.type_oid, text) {
+ TextDatum::Datum(d) => Image::Value(Some(d)),
+ TextDatum::Null => Image::Value(Some(Datum::Null)),
+ TextDatum::Opaque if is_text_type(att.type_oid) => {
+ Image::Value(Some(Datum::Varlena(text.to_vec())))
+ }
+ TextDatum::Opaque => Image::Pending(att.type_oid),
+ },
+ // Text-typed defaults compare as their text
+ d if local_matrix_covers(att.type_oid, att.type_len) => {
+ Image::Value(resolve(att, d).map(|d| {
+ match (d, att.type_oid) {
+ (Datum::Fixed(b), NAMEOID) => {
+ let end = b.iter().position(|&c| c == 0).unwrap_or(b.len());
+ Datum::Varlena(b[..end].to_vec())
+ }
+ (Datum::Varlena(b), JSONBOID) => crate::decode::codecs::decode_jsonb(&b)
+ .map_or(Datum::BadTag(b), |text| Datum::Varlena(text.into_bytes())),
+ (d, _) => d,
+ }
+ }))
+ }
+ DatumRef::Null => Image::Value(Some(Datum::Null)),
+ _ => Image::Pending(att.type_oid),
+ };
+ image(a) == image(b)
}
-fn missing_value_from_array(att: &RelAttr, arr: &[u8]) -> ColumnValue {
+/// `attmissingval[1]`, an `ArrayType` with one non-null element
+fn missing_array_element<'a>(att: &RelAttr, arr: &'a [u8]) -> DatumRef<'a> {
// ArrayType: vl_len_(4) ndim(4) dataoffset(4) elemtype(4) dims[] lbounds[] [nulls] data
- let decode = || -> Option {
- if arr.len() < 16 {
- return None;
- }
- let ndim = i32::from_le_bytes(arr[4..8].try_into().ok()?);
- let dataoffset = i32::from_le_bytes(arr[8..12].try_into().ok()?);
+ let element = || -> Option> {
+ let ndim = i32::from_le_bytes(arr.get(4..8)?.try_into().ok()?);
+ let dataoffset = i32::from_le_bytes(arr.get(8..12)?.try_into().ok()?);
if ndim != 1 {
return None;
}
let abs = if dataoffset != 0 {
- dataoffset as usize
+ usize::try_from(dataoffset).ok()?
} else {
// MAXALIGN(16 + 8*ndim), no null bitmap
(16 + 8 * ndim as usize).next_multiple_of(8)
};
- decode_one_value(att, arr, abs).ok().map(|(v, _)| v)
+ read_datum(att, arr, abs).ok().map(|(d, _)| d)
};
- decode().unwrap_or(ColumnValue::Null)
+ element().unwrap_or(DatumRef::Null)
+}
+
+fn is_text_type(type_oid: u32) -> bool {
+ matches!(
+ type_oid,
+ TEXTOID | VARCHAROID | BPCHAROID | NAMEOID | JSONOID | JSONBOID
+ )
+}
+
+/// `typoutput` text read back into a stored datum
+#[derive(Debug, PartialEq)]
+pub enum TextDatum {
+ Datum(Datum),
+ /// Text the type's parser rejects
+ Null,
+ /// No local parser; String targets take the text as is
+ Opaque,
}
-/// `attmissingval` element from its `anyarray_out` text form (SQL/mirror path).
-pub(crate) fn missing_value_from_text(type_oid: u32, text: &str) -> ColumnValue {
+/// Parse `typoutput` text of types with a local reader into stored form
+pub fn text_datum(type_oid: u32, text: &[u8]) -> TextDatum {
+ let Ok(text) = simdutf8::basic::from_utf8(text) else {
+ return TextDatum::Opaque;
+ };
+ fn fixed>(b: Option) -> TextDatum {
+ b.map_or(TextDatum::Null, |b| {
+ TextDatum::Datum(Datum::fixed(b.as_ref()))
+ })
+ }
match type_oid {
- BOOLOID => match text {
- "t" | "true" | "yes" | "on" | "1" => ColumnValue::Bool(true),
- _ => ColumnValue::Bool(false),
- },
- CHAROID => text
- .as_bytes()
- .first()
- .map(|b| ColumnValue::Char(*b as i8))
- .unwrap_or(ColumnValue::Null),
- INT2OID => text
- .parse::()
- .map(ColumnValue::Int2)
- .unwrap_or(ColumnValue::Null),
- INT4OID => text
- .parse::()
- .map(ColumnValue::Int4)
- .unwrap_or(ColumnValue::Null),
- INT8OID => text
- .parse::()
- .map(ColumnValue::Int8)
- .unwrap_or(ColumnValue::Null),
- FLOAT4OID => text
- .parse::()
- .map(ColumnValue::Float4)
- .unwrap_or(ColumnValue::Null),
- FLOAT8OID => text
- .parse::()
- .map(ColumnValue::Float8)
- .unwrap_or(ColumnValue::Null),
- OIDOID | XIDOID | CIDOID => text
- .parse::()
- .map(ColumnValue::Oid)
- .unwrap_or(ColumnValue::Null),
- OID8OID | XID8OID => text
- .parse::()
- .map(ColumnValue::Oid8)
- .unwrap_or(ColumnValue::Null),
- PG_LSNOID => crate::pg::parse_pg_lsn(text)
- .map(ColumnValue::Oid8)
- .unwrap_or(ColumnValue::Null),
- TEXTOID | VARCHAROID | BPCHAROID | NAMEOID => ColumnValue::Text(text.to_owned()),
- MACADDROID => crate::decode::codecs::parse_macaddr(text)
- .map(ColumnValue::MacAddr)
- .unwrap_or(ColumnValue::Null),
- MACADDR8OID => crate::decode::codecs::parse_macaddr(text)
- .map(ColumnValue::MacAddr8)
- .unwrap_or(ColumnValue::Null),
- TIDOID => crate::decode::codecs::parse_tid(text)
- .map(|(block, offset)| ColumnValue::Tid { block, offset })
- .unwrap_or(ColumnValue::Null),
- // `jsonb` arrives as `jsonb_out` text, already what the emitter ships
- JSONOID | JSONBOID => ColumnValue::Json(text.to_owned()),
- // attmissingval text requires typinput
- _ => ColumnValue::PgPendingText {
- type_oid,
- text: text.to_owned(),
- },
+ BOOLOID => fixed(Some([u8::from(matches!(
+ text,
+ "t" | "true" | "yes" | "on" | "1"
+ ))])),
+ CHAROID => fixed(text.as_bytes().first().map(|b| [*b])),
+ INT2OID => fixed(text.parse::().ok().map(i16::to_le_bytes)),
+ INT4OID => fixed(text.parse::().ok().map(i32::to_le_bytes)),
+ INT8OID => fixed(text.parse::().ok().map(i64::to_le_bytes)),
+ FLOAT4OID => fixed(text.parse::().ok().map(f32::to_le_bytes)),
+ FLOAT8OID => fixed(text.parse::().ok().map(f64::to_le_bytes)),
+ OIDOID | XIDOID | CIDOID => fixed(text.parse::().ok().map(u32::to_le_bytes)),
+ OID8OID | XID8OID => fixed(text.parse::().ok().map(u64::to_le_bytes)),
+ PG_LSNOID => fixed(crate::pg::parse_pg_lsn(text).ok().map(u64::to_le_bytes)),
+ MACADDROID => fixed(crate::decode::codecs::parse_macaddr::<6>(text)),
+ MACADDR8OID => fixed(crate::decode::codecs::parse_macaddr::<8>(text)),
+ TIDOID => fixed(
+ crate::decode::codecs::parse_tid(text)
+ .map(|(block, offset)| crate::decode::codecs::encode_tid(block, offset)),
+ ),
+ // TODO(0.2.0): remove once 0.1.x descriptor logs migrate
+ NUMERICOID => crate::decode::codecs::numeric_from_text(text)
+ .map_or(TextDatum::Null, |b| TextDatum::Datum(Datum::Varlena(b))),
+ _ => TextDatum::Opaque,
}
}
@@ -1062,6 +1146,16 @@ fn decoded_size_or_skip(att: &RelAttr) -> Option {
/// MAXALIGN dispatch over `pg_type.typalign`, for in-prefix / past-EOF where
/// we can't peek to detect a short varlena header (`att_align_pointer`).
+#[cfg(any(test, feature = "test-support"))]
+fn align_bytes(attalign: char) -> usize {
+ match attalign {
+ 's' => 2,
+ 'i' => 4,
+ 'd' => 8,
+ _ => 1,
+ }
+}
+
fn align_for(cur: usize, attalign: char) -> usize {
match attalign {
's' => align_up(cur, 2),
@@ -1072,14 +1166,6 @@ fn align_for(cur: usize, attalign: char) -> usize {
}
}
-/// Always `rel.attributes`. Replica-identity-only writes (UsingIndex/Default)
-/// carry NULL placeholders for non-indexed cols, so wire natts equals rel
-/// natts. Post-ALTER-ADD-COLUMN trailing attrs (`natts_in_wal < natts`)
-/// surface via PG missing-column handling.
-fn effective_attrs(rel: &RelDescriptor, _natts: usize) -> &[RelAttr] {
- &rel.attributes
-}
-
/// PG `att_align_nominal` (tupmacs.h). For varlena, PG `att_align_pointer`
/// skips alignment when next byte is a 1-byte short-header datum
/// (`!VARATT_NOT_PAD_BYTE`); mirror by peeking `buf[abs]` before aligning.
@@ -1127,116 +1213,73 @@ fn read_u16(buf: &[u8], cur: &mut usize) -> Result {
Ok(v)
}
-/// Returns `(value, bytes_consumed)`. Varlena reports total on-disk length
-/// (header + body, including TOAST pointer's full 18 bytes).
-pub(crate) fn decode_one_value(
+/// Read one attribute at `abs`, returning datum and on-disk length. Varlena
+/// reports header + body, including TOAST pointer's full 18 bytes
+pub(crate) fn read_datum<'a>(
att: &RelAttr,
- buf: &[u8],
+ buf: &'a [u8],
abs: usize,
-) -> Result<(ColumnValue, usize), Truncated> {
- if att.dropped {
- // Dropped cols retain attlen/attalign/typbyval per catalog
- // convention; still read varlena header to advance cursor
- let consumed = consume_dropped(att, buf, abs)?;
- return Ok((ColumnValue::Null, consumed));
- }
-
- if att.type_len == -1 {
- return decode_varlena(att, buf, abs);
- }
- if att.type_len == -2 {
- return decode_cstring(buf, abs);
- }
- let len = att.type_len as usize;
- if buf.len() < abs + len {
- return Err(Truncated {
- offset: abs,
- need: len,
- have: buf.len().saturating_sub(abs),
- });
- }
- Ok((fixed_value(att, &buf[abs..abs + len]), len))
-}
-
-/// Fixed-width value from its on-disk bytes. Inlined so callers construct
-/// in place: returning a 40-byte enum through memory then reloading it wide
-/// stalls store forwarding, which dominated fixed-width decode
-#[inline(always)]
-fn fixed_value(att: &RelAttr, body: &[u8]) -> ColumnValue {
- match att.type_oid {
- BOOLOID => ColumnValue::Bool(body[0] != 0),
- CHAROID => ColumnValue::Char(body[0] as i8),
- INT2OID => ColumnValue::Int2(i16::from_le_bytes(body.try_into().unwrap())),
- INT4OID => ColumnValue::Int4(i32::from_le_bytes(body.try_into().unwrap())),
- // `money` counts minor units, scale set by source `lc_monetary`
- INT8OID | MONEYOID => ColumnValue::Int8(i64::from_le_bytes(body.try_into().unwrap())),
- FLOAT4OID => ColumnValue::Float4(f32::from_le_bytes(body.try_into().unwrap())),
- FLOAT8OID => ColumnValue::Float8(f64::from_le_bytes(body.try_into().unwrap())),
- OIDOID | XIDOID | CIDOID => ColumnValue::Oid(u32::from_le_bytes(body.try_into().unwrap())),
- OID8OID | XID8OID | PG_LSNOID => {
- ColumnValue::Oid8(u64::from_le_bytes(body.try_into().unwrap()))
- }
- // NUL-padded to NAMEDATALEN
- NAMEOID => {
- let end = body.iter().position(|&b| b == 0).unwrap_or(body.len());
- match simdutf8::basic::from_utf8(&body[..end]) {
- Ok(s) => ColumnValue::Name(s.to_owned()),
- Err(_) => ColumnValue::Unsupported {
- type_oid: att.type_oid,
- raw: body.to_vec(),
- },
- }
- }
- MACADDROID => ColumnValue::MacAddr(body.try_into().unwrap()),
- MACADDR8OID => ColumnValue::MacAddr8(body.try_into().unwrap()),
- TIDOID => {
- let (block, offset) = crate::decode::codecs::decode_tid(body.try_into().unwrap());
- ColumnValue::Tid { block, offset }
- }
- DATEOID => ColumnValue::Date(i32::from_le_bytes(body.try_into().unwrap())),
- TIMEOID => ColumnValue::Time(i64::from_le_bytes(body.try_into().unwrap())),
- TIMESTAMPOID => ColumnValue::Timestamp(i64::from_le_bytes(body.try_into().unwrap())),
- TIMESTAMPTZOID => ColumnValue::TimestampTz(i64::from_le_bytes(body.try_into().unwrap())),
- TIMETZOID => {
- // 12 bytes: 8-byte time + 4-byte tz offset
- let micros = i64::from_le_bytes(body[..8].try_into().unwrap());
- let tz_seconds = i32::from_le_bytes(body[8..12].try_into().unwrap());
- ColumnValue::TimeTz { micros, tz_seconds }
+) -> Result<(DatumRef<'a>, usize), Truncated> {
+ let (datum, len) = match att.type_len {
+ -1 => read_varlena(buf, abs)?,
+ -2 => read_cstring(buf, abs)?,
+ len => {
+ let len = len as usize;
+ let body = buf.get(abs..abs + len).ok_or(Truncated {
+ offset: abs,
+ need: len,
+ have: buf.len().saturating_sub(abs),
+ })?;
+ (DatumRef::Fixed(body), len)
}
- UUIDOID => ColumnValue::Uuid(body.try_into().unwrap()),
- INTERVALOID => match crate::decode::codecs::decode_interval(body) {
- Ok(v) => ColumnValue::Interval(v),
- Err(_) => ColumnValue::Unsupported {
- type_oid: att.type_oid,
- raw: body.to_vec(),
- },
- },
- _ => ColumnValue::Unsupported {
- type_oid: att.type_oid,
- raw: body.to_vec(),
+ };
+ // Dropped cols retain attlen/attalign/typbyval per catalog convention, so
+ // reading still advances cursor
+ Ok((
+ if att.dropped {
+ DatumRef::Dropped
+ } else {
+ datum
},
- }
+ len,
+ ))
}
-/// Skip a dropped column; still read varlena header to advance correctly.
-fn consume_dropped(att: &RelAttr, buf: &[u8], abs: usize) -> Result {
- if att.type_len == -1 {
- let (_, consumed) = decode_varlena(att, buf, abs)?;
- Ok(consumed)
- } else if att.type_len == -2 {
- let (_, consumed) = decode_cstring(buf, abs)?;
- Ok(consumed)
- } else {
- let len = att.type_len as usize;
- if buf.len() < abs + len {
- return Err(Truncated {
- offset: abs,
- need: len,
- have: buf.len().saturating_sub(abs),
- });
+/// Owned copy of `datum`, `None` when absent from WAL image. Fast defaults
+/// resolve, stored where a local reader parses their text, and inline
+/// compressed bodies inflate
+pub fn resolve(att: &RelAttr, datum: DatumRef<'_>) -> Option {
+ Some(match datum {
+ DatumRef::Absent => return None,
+ DatumRef::Null | DatumRef::Dropped => Datum::Null,
+ DatumRef::Missing => match missing_datum(att) {
+ DatumRef::Text(t) => match text_datum(att.type_oid, t) {
+ TextDatum::Datum(d) => d,
+ TextDatum::Null => Datum::Null,
+ TextDatum::Opaque => Datum::Text(t.to_vec()),
+ },
+ d => return resolve(att, d),
+ },
+ DatumRef::Fixed(b) => Datum::fixed(b),
+ DatumRef::Varlena(b) => Datum::Varlena(b.to_vec()),
+ DatumRef::Compressed(datum) => {
+ inflate(datum).map_or_else(|| Datum::BadTag(datum.to_vec()), Datum::Varlena)
}
- Ok(len)
- }
+ DatumRef::External(p) => Datum::External(p),
+ DatumRef::Cstring(b) => Datum::Cstring(b.to_vec()),
+ DatumRef::BadTag(b) => Datum::BadTag(b.to_vec()),
+ DatumRef::Text(b) => Datum::Text(b.to_vec()),
+ DatumRef::Send(b) => Datum::Send(b.to_vec()),
+ })
+}
+
+/// Body of an inline compressed varlena, `None` on a corrupt stream
+pub(crate) fn inflate(datum: &[u8]) -> Option> {
+ // Reader checked datum spans header and va_tcinfo
+ let tcinfo = u32::from_le_bytes(datum.get(4..8)?.try_into().ok()?);
+ let method = (tcinfo >> VARLENA_EXTSIZE_BITS) as u8;
+ let raw_len = (tcinfo & VARLENA_EXTSIZE_MASK) as usize;
+ decompress_varlena(method, &datum[8..], raw_len)
}
/// PG `va_tcinfo`/`va_extinfo`: codec method in the top 2 bits, size in the low 30.
@@ -1259,20 +1302,15 @@ pub(crate) fn decompress_varlena(method: u8, src: &[u8], raw_len: usize) -> Opti
}
}
-/// Decode a varlena attribute (`typlen == -1`). PG varlena formats (`varatt.h`):
+/// Read a varlena attribute (`typlen == -1`). PG varlena formats (`varatt.h`):
///
/// - `0bxxxxxx00` 4-byte length, uncompressed.
-/// - `0bxxxxxx10` 4-byte length, in-line compressed (surfaced `Unsupported`;
-/// later codec detoasts pglz/lz4).
+/// - `0bxxxxxx10` 4-byte length, in-line compressed.
/// - `0b00000001` TOAST pointer (`varattrib_1b_e`, 18 bytes on disk).
/// - `0bxxxxxxx1` 1-byte length, uncompressed short header.
///
/// Little-endian only (`WORDS_BIGENDIAN` dead on x86/aarch64).
-fn decode_varlena(
- att: &RelAttr,
- buf: &[u8],
- abs: usize,
-) -> Result<(ColumnValue, usize), Truncated> {
+fn read_varlena(buf: &[u8], abs: usize) -> Result<(DatumRef<'_>, usize), Truncated> {
if buf.len() <= abs {
return Err(Truncated {
offset: abs,
@@ -1301,29 +1339,17 @@ fn decode_varlena(
});
}
let body = &buf[abs + 2..abs + 2 + 16];
- let va_rawsize = i32::from_le_bytes(body[0..4].try_into().unwrap());
- let va_extinfo = u32::from_le_bytes(body[4..8].try_into().unwrap());
- let va_valueid = u32::from_le_bytes(body[8..12].try_into().unwrap());
- let va_toastrelid = u32::from_le_bytes(body[12..16].try_into().unwrap());
- return Ok((
- ColumnValue::ExternalToast(ToastPointer {
- va_rawsize,
- va_extinfo,
- va_valueid,
- va_toastrelid,
- }),
- 18,
- ));
+ let pointer = ToastPointer {
+ va_rawsize: i32::from_le_bytes(body[0..4].try_into().unwrap()),
+ va_extinfo: u32::from_le_bytes(body[4..8].try_into().unwrap()),
+ va_valueid: u32::from_le_bytes(body[8..12].try_into().unwrap()),
+ va_toastrelid: u32::from_le_bytes(body[12..16].try_into().unwrap()),
+ };
+ return Ok((DatumRef::External(pointer), 18));
}
// INDIRECT/EXPANDED tags are in-memory only, never on disk (varatt.h);
// forward Unsupported so a corrupt stream surfaces
- return Ok((
- ColumnValue::Unsupported {
- type_oid: att.type_oid,
- raw: buf[abs..].to_vec(),
- },
- buf.len() - abs,
- ));
+ return Ok((DatumRef::BadTag(&buf[abs..]), buf.len() - abs));
}
if first & 0x01 != 0 {
// Short header: low bit set, total length (incl header byte) in upper 7;
@@ -1336,9 +1362,7 @@ fn decode_varlena(
have: buf.len() - abs,
});
}
- let body = &buf[abs + 1..abs + total];
- let v = varlena_to_value(att.type_oid, Cow::Borrowed(body));
- return Ok((v, total));
+ return Ok((DatumRef::Varlena(&buf[abs + 1..abs + total]), total));
}
// 4-byte length: bits [1:0] = 00 uncompressed, 10 compressed
if buf.len() < abs + 4 {
@@ -1374,75 +1398,9 @@ fn decode_varlena(
have: total,
});
}
- let tcinfo = u32::from_le_bytes(buf[abs + 4..abs + 8].try_into().unwrap());
- let raw_len = (tcinfo & VARLENA_EXTSIZE_MASK) as usize;
- let method = (tcinfo >> VARLENA_EXTSIZE_BITS) as u8;
- let body = &buf[abs + 8..abs + total];
- let v = decompress_varlena(method, body, raw_len)
- .map(|out| varlena_to_value(att.type_oid, Cow::Owned(out)))
- .unwrap_or_else(|| ColumnValue::Unsupported {
- type_oid: att.type_oid,
- raw: buf[abs..abs + total].to_vec(),
- });
- return Ok((v, total));
- }
- let body = &buf[abs + 4..abs + total];
- Ok((varlena_to_value(att.type_oid, Cow::Borrowed(body)), total))
-}
-
-/// Map a varlena body (header stripped, decompressed) to a typed value by the
-/// column's type OID. Shared by the inline decoder (`Cow::Borrowed` into the
-/// WAL buffer) and the TOAST reassembler (`Cow::Owned`, so `detoasted_value`
-/// moves the reassembled buffer into the value instead of copying it). A
-/// detoasted value resolves identically to an inline one — Tier 3 types land
-/// as `PgPending`, resolved at emit by the oracle.
-pub(crate) fn varlena_to_value(type_oid: u32, body: Cow<[u8]>) -> ColumnValue {
- match type_oid {
- BYTEAOID => ColumnValue::Bytea(body.into_owned()),
- TEXTOID | VARCHAROID | BPCHAROID | JSONOID => {
- // Owned arm keeps detoasted buffer rather than copy it
- let text = match body {
- Cow::Borrowed(b) => simdutf8::basic::from_utf8(b)
- .map(str::to_owned)
- .map_err(|_| b.to_vec()),
- Cow::Owned(v) => String::from_utf8(v).map_err(FromUtf8Error::into_bytes),
- };
- // PG validates UTF-8 on input; surface bytea not crash if ever invalid
- match text {
- Ok(s) if type_oid == JSONOID => ColumnValue::Json(s),
- Ok(s) => ColumnValue::Text(s),
- Err(b) => ColumnValue::Bytea(b),
- }
- }
- NUMERICOID => match crate::decode::codecs::decode_numeric(&body) {
- Ok(v) => ColumnValue::Numeric(v),
- Err(_) => ColumnValue::Unsupported {
- type_oid,
- raw: body.into_owned(),
- },
- },
- INETOID | CIDROID => match crate::decode::codecs::decode_inet(&body, type_oid == CIDROID) {
- Ok(v) => ColumnValue::Inet(v),
- Err(_) => ColumnValue::Unsupported {
- type_oid,
- raw: body.into_owned(),
- },
- },
- JSONBOID => match crate::decode::codecs::decode_jsonb(&body) {
- Ok(s) => ColumnValue::Json(s),
- Err(_) => ColumnValue::Unsupported {
- type_oid,
- raw: body.into_owned(),
- },
- },
- // Tier 3 deferred: range types, arrays (typcategory='A'),
- // tsvector etc. Carries full on-disk body so the SQL bridge
- // reconstructs the varlena Datum; walshadow extension resolves at emit
- _ => ColumnValue::PgPending {
- type_oid,
- raw: body.into_owned(),
- },
+ return Ok((DatumRef::Compressed(&buf[abs..abs + total]), total));
}
+ Ok((DatumRef::Varlena(&buf[abs + 4..abs + total]), total))
}
/// Match decoder paths producing typed values
@@ -1492,47 +1450,39 @@ pub(crate) fn local_matrix_covers(type_oid: u32, type_len: i16) -> bool {
}
}
-/// Handle `typlen=-2` `\0`-terminated cstring (regtype/regclass typoutput
+/// Read `typlen=-2` `\0`-terminated cstring (regtype/regclass typoutput
/// intermediates). Not in the type matrix, but tracking it lets dropped
/// cstring columns walk correctly. (`name`, typlen=64, goes the fixed path.)
-fn decode_cstring(buf: &[u8], abs: usize) -> Result<(ColumnValue, usize), Truncated> {
- let mut end = abs;
- while end < buf.len() && buf[end] != 0 {
- end += 1;
- }
- if end >= buf.len() {
- return Err(Truncated {
+fn read_cstring(buf: &[u8], abs: usize) -> Result<(DatumRef<'_>, usize), Truncated> {
+ let end = buf
+ .get(abs..)
+ .and_then(|rest| rest.iter().position(|&b| b == 0))
+ .ok_or(Truncated {
offset: abs,
need: 1,
have: 0,
- });
- }
- let body = &buf[abs..end];
- let value = simdutf8::basic::from_utf8(body)
- .map(|s| ColumnValue::Text(s.to_owned()))
- .unwrap_or_else(|_| ColumnValue::Bytea(body.to_vec()));
- Ok((value, (end - abs) + 1)) // include trailing NUL
+ })?;
+ // include trailing NUL
+ Ok((DatumRef::Cstring(&buf[abs..abs + end]), end + 1))
}
-/// Take `(chunk_id, chunk_seq, chunk_data)` out of a `pg_toast_*` tuple's
-/// 3 columns (`chunk_id oid`, `chunk_seq int4`, `chunk_data bytea`);
+/// `(chunk_id, chunk_seq, chunk_data)` of a `pg_toast_*` tuple's 3 datums;
/// `None` for shapes that don't fit
-pub(crate) fn take_toast_chunk_columns(
- cols: &mut [Option],
-) -> Option<(u32, u32, Vec)> {
- if cols.len() < 3 {
- return None;
- }
- let &ColumnValue::Oid(chunk_id) = cols[0].as_ref()? else {
- return None;
- };
- let &ColumnValue::Int4(chunk_seq) = cols[1].as_ref()? else {
+pub(crate) fn toast_chunk_datums<'a>(datums: &[DatumRef<'a>]) -> Option<(u32, u32, &'a [u8])> {
+ let [
+ DatumRef::Fixed(id),
+ DatumRef::Fixed(seq),
+ DatumRef::Varlena(data),
+ ..,
+ ] = *datums
+ else {
return None;
};
- let ColumnValue::Bytea(chunk_data) = cols[2].take()? else {
- return None;
- };
- Some((chunk_id, chunk_seq as u32, chunk_data))
+ Some((
+ u32::from_le_bytes(id.try_into().ok()?),
+ u32::from_le_bytes(seq.try_into().ok()?),
+ data,
+ ))
}
/// True iff `attnum` is part of the relation's replica identity. Emitter gates
@@ -1549,41 +1499,7 @@ pub fn is_replica_identity_attr(replident: &ReplIdent, attnum: i16) -> bool {
}
}
-/// Build a delete marker when UPDATE changes primary/index key, or any column
-/// under `REPLICA IDENTITY FULL` without a primary key. Use `lsn - 1` so new
-/// row wins if both share a destination key. PostgreSQL default/index identity
-/// logs old values only when key changes or contains TOASTed values
-pub fn take_key_change_tombstone(
- heap: &mut DecodedHeap,
- rel: &RelDescriptor,
-) -> Option {
- let hot = match heap.op {
- HeapOp::Update => false,
- HeapOp::HotUpdate => true,
- _ => return None,
- };
- let (old, new) = (heap.old.as_ref()?, heap.new.as_ref()?);
- let differs = |a: usize| matches!((old.columns.get(a), new.columns.get(a)), (Some(Some(o)), Some(Some(n))) if o != n);
- let keys = crate::schema::replident_key_attnums(rel);
- let moved = if keys.is_empty() {
- matches!(rel.replident, ReplIdent::Full { pk_attnums: None })
- && (0..new.columns.len()).any(differs)
- } else {
- // PostgreSQL HOT updates leave indexed columns unchanged
- !hot && keys.iter().any(|&a| differs(a as usize - 1))
- };
- moved.then(|| DecodedHeap {
- rfn: heap.rfn,
- xid: heap.xid,
- source_lsn: heap.source_lsn - 1,
- op: HeapOp::Delete,
- new: None,
- old: heap.old.take(),
- })
-}
-
/// PG `attmissingval`: one-element, no-null `ArrayType`, payload at `MAXALIGN(24)`
-#[cfg(test)]
pub(crate) fn raw_missing_array(elemtype: u32, elem: &[u8]) -> Vec {
let mut v = Vec::with_capacity(24 + elem.len());
v.extend_from_slice(&0u32.to_le_bytes()); // vl_len_ (ignored)
@@ -1609,6 +1525,7 @@ pub(crate) fn short_varlena(content: &[u8]) -> Vec {
#[cfg(test)]
mod tests {
use super::*;
+ use crate::decode::datum::{Datum, build};
use crate::schema::RelName;
use walrus::pg::walparser::{
BlockLocation, XLogRecordBlock, XLogRecordBlockHeader, XLogRecordHeader,
@@ -1662,21 +1579,13 @@ mod tests {
#[test]
fn approx_bytes_counts_scalars_and_varlena() {
- assert_eq!(ColumnValue::Null.approx_bytes(), 0);
- assert_eq!(ColumnValue::Int2(0).approx_bytes(), 2);
- assert_eq!(ColumnValue::Int4(0).approx_bytes(), 4);
- assert_eq!(ColumnValue::Int8(0).approx_bytes(), 8);
- assert_eq!(ColumnValue::Uuid([0; 16]).approx_bytes(), 16);
- assert_eq!(ColumnValue::Text("héllo".into()).approx_bytes(), 6);
- assert_eq!(ColumnValue::Bytea(vec![0u8; 100]).approx_bytes(), 100);
- assert_eq!(
- ColumnValue::PgPending {
- type_oid: 1,
- raw: vec![0u8; 7],
- }
- .approx_bytes(),
- 7
- );
+ assert_eq!(Datum::Null.approx_bytes(), 0);
+ assert_eq!(build::int2(0).approx_bytes(), 2);
+ assert_eq!(build::int4(0).approx_bytes(), 4);
+ assert_eq!(build::int8(0).approx_bytes(), 8);
+ assert_eq!(build::uuid([0; 16]).approx_bytes(), 16);
+ assert_eq!(build::text("héllo").approx_bytes(), 6);
+ assert_eq!(build::bytes(vec![0u8; 100]).approx_bytes(), 100);
let heap = DecodedHeap {
rfn: RelFileNode {
@@ -1688,14 +1597,11 @@ mod tests {
source_lsn: 0,
op: HeapOp::Update,
new: Some(DecodedTuple {
- columns: vec![
- Some(ColumnValue::Int4(1)),
- Some(ColumnValue::Bytea(vec![0u8; 10])),
- ],
+ columns: vec![Some(build::int4(1)), Some(build::bytes(vec![0u8; 10]))],
partial: false,
}),
old: Some(DecodedTuple {
- columns: vec![None, Some(ColumnValue::Text("abc".into()))],
+ columns: vec![None, Some(build::text("abc"))],
partial: false,
}),
};
@@ -1748,75 +1654,67 @@ mod tests {
}
}
- #[test]
- fn varlena_text_preserves_bytes_and_owned_buffers() {
- let suffixes: &[&[u8]] = &[
- b"",
- "é😀".as_bytes(),
- &[0xff],
- &[0xc0, 0xaf],
- &[0xed, 0xa0, 0x80],
- &[0xf0, 0x9f, 0x98],
- ];
- for type_oid in [TEXTOID, VARCHAROID, BPCHAROID, JSONOID] {
- for prefix in [0, 63, 64, 65, 127, 128, 129] {
- for suffix in suffixes {
- let mut raw = vec![b'a'; prefix];
- raw.extend_from_slice(suffix);
- let expected = match String::from_utf8(raw.clone()) {
- Ok(s) if type_oid == JSONOID => ColumnValue::Json(s),
- Ok(s) => ColumnValue::Text(s),
- Err(_) => ColumnValue::Bytea(raw.clone()),
- };
- for body in [Cow::Borrowed(raw.as_slice()), Cow::Owned(raw.clone())] {
- let owned = matches!(body, Cow::Owned(_));
- let ptr = body.as_ptr();
- let value = varlena_to_value(type_oid, body);
- assert_eq!(value, expected);
- if owned {
- assert!(
- matches!(&value, ColumnValue::Text(s) | ColumnValue::Json(s) if s.as_ptr() == ptr)
- || matches!(&value, ColumnValue::Bytea(b) if b.as_ptr() == ptr),
- "owned body reused in place"
- );
- }
- }
- }
- }
+ /// Every decode here also checks borrowed views build equal cells
+ fn decode_heap_record(
+ record: &XLogRecord,
+ source_lsn: u64,
+ rel: &RelDescriptor,
+ ) -> Result {
+ let owned = super::decode_heap_record(record, source_lsn, rel);
+ let cells = |t: &Option>| t.as_ref().map(|t| t.to_tuple(rel));
+ match (&owned, view_heap_record(record, rel)) {
+ (Ok(heaps), Ok(views)) => assert_eq!(
+ heaps
+ .iter()
+ .map(|h| (h.op, h.new.clone(), h.old.clone()))
+ .collect::>(),
+ views
+ .iter()
+ .map(|v| (v.op, cells(&v.new), cells(&v.old)))
+ .collect::>(),
+ ),
+ (Err(a), Err(b)) => assert_eq!(a.to_string(), b.to_string()),
+ (a, b) => panic!("owned {a:?} vs view {b:?}"),
}
+ owned
+ }
+
+ /// Owned datum at `abs` and its stored length
+ fn decode_one_value(
+ att: &RelAttr,
+ buf: &[u8],
+ abs: usize,
+ ) -> Result<(Datum, usize), Truncated> {
+ let (datum, len) = read_datum(att, buf, abs)?;
+ Ok((resolve(att, datum).unwrap_or(Datum::Null), len))
+ }
+
+ fn missing_value_for(att: &RelAttr) -> Datum {
+ resolve(att, DatumRef::Missing).expect("fast default resolves")
}
#[test]
- fn decode_cstring_text_bytea_and_truncated() {
- let (v, n) = decode_cstring(b"hello\0", 0).unwrap();
- assert_eq!(v, ColumnValue::Text("hello".into()));
+ fn decode_cstring_bytes_and_truncated() {
+ let att = rel_attr(1, "c", 0, -2, 'c');
+ let cstring = |s: &[u8]| Datum::Cstring(s.to_vec());
+ let (v, n) = decode_one_value(&att, b"hello\0", 0).unwrap();
+ assert_eq!(v, cstring(b"hello"));
assert_eq!(n, 6);
- let (v, n) = decode_cstring(&[0xFF, 0xFE, 0x00], 0).unwrap();
- assert_eq!(v, ColumnValue::Bytea(vec![0xFF, 0xFE]));
+ let (v, n) = decode_one_value(&att, &[0xFF, 0xFE, 0x00], 0).unwrap();
+ assert_eq!(v, cstring(&[0xFF, 0xFE]));
assert_eq!(n, 3);
// non-zero offset within multi-string buffer
- let (v, n) = decode_cstring(b"ab\0cd\0", 3).unwrap();
- assert_eq!(v, ColumnValue::Text("cd".into()));
+ let (v, n) = decode_one_value(&att, b"ab\0cd\0", 3).unwrap();
+ assert_eq!(v, cstring(b"cd"));
assert_eq!(n, 3);
// no terminator before buffer end
- let err = decode_cstring(b"hello", 0).unwrap_err();
+ let err = decode_one_value(&att, b"hello", 0).unwrap_err();
assert_eq!(
format!("{err:?}"),
"Truncated { offset: 0, need: 1, have: 0 }"
);
}
- #[test]
- fn decode_one_value_maps_char_to_signed_i8() {
- let att = rel_attr(1, "c", CHAROID, 1, 'c');
- let (v, n) = decode_one_value(&att, &[65u8], 0).unwrap();
- assert_eq!(v, ColumnValue::Char(65));
- assert_eq!(n, 1);
- // 0xFF -> i8 -1
- let (v, _) = decode_one_value(&att, &[0xFFu8], 0).unwrap();
- assert_eq!(v, ColumnValue::Char(-1));
- }
-
#[test]
fn decode_insert_int4_int8() {
let rel = descriptor(
@@ -1840,8 +1738,8 @@ mod tests {
let new = out.new.unwrap();
assert!(!new.partial);
assert_eq!(new.columns.len(), 2);
- assert_eq!(new.columns[0], Some(ColumnValue::Int4(12345)));
- assert_eq!(new.columns[1], Some(ColumnValue::Int8(999_999)));
+ assert_eq!(new.columns[0], Some(build::int4(12345)));
+ assert_eq!(new.columns[1], Some(build::int8(999_999)));
}
#[test]
@@ -1867,8 +1765,8 @@ mod tests {
let rec = record_with(RmId::Heap, XLOG_HEAP_INSERT, main_data, payload);
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
- assert_eq!(new.columns[0], Some(ColumnValue::Bool(true)));
- assert_eq!(new.columns[1], Some(ColumnValue::Text("hi".into())));
+ assert_eq!(new.columns[0], Some(build::bool_(true)));
+ assert_eq!(new.columns[1], Some(build::text("hi")));
}
#[test]
@@ -1886,7 +1784,7 @@ mod tests {
let rec = record_with(RmId::Heap, XLOG_HEAP_INSERT, main_data, payload);
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
- assert_eq!(new.columns[0], Some(ColumnValue::Text("hi".into())));
+ assert_eq!(new.columns[0], Some(build::text("hi")));
}
fn inline_compressed_varlena(method: u8, body: &[u8], raw_len: usize) -> Vec {
@@ -1906,9 +1804,9 @@ mod tests {
assert!(comp.len() < text.len(), "must actually compress");
let buf = inline_compressed_varlena(0, &comp, text.len());
let att = rel_attr(1, "val", TEXTOID, -1, 'i');
- let (v, consumed) = decode_varlena(&att, &buf, 0).unwrap();
+ let (v, consumed) = decode_one_value(&att, &buf, 0).unwrap();
assert_eq!(consumed, buf.len());
- assert_eq!(v, ColumnValue::Text(text));
+ assert_eq!(v, build::text(&text));
}
#[test]
@@ -1917,24 +1815,18 @@ mod tests {
let comp = lz4::block::compress(text.as_bytes(), None, false).expect("lz4 compress");
let buf = inline_compressed_varlena(1, &comp, text.len());
let att = rel_attr(1, "val", TEXTOID, -1, 'i');
- let (v, consumed) = decode_varlena(&att, &buf, 0).unwrap();
+ let (v, consumed) = decode_one_value(&att, &buf, 0).unwrap();
assert_eq!(consumed, buf.len());
- assert_eq!(v, ColumnValue::Text(text));
+ assert_eq!(v, build::text(&text));
}
#[test]
fn decode_inline_compressed_corrupt_surfaces_unsupported() {
let buf = inline_compressed_varlena(0, &[0xff, 0xff, 0xff, 0xff], 4096);
let att = rel_attr(1, "val", TEXTOID, -1, 'i');
- let (v, consumed) = decode_varlena(&att, &buf, 0).unwrap();
+ let (v, consumed) = decode_one_value(&att, &buf, 0).unwrap();
assert_eq!(consumed, buf.len());
- assert!(matches!(
- v,
- ColumnValue::Unsupported {
- type_oid: TEXTOID,
- ..
- }
- ));
+ assert!(matches!(v, Datum::BadTag(_)));
}
#[test]
@@ -1979,9 +1871,9 @@ mod tests {
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
assert_eq!(new.columns.len(), 3);
- assert_eq!(new.columns[0], Some(ColumnValue::Int4(100)));
- assert_eq!(new.columns[1], Some(ColumnValue::Null));
- assert_eq!(new.columns[2], Some(ColumnValue::Int4(300)));
+ assert_eq!(new.columns[0], Some(build::int4(100)));
+ assert_eq!(new.columns[1], Some(Datum::Null));
+ assert_eq!(new.columns[2], Some(build::int4(300)));
}
#[test]
@@ -1999,7 +1891,7 @@ mod tests {
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
assert_eq!(out.op, HeapOp::Delete);
let old = out.old.unwrap();
- assert_eq!(old.columns[0], Some(ColumnValue::Int4(7777)));
+ assert_eq!(old.columns[0], Some(build::int4(7777)));
}
#[test]
@@ -2028,7 +1920,7 @@ mod tests {
let new = out.new.unwrap();
assert!(new.partial, "prefix-compressed UPDATE flagged partial");
assert_eq!(new.columns[0], None);
- assert_eq!(new.columns[1], Some(ColumnValue::Int4(999)));
+ assert_eq!(new.columns[1], Some(build::int4(999)));
}
#[test]
@@ -2046,7 +1938,7 @@ mod tests {
assert_eq!(out.op, HeapOp::HotUpdate);
assert!(out.old.is_none());
let new = out.new.unwrap();
- assert_eq!(new.columns[0], Some(ColumnValue::Int4(55)));
+ assert_eq!(new.columns[0], Some(build::int4(55)));
}
#[test]
@@ -2062,8 +1954,8 @@ mod tests {
let rec = record_with(RmId::Heap, XLOG_HEAP_INSERT, main_data, payload);
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
- assert_eq!(new.columns[0], Some(ColumnValue::Null));
- assert_eq!(new.columns[1], Some(ColumnValue::Int4(8888)));
+ assert_eq!(new.columns[0], Some(Datum::Null));
+ assert_eq!(new.columns[1], Some(build::int4(8888)));
}
#[test]
@@ -2097,7 +1989,7 @@ mod tests {
let rec = record_with(RmId::Heap, XLOG_HEAP_INSERT, main_data, payload);
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
- assert_eq!(new.columns[0], Some(ColumnValue::Uuid(uuid_bytes)));
+ assert_eq!(new.columns[0], Some(build::uuid(uuid_bytes)));
}
#[test]
@@ -2118,7 +2010,7 @@ mod tests {
let new = out.new.unwrap();
assert_eq!(
new.columns[0],
- Some(ColumnValue::ExternalToast(ToastPointer {
+ Some(Datum::External(ToastPointer {
va_rawsize: 12345,
va_extinfo: 678,
va_valueid: 12,
@@ -2129,8 +2021,8 @@ mod tests {
#[test]
fn decode_unsupported_type_emits_pending() {
- // tsvector is outside the local matrix; varlena fall-through yields
- // PgPending preserving raw bytes (shadow extension resolves at emit)
+ // tsvector is outside the local matrix; its stored body survives for
+ // the shadow extension to convert at emit
let rel = descriptor(16397, vec![rel_attr(1, "t", TSVECTOROID, -1, 'i')]);
let body = b"\x01opaque";
let total = 4 + body.len();
@@ -2143,13 +2035,7 @@ mod tests {
let rec = record_with(RmId::Heap, XLOG_HEAP_INSERT, main_data, payload);
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
- assert_eq!(
- new.columns[0],
- Some(ColumnValue::PgPending {
- type_oid: TSVECTOROID,
- raw: body.to_vec(),
- })
- );
+ assert_eq!(new.columns[0], Some(build::bytes(body)));
}
#[test]
@@ -2175,9 +2061,9 @@ mod tests {
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
assert_eq!(new.columns.len(), 3);
- assert_eq!(new.columns[0], Some(ColumnValue::Int4(100)));
- assert_eq!(new.columns[1], Some(ColumnValue::Int4(200)));
- assert_eq!(new.columns[2], Some(ColumnValue::Int4(7)));
+ assert_eq!(new.columns[0], Some(build::int4(100)));
+ assert_eq!(new.columns[1], Some(build::int4(200)));
+ assert_eq!(new.columns[2], Some(build::int4(7)));
}
#[test]
@@ -2199,7 +2085,7 @@ mod tests {
let rec = record_with(RmId::Heap, XLOG_HEAP_INSERT, main_data, payload);
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
- assert_eq!(new.columns[2], Some(ColumnValue::Null));
+ assert_eq!(new.columns[2], Some(Datum::Null));
}
#[test]
@@ -2220,8 +2106,38 @@ mod tests {
let rec = record_with(RmId::Heap, XLOG_HEAP_INSERT, main_data, payload);
let out = decode_heap_record(&rec, 0, &rel).unwrap().remove(0);
let new = out.new.unwrap();
- assert_eq!(new.columns[0], Some(ColumnValue::Int4(55)));
- assert_eq!(new.columns[1], Some(ColumnValue::Null));
+ assert_eq!(new.columns[0], Some(build::int4(55)));
+ assert_eq!(new.columns[1], Some(Datum::Null));
+ }
+
+ #[test]
+ fn record_shape_reads_headers() {
+ let shape = |rm: RmId, info, main_data: &[u8]| {
+ record_shape(HeapRecordBytes {
+ rm: rm as u8,
+ info,
+ main_data,
+ block: None,
+ })
+ };
+ let mut update = [0u8; SIZE_OF_HEAP_UPDATE];
+ update[7] = XLH_UPDATE_SUFFIX_FROM_OLD;
+ let s = shape(RmId::Heap, XLOG_HEAP_HOT_UPDATE, &update)
+ .unwrap()
+ .unwrap();
+ assert_eq!((s.op, s.tuples, s.partial), (HeapOp::HotUpdate, 1, true));
+ let multi = [XLH_INSERT_CONTAINS_NEW_TUPLE, 0, 3, 0];
+ let s = shape(RmId::Heap2, XLOG_HEAP2_MULTI_INSERT, &multi)
+ .unwrap()
+ .unwrap();
+ assert_eq!((s.op, s.tuples, s.partial), (HeapOp::Insert, 3, false));
+ assert!(matches!(
+ shape(RmId::Heap2, XLOG_HEAP2_MULTI_INSERT, &[0, 0, 3, 0]),
+ Ok(None)
+ ));
+ assert!(matches!(shape(RmId::Heap, XLOG_HEAP_LOCK, &[]), Ok(None)));
+ assert!(shape(RmId::Heap2, XLOG_HEAP2_MULTI_INSERT, &[0, 0, 0, 0]).is_err());
+ assert!(shape(RmId::Heap, XLOG_HEAP_DELETE, &[0; 4]).is_err());
}
#[test]
@@ -2258,7 +2174,7 @@ mod tests {
assert_eq!(out[i].source_lsn, 0x2000);
let new = out[i].new.as_ref().unwrap();
assert_eq!(new.columns.len(), 1);
- assert_eq!(new.columns[0], Some(ColumnValue::Int4(*expected)));
+ assert_eq!(new.columns[0], Some(build::int4(*expected)));
}
}
@@ -2307,14 +2223,11 @@ mod tests {
let cols: Vec<_> = out.iter().map(|h| h.new.clone().unwrap()).collect();
assert_eq!(
cols[0].columns,
- vec![
- Some(ColumnValue::Int4(7)),
- Some(ColumnValue::Text("ab".into()))
- ]
+ vec![Some(build::int4(7)), Some(build::text("ab"))]
);
assert_eq!(
cols[1].columns,
- vec![Some(ColumnValue::Int4(8)), Some(ColumnValue::Null)]
+ vec![Some(build::int4(8)), Some(Datum::Null)]
);
assert!(!cols[0].partial && !cols[1].partial);
}
@@ -2347,14 +2260,11 @@ mod tests {
let out = decode_heap_record(&rec, 0, &rel).unwrap();
let first = out[0].new.as_ref().unwrap();
assert!(first.partial);
- assert_eq!(first.columns, vec![Some(ColumnValue::Int4(7)), None]);
+ assert_eq!(first.columns, vec![Some(build::int4(7)), None]);
let second = out[1].new.as_ref().unwrap();
assert_eq!(
second.columns,
- vec![
- Some(ColumnValue::Int4(8)),
- Some(ColumnValue::Text("z".into()))
- ]
+ vec![Some(build::int4(8)), Some(build::text("z"))]
);
}
@@ -2409,24 +2319,19 @@ mod tests {
#[test]
fn fixed_system_types_decode_values() {
- let cases: [(u32, Vec, char, ColumnValue); 7] = [
- (
- XIDOID,
- 7u32.to_le_bytes().to_vec(),
- 'i',
- ColumnValue::Oid(7),
- ),
+ let cases: [(u32, Vec, char, Datum); 7] = [
+ (XIDOID, 7u32.to_le_bytes().to_vec(), 'i', build::oid(7)),
(
PG_LSNOID,
0x16_B374_D848u64.to_le_bytes().to_vec(),
'd',
- ColumnValue::Oid8(0x16_B374_D848),
+ build::oid8(0x16_B374_D848),
),
(
MONEYOID,
(-123_456i64).to_le_bytes().to_vec(),
'd',
- ColumnValue::Int8(-123_456),
+ build::int8(-123_456),
),
(
NAMEOID,
@@ -2436,28 +2341,25 @@ mod tests {
b
},
'c',
- ColumnValue::Name("pg_class".into()),
+ build::name("pg_class"),
),
(
MACADDROID,
vec![0x08, 0x00, 0x2b, 0x01, 0x02, 0xab],
'i',
- ColumnValue::MacAddr([0x08, 0x00, 0x2b, 0x01, 0x02, 0xab]),
+ Datum::fixed(&[0x08, 0x00, 0x2b, 0x01, 0x02, 0xab]),
),
(
MACADDR8OID,
vec![0x08, 0x00, 0x2b, 0xff, 0xfe, 0x01, 0x02, 0x03],
'i',
- ColumnValue::MacAddr8([0x08, 0x00, 0x2b, 0xff, 0xfe, 0x01, 0x02, 0x03]),
+ Datum::fixed(&[0x08, 0x00, 0x2b, 0xff, 0xfe, 0x01, 0x02, 0x03]),
),
(
TIDOID,
vec![0, 0, 5, 0, 3, 0],
's',
- ColumnValue::Tid {
- block: 5,
- offset: 3,
- },
+ Datum::fixed(&crate::decode::codecs::encode_tid(5, 3)),
),
];
for (oid, body, align, want) in cases {
@@ -2471,15 +2373,15 @@ mod tests {
fn missing_value_text_parses_system_types() {
let mut lsn = rel_attr(1, "l", PG_LSNOID, 8, 'd');
lsn.missing_default = Some("16/B374D848".into());
- assert_eq!(missing_value_for(&lsn), ColumnValue::Oid8(0x16_B374_D848));
+ assert_eq!(missing_value_for(&lsn), build::oid8(0x16_B374_D848));
let mut x8 = rel_attr(1, "x", XID8OID, 8, 'd');
x8.missing_default = Some("18446744073709551615".into());
- assert_eq!(missing_value_for(&x8), ColumnValue::Oid8(u64::MAX));
+ assert_eq!(missing_value_for(&x8), build::oid8(u64::MAX));
let mut mac = rel_attr(1, "m", MACADDROID, 6, 'i');
mac.missing_default = Some("08:00:2b:01:02:03".into());
assert_eq!(
missing_value_for(&mac),
- ColumnValue::MacAddr([0x08, 0x00, 0x2b, 0x01, 0x02, 0x03])
+ Datum::fixed(&[0x08, 0x00, 0x2b, 0x01, 0x02, 0x03])
);
}
@@ -2487,34 +2389,33 @@ mod tests {
fn missing_value_for_type_matrix() {
let mut bool_att = rel_attr(1, "b", BOOLOID, 1, 'c');
bool_att.missing_default = Some("t".into());
- assert_eq!(missing_value_for(&bool_att), ColumnValue::Bool(true));
+ assert_eq!(missing_value_for(&bool_att), build::bool_(true));
bool_att.missing_default = Some("false".into());
- assert_eq!(missing_value_for(&bool_att), ColumnValue::Bool(false));
+ assert_eq!(missing_value_for(&bool_att), build::bool_(false));
let mut i4 = rel_attr(1, "x", INT4OID, 4, 'i');
i4.missing_default = Some("42".into());
- assert_eq!(missing_value_for(&i4), ColumnValue::Int4(42));
+ assert_eq!(missing_value_for(&i4), build::int4(42));
let mut i8 = rel_attr(1, "x", INT8OID, 8, 'd');
i8.missing_default = Some("-9223372036854775808".into());
- assert_eq!(missing_value_for(&i8), ColumnValue::Int8(i64::MIN));
+ assert_eq!(missing_value_for(&i8), build::int8(i64::MIN));
let mut txt = rel_attr(1, "n", TEXTOID, -1, 'i');
txt.missing_default = Some("hello".into());
- assert_eq!(missing_value_for(&txt), ColumnValue::Text("hello".into()));
+ assert_eq!(missing_value_for(&txt), build::out_text("hello"));
let mut num = rel_attr(1, "n", NUMERICOID, -1, 'i');
num.missing_default = Some("3.14".into());
- assert_eq!(
- missing_value_for(&num),
- ColumnValue::PgPendingText {
- type_oid: NUMERICOID,
- text: "3.14".into(),
- }
- );
+ assert_eq!(missing_value_for(&num), build::numeric("3.14"));
+ num.missing_default = Some("pi".into());
+ assert_eq!(missing_value_for(&num), Datum::Null);
+ let mut ts = rel_attr(1, "t", TSVECTOROID, -1, 'i');
+ ts.missing_default = Some("'a'".into());
+ assert_eq!(missing_value_for(&ts), build::out_text("'a'"));
let none = rel_attr(1, "x", INT4OID, 4, 'i');
- assert_eq!(missing_value_for(&none), ColumnValue::Null);
+ assert_eq!(missing_value_for(&none), Datum::Null);
}
#[test]
@@ -2524,7 +2425,7 @@ mod tests {
INT4OID,
&7i32.to_le_bytes(),
)));
- assert_eq!(missing_value_for(&a), ColumnValue::Int4(7));
+ assert_eq!(missing_value_for(&a), build::int4(7));
}
#[test]
@@ -2534,7 +2435,7 @@ mod tests {
TEXTOID,
&short_varlena(b"hi"),
)));
- assert_eq!(missing_value_for(&a), ColumnValue::Text("hi".into()));
+ assert_eq!(missing_value_for(&a), build::text("hi"));
}
#[test]
@@ -2547,31 +2448,28 @@ mod tests {
&short_varlena(&body),
)));
// Tier-3 fast default stays pending
- assert_eq!(
- missing_value_for(&a),
- ColumnValue::PgPending {
- type_oid: TSVECTOROID,
- raw: body.to_vec(),
- }
- );
+ assert_eq!(missing_value_for(&a), build::bytes(body));
}
#[test]
- fn fast_default_raw_jsonb_renders_its_document() {
+ fn fast_default_raw_jsonb_stays_stored() {
// `{}`: bare container header, JB_FOBJECT with no pairs
let mut a = rel_attr(1, "j", JSONBOID, -1, 'i');
a.missing_default = Some(MissingDefault::Raw(raw_missing_array(
JSONBOID,
&short_varlena(&0x2000_0000u32.to_le_bytes()),
)));
- assert_eq!(missing_value_for(&a), ColumnValue::Json("{}".into()));
+ assert_eq!(
+ missing_value_for(&a),
+ build::bytes(0x2000_0000u32.to_le_bytes())
+ );
}
#[test]
fn fast_default_malformed_raw_is_null_not_panic() {
let mut a = rel_attr(1, "x", INT4OID, 4, 'i');
a.missing_default = Some(MissingDefault::Raw(vec![0, 1, 2]));
- assert_eq!(missing_value_for(&a), ColumnValue::Null);
+ assert_eq!(missing_value_for(&a), Datum::Null);
}
#[test]
@@ -2599,6 +2497,80 @@ mod tests {
assert!(missing_defaults_equivalent(&raw_ts, &text_ts));
}
+ #[test]
+ fn fast_default_equivalent_across_raw_and_text_name() {
+ let mut raw = rel_attr(1, "n", NAMEOID, 64, 'c');
+ let name = build::name("nm");
+ let Datum::Fixed(stored) = &name else {
+ panic!("name is fixed width");
+ };
+ raw.missing_default = Some(MissingDefault::Raw(raw_missing_array(NAMEOID, stored)));
+ let mut text = rel_attr(1, "n", NAMEOID, 64, 'c');
+ text.missing_default = Some("nm".into());
+ assert!(missing_defaults_equivalent(&raw, &text));
+
+ // Text the parser rejects reads as NULL on both sides
+ let mut bad = rel_attr(1, "x", INT4OID, 4, 'i');
+ bad.missing_default = Some("x".into());
+ assert!(missing_defaults_equivalent(&bad, &bad.clone()));
+ }
+
+ /// Owned datums view and resolve back to themselves, holding their bytes
+ #[test]
+ fn owned_datums_round_trip_through_views() {
+ let att = rel_attr(1, "x", INT4OID, 4, 'i');
+ let pointer = ToastPointer {
+ va_rawsize: 8,
+ va_extinfo: 4,
+ va_valueid: 1,
+ va_toastrelid: 9,
+ };
+ let cases = [
+ (Datum::Null, 0),
+ (build::int4(3), 4),
+ (build::text("ab"), 2),
+ (
+ Datum::External(pointer),
+ std::mem::size_of::(),
+ ),
+ (Datum::Cstring(b"c".to_vec()), 1),
+ (Datum::BadTag(b"de".to_vec()), 2),
+ (Datum::Text(b"f".to_vec()), 1),
+ (Datum::Send(b"gh".to_vec()), 2),
+ ];
+ for (d, bytes) in cases {
+ assert_eq!(resolve(&att, d.view()).as_ref(), Some(&d));
+ assert_eq!(d.approx_bytes(), bytes, "{d:?}");
+ }
+ }
+
+ #[test]
+ fn text_datum_parses_local_types() {
+ let fixed = |b: &[u8]| TextDatum::Datum(Datum::fixed(b));
+ let cases: [(u32, &str, TextDatum); 8] = [
+ (CHAROID, "q", fixed(b"q")),
+ (INT2OID, "-2", fixed(&(-2i16).to_le_bytes())),
+ (FLOAT4OID, "1.5", fixed(&1.5f32.to_le_bytes())),
+ (FLOAT8OID, "-0.25", fixed(&(-0.25f64).to_le_bytes())),
+ (OIDOID, "16384", fixed(&16384u32.to_le_bytes())),
+ (
+ MACADDR8OID,
+ "08:00:2b:01:02:03:04:05",
+ fixed(&[8, 0, 0x2b, 1, 2, 3, 4, 5]),
+ ),
+ (
+ TIDOID,
+ "(1,2)",
+ fixed(&crate::decode::codecs::encode_tid(1, 2)),
+ ),
+ (INT2OID, "70000", TextDatum::Null),
+ ];
+ for (oid, text, want) in cases {
+ assert_eq!(text_datum(oid, text.as_bytes()), want, "{oid} {text}");
+ }
+ assert_eq!(text_datum(INT4OID, &[0xFF]), TextDatum::Opaque);
+ }
+
/// Raw and text both render, so jsonb equivalence is by document
#[test]
fn fast_default_equivalent_across_raw_and_text_jsonb() {
@@ -2617,7 +2589,8 @@ mod tests {
}
#[test]
- fn local_matrix_agrees_with_decoder() {
+ fn local_matrix_agrees_with_codecs() {
+ use crate::emit::ch_emitter::Codec;
let fixed: [(u32, i16); 26] = [
(BOOLOID, 1),
(CHAROID, 1),
@@ -2648,20 +2621,14 @@ mod tests {
];
for (oid, len) in fixed {
let att = rel_attr(1, "c", oid, len, 'i');
- let buf = vec![0u8; len as usize];
- let (v, _) = decode_one_value(&att, &buf, 0).expect("decodes");
- let raw = matches!(
- v,
- ColumnValue::PgPending { .. } | ColumnValue::Unsupported { .. }
- );
assert_eq!(
- !raw,
+ Codec::choose(&att) != Codec::Pending,
local_matrix_covers(oid, len),
"oid {oid} typlen {len}"
);
}
- let bodies: [(u32, Vec); 9] = [
+ let varlena: [(u32, Vec); 9] = [
(BYTEAOID, vec![1, 2]),
(TEXTOID, b"x".to_vec()),
(VARCHAROID, b"x".to_vec()),
@@ -2682,13 +2649,13 @@ mod tests {
(JSONOID, b"{}".to_vec()),
(3802, vec![0, 0, 0, 0]),
];
- for (oid, body) in bodies {
- let v = varlena_to_value(oid, Cow::Owned(body));
- let raw = matches!(
- v,
- ColumnValue::PgPending { .. } | ColumnValue::Unsupported { .. }
+ for (oid, _) in varlena {
+ let att = rel_attr(1, "c", oid, -1, 'i');
+ assert_eq!(
+ Codec::choose(&att) != Codec::Pending,
+ local_matrix_covers(oid, -1),
+ "varlena oid {oid}"
);
- assert_eq!(!raw, local_matrix_covers(oid, -1), "varlena oid {oid}");
}
}
@@ -2715,112 +2682,6 @@ mod tests {
assert!(!is_replica_identity_attr(&idx, 1));
}
- #[test]
- fn key_change_tombstone_matrix() {
- let tuple = |id: i32, v: &str| DecodedTuple {
- columns: vec![
- Some(ColumnValue::Int4(id)),
- Some(ColumnValue::Text(v.into())),
- ],
- partial: false,
- };
- let heap = |op, old: Option| DecodedHeap {
- rfn: RelFileNode::default(),
- xid: 7,
- source_lsn: 0x100,
- op,
- new: Some(tuple(2, "b")),
- old,
- };
- let mut rel = descriptor(1, vec![]);
- let pk = Some(vec![1]);
- let cases = [
- (
- ReplIdent::Default {
- pk_attnums: pk.clone(),
- },
- HeapOp::Update,
- Some(tuple(1, "")),
- true,
- ),
- (
- ReplIdent::Default {
- pk_attnums: pk.clone(),
- },
- HeapOp::Update,
- None,
- false,
- ),
- // PostgreSQL logs TOASTed keys even when unchanged
- (
- ReplIdent::Default {
- pk_attnums: pk.clone(),
- },
- HeapOp::Update,
- Some(tuple(2, "")),
- false,
- ),
- (
- ReplIdent::Full {
- pk_attnums: pk.clone(),
- },
- HeapOp::Update,
- Some(tuple(2, "a")),
- false,
- ),
- (
- ReplIdent::Full {
- pk_attnums: pk.clone(),
- },
- HeapOp::HotUpdate,
- Some(tuple(1, "a")),
- false,
- ),
- (
- ReplIdent::Full { pk_attnums: None },
- HeapOp::HotUpdate,
- Some(tuple(2, "a")),
- true,
- ),
- (
- ReplIdent::Full { pk_attnums: None },
- HeapOp::Update,
- Some(tuple(2, "b")),
- false,
- ),
- (
- ReplIdent::Full { pk_attnums: None },
- HeapOp::Delete,
- Some(tuple(2, "a")),
- false,
- ),
- (
- ReplIdent::UsingIndex {
- index_oid: 9,
- key_attnums: vec![2],
- },
- HeapOp::Update,
- Some(tuple(2, "a")),
- true,
- ),
- ];
- for (replident, op, old, split) in cases {
- rel.replident = replident;
- let mut h = heap(op, old.clone());
- let tomb = take_key_change_tombstone(&mut h, &rel);
- assert_eq!(tomb.is_some(), split, "{:?} {op:?} {old:?}", rel.replident);
- let Some(tomb) = tomb else {
- assert_eq!(h.old, old);
- continue;
- };
- assert_eq!(
- (tomb.op, tomb.source_lsn, tomb.xid),
- (HeapOp::Delete, 0xff, 7)
- );
- assert_eq!((tomb.old, tomb.new, h.old), (old, None, None));
- }
- }
-
#[test]
fn heap_op_index_maps_every_op() {
let (heap, heap2) = (RmId::Heap as u8, RmId::Heap2 as u8);
@@ -2989,10 +2850,7 @@ mod tests {
.new
.unwrap();
assert!(new.partial);
- assert_eq!(
- new.columns,
- vec![Some(ColumnValue::Int4(5)), None, None, None]
- );
+ assert_eq!(new.columns, vec![Some(build::int4(5)), None, None, None]);
// Truncate suffixlen
let rec = record_with(RmId::Heap, XLOG_HEAP_UPDATE, main_data, vec![12]);
let r = decode_heap_record(&rec, 0, &rel);
@@ -3040,18 +2898,18 @@ mod tests {
.new
.unwrap();
assert!(new.partial);
- assert_eq!(new.columns, vec![Some(ColumnValue::Int4(9)), None, None]);
+ assert_eq!(new.columns, vec![Some(build::int4(9)), None, None]);
}
#[test]
fn varlena_alignment_skips_zero_pad_per_typalign() {
let mut long_body = ((64u32) << 2).to_le_bytes().to_vec();
long_body.extend_from_slice(&[b'x'; 60]);
- let cases: [(char, Vec, ColumnValue); 3] = [
- ('s', short_varlena(b"hi"), ColumnValue::Text("hi".into())),
- ('d', short_varlena(b"hi"), ColumnValue::Text("hi".into())),
+ let cases: [(char, Vec, Datum); 3] = [
+ ('s', short_varlena(b"hi"), build::text("hi")),
+ ('d', short_varlena(b"hi"), build::text("hi")),
// Character alignment needs no padding; leading zero belongs to header
- ('c', long_body, ColumnValue::Text("x".repeat(60))),
+ ('c', long_body, build::text("x".repeat(60))),
];
for (align, value, expect) in cases {
let rel = descriptor(
@@ -3079,7 +2937,7 @@ mod tests {
assert!(!new.partial, "align {align}");
assert_eq!(
new.columns,
- vec![Some(ColumnValue::Bool(true)), Some(expect)],
+ vec![Some(build::bool_(true)), Some(expect)],
"align {align}"
);
}
@@ -3112,13 +2970,7 @@ mod tests {
let indirect = vec![0x01, 1, 0xAA, 0xBB];
assert_eq!(
decode_one_value(&text, &indirect, 0).unwrap(),
- (
- ColumnValue::Unsupported {
- type_oid: TEXTOID,
- raw: indirect.clone(),
- },
- 4
- )
+ (Datum::BadTag(indirect.clone()), 4)
);
}
@@ -3127,14 +2979,14 @@ mod tests {
let cstring = rel_attr(1, "c", 2275, -2, 'c');
assert_eq!(
decode_one_value(&cstring, b"ab\0", 0).unwrap(),
- (ColumnValue::Text("ab".into()), 3)
+ (Datum::Cstring(b"ab".to_vec()), 3)
);
let dropped = |type_len, align| {
let mut att = rel_attr(1, "gone", TEXTOID, type_len, align);
att.dropped = true;
att
};
- let null3 = (ColumnValue::Null, 3);
+ let null3 = (Datum::Null, 3);
assert_eq!(
decode_one_value(&dropped(-1, 'i'), &short_varlena(b"zz"), 0).unwrap(),
null3
@@ -3167,27 +3019,6 @@ mod tests {
),
"{r:?}"
);
-
- let unsupported = |att: &RelAttr, buf: &[u8]| {
- let (v, _) = decode_one_value(att, buf, 0).unwrap();
- assert!(
- matches!(v, ColumnValue::Unsupported { type_oid, .. } if type_oid == att.type_oid),
- "{}: {v:?}",
- att.type_oid
- );
- };
- unsupported(&rel_attr(1, "n", NAMEOID, 64, 'c'), &[0xFF; 64]);
- // PostgreSQL intervals require 16 bytes
- unsupported(&rel_attr(1, "iv", INTERVALOID, 8, 'd'), &[0; 8]);
- unsupported(
- &rel_attr(1, "num", NUMERICOID, -1, 'i'),
- &short_varlena(&[1]),
- );
- unsupported(&rel_attr(1, "ip", INETOID, -1, 'i'), &short_varlena(&[9]));
- unsupported(
- &rel_attr(1, "j", JSONBOID, -1, 'i'),
- &short_varlena(&[0xFF; 4]),
- );
}
#[test]
@@ -3195,68 +3026,54 @@ mod tests {
assert!(local_matrix_covers(2275, -2));
}
- #[test]
- fn take_toast_chunk_columns_rejects_misshapen_rows() {
- let good = || {
- vec![
- Some(ColumnValue::Oid(7)),
- Some(ColumnValue::Int4(2)),
- Some(ColumnValue::Bytea(vec![1])),
- ]
- };
- assert_eq!(take_toast_chunk_columns(&mut good()), Some((7, 2, vec![1])));
- assert_eq!(take_toast_chunk_columns(&mut good()[..2].to_vec()), None);
- for slot in 0..3 {
- let mut cols = good();
- cols[slot] = Some(ColumnValue::Text("x".into()));
- assert_eq!(take_toast_chunk_columns(&mut cols), None, "slot {slot}");
- }
- }
-
#[test]
fn fast_default_raw_honours_dataoffset_and_rejects_multidim() {
let mut att = rel_attr(1, "x", INT4OID, 4, 'i');
let mut arr = raw_missing_array(INT4OID, &77i32.to_le_bytes());
arr[8..12].copy_from_slice(&24i32.to_le_bytes());
att.missing_default = Some(MissingDefault::Raw(arr.clone()));
- assert_eq!(missing_value_for(&att), ColumnValue::Int4(77));
+ assert_eq!(missing_value_for(&att), build::int4(77));
arr[4..8].copy_from_slice(&2i32.to_le_bytes());
att.missing_default = Some(MissingDefault::Raw(arr));
- assert_eq!(missing_value_for(&att), ColumnValue::Null);
+ assert_eq!(missing_value_for(&att), Datum::Null);
}
#[test]
- fn missing_value_text_scalar_matrix() {
+ fn text_datum_scalar_matrix() {
+ let d = TextDatum::Datum;
let cases = [
- (CHAROID, "a", ColumnValue::Char(b'a' as i8)),
- (CHAROID, "", ColumnValue::Null),
- (INT2OID, "-7", ColumnValue::Int2(-7)),
- (INT2OID, "70000", ColumnValue::Null),
- (FLOAT4OID, "1.5", ColumnValue::Float4(1.5)),
- (FLOAT4OID, "x", ColumnValue::Null),
- (FLOAT8OID, "-2.25", ColumnValue::Float8(-2.25)),
- (FLOAT8OID, "x", ColumnValue::Null),
- (OIDOID, "4294967295", ColumnValue::Oid(u32::MAX)),
- (OIDOID, "-1", ColumnValue::Null),
+ (CHAROID, "a", d(Datum::fixed(b"a"))),
+ (CHAROID, "", TextDatum::Null),
+ (INT2OID, "-7", d(build::int2(-7))),
+ (INT2OID, "70000", TextDatum::Null),
+ (FLOAT4OID, "1.5", d(Datum::fixed(&1.5f32.to_le_bytes()))),
+ (FLOAT4OID, "x", TextDatum::Null),
+ (
+ FLOAT8OID,
+ "-2.25",
+ d(Datum::fixed(&(-2.25f64).to_le_bytes())),
+ ),
+ (FLOAT8OID, "x", TextDatum::Null),
+ (OIDOID, "4294967295", d(build::oid(u32::MAX))),
+ (OIDOID, "-1", TextDatum::Null),
(
MACADDR8OID,
"08:00:2b:ff:fe:01:02:03",
- ColumnValue::MacAddr8([0x08, 0x00, 0x2b, 0xff, 0xfe, 0x01, 0x02, 0x03]),
+ d(Datum::fixed(&[
+ 0x08, 0x00, 0x2b, 0xff, 0xfe, 0x01, 0x02, 0x03,
+ ])),
),
- (MACADDR8OID, "08:00:2b:01:02:03", ColumnValue::Null),
+ (MACADDR8OID, "08:00:2b:01:02:03", TextDatum::Null),
(
TIDOID,
"(3,4)",
- ColumnValue::Tid {
- block: 3,
- offset: 4,
- },
+ d(Datum::fixed(&crate::decode::codecs::encode_tid(3, 4))),
),
- (TIDOID, "3,4", ColumnValue::Null),
+ (TIDOID, "3,4", TextDatum::Null),
];
for (type_oid, text, expect) in cases {
assert_eq!(
- missing_value_from_text(type_oid, text),
+ text_datum(type_oid, text.as_bytes()),
expect,
"{type_oid} {text:?}"
);
diff --git a/src/decode/mod.rs b/src/decode/mod.rs
index 33c9b9e2..271ab5ef 100644
--- a/src/decode/mod.rs
+++ b/src/decode/mod.rs
@@ -1,4 +1,5 @@
pub mod codecs;
+pub mod datum;
pub mod decoder_sink;
pub mod fpi;
pub mod heap_decoder;
diff --git a/src/emit/ch_ddl.rs b/src/emit/ch_ddl.rs
index 728dc657..151b5ed4 100644
--- a/src/emit/ch_ddl.rs
+++ b/src/emit/ch_ddl.rs
@@ -40,7 +40,8 @@ use crate::column_rules::ColumnRules;
use crate::config::{ConfigResolver, DestEmitter, ResolvedConfig};
#[cfg(feature = "test-support")]
use crate::crash_test;
-use crate::decode::heap_decoder::{self, ColumnValue};
+use crate::decode::datum::DatumRef;
+use crate::decode::heap_decoder;
use crate::emit::ch_emitter::{DATETIME64_PG_EPOCH_US, EmitterConfig};
use crate::emit::route::{Routing, routing};
use crate::mapping::{
@@ -1483,14 +1484,24 @@ async fn resolve_disk_default<'a>(
db: Oid,
att: &'a RelAttr,
) -> Cow<'a, RelAttr> {
- let ColumnValue::PgPending { raw, .. } = heap_decoder::missing_value_for(att) else {
+ if heap_decoder::local_matrix_covers(att.type_oid, att.type_len) {
+ return Cow::Borrowed(att);
+ }
+ let (DatumRef::Fixed(raw) | DatumRef::Varlena(raw) | DatumRef::Cstring(raw)) =
+ heap_decoder::missing_datum(att)
+ else {
return Cow::Borrowed(att);
};
let Some(oracle) = oracle else {
return Cow::Borrowed(att);
};
match oracle
- .text_value(db, att.type_oid, att.typmod, OracleCell::DiskRaw(raw))
+ .text_value(
+ db,
+ att.type_oid,
+ att.typmod,
+ OracleCell::DiskRaw(raw.to_vec()),
+ )
.await
{
Ok(text) => {
@@ -1774,7 +1785,7 @@ mod tests {
let sql = "CREATE DATABASE IF NOT EXISTS retry_test";
for (retries, idle) in [(0, false), (1, false), (2, false), (1, true)] {
let (config, server) =
- crate::ch::test_support::retry_server(retries, sql, false, idle).await;
+ crate::ch::test_support::retry_server(retries, sql, 0, idle, None).await;
let resolved = Arc::new(ResolvedConfig::default());
let ddl = DdlConfig::from_resolved(
&resolved,
diff --git a/src/emit/ch_emitter.rs b/src/emit/ch_emitter.rs
index 0fec3f0b..73431ae7 100644
--- a/src/emit/ch_emitter.rs
+++ b/src/emit/ch_emitter.rs
@@ -34,9 +34,13 @@ use clickhouse_c::{Allocator, ColumnBuilder, Kind, TypeAst};
use crate::ch::is_retryable;
use crate::ch::{CompressionChoice, ConnectionConfig, EmitterError, quote_ident, types};
use crate::column_rules::{ColumnEntry, ColumnRule, ColumnRules, Substitute};
+use crate::decode::codecs::{Numeric, parse_numeric};
+use crate::decode::datum::{Datum, DatumRef};
#[cfg(test)]
use crate::decode::decoder_sink::DecoderSinkError;
-use crate::decode::heap_decoder::{ColumnValue, CommittedTuple, HeapOp};
+#[cfg(any(test, feature = "test-support"))]
+use crate::decode::heap_decoder::{CommittedTuple, HeapOp};
+use crate::decode::heap_decoder::{TextDatum, inflate, missing_datum, text_datum};
use crate::mapping::{
ColumnMapping, DropTableStrategy, NamespaceMapping, SystemColumnNames, SystemColumns,
TableMapping, TableTarget, named_instance,
@@ -52,6 +56,7 @@ use crate::source::queueing_record_sink::{
};
use crate::table_rules::{MatchKind, PartitionRoute, TableRule};
use foldhash::{HashMap, HashMapExt};
+use smallvec::SmallVec;
/// Microseconds between PG `TimestampTz` epoch (2000-01-01 UTC) and Unix
/// epoch. CH `DateTime64(6)` is Unix microseconds; PG commit-record
@@ -123,6 +128,8 @@ pub fn default_database<'a>(
pub struct EmitterConfig {
pub instances: BTreeMap,
pub conn: InstanceConfig,
+ /// Wire compression for every destination, so blocks seal once
+ pub compression: CompressionChoice,
/// Custom rustls roots/config for `secure` path: private CA, pinned
/// self-signed cert, or mTLS. `None` uses public webpki roots via
/// [`clickhouse_c::tls::default_config`]. Not parsed from TOML;
@@ -615,7 +622,7 @@ impl ConnectionConfig for EmitterConfig {
}
fn compression(&self) -> CompressionChoice {
- self.conn.compression
+ self.compression
}
fn idle_reconnect(&self) -> Duration {
@@ -786,8 +793,6 @@ pub struct InstanceConfig {
/// handshake still names what the certificate was issued for. Empty
/// uses `host`.
pub tls_server_name: String,
- #[serde(deserialize_with = "crate::toml_de::de_parse")]
- pub compression: CompressionChoice,
}
impl Default for InstanceConfig {
@@ -801,7 +806,6 @@ impl Default for InstanceConfig {
password: ch.password,
secure: ch.secure,
tls_server_name: ch.tls_server_name,
- compression: ch.compression,
}
}
}
@@ -1013,8 +1017,8 @@ impl EmitterConfig {
password: ch.password,
secure: ch.secure,
tls_server_name: ch.tls_server_name,
- compression: ch.compression,
},
+ compression: ch.compression,
tls_config: None,
row_budget: ch.row_budget,
byte_budget: ch.byte_budget,
@@ -1141,7 +1145,6 @@ impl EmitterConfig {
"ch.instances.{name}: host and port must be nonempty"
)));
}
- instance.compression.build_codec()?;
}
out.auto_create_name_all = prefix_all
.as_deref()
@@ -1429,20 +1432,104 @@ pub(crate) struct ColumnPlan {
pub(crate) datetime64_scale: u32,
pub(crate) non_finite: NonFinite,
pub(crate) encoding: ColumnEncoding,
+ /// Source attribute feeding this column, `None` when descriptor holds
+ /// it dropped
+ pub(crate) source: Option,
+}
+
+/// Per-row handling of a source attribute, compiled with its target
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub(crate) struct SourceColumn {
+ /// attnum - 1, indexes tuple images
+ pub(crate) index: usize,
+ pub(crate) codec: Codec,
+ /// Zero for values patched past the descriptor's attributes
+ pub(crate) type_oid: u32,
+}
+
+/// Conversion of one source type's stored bytes into its column
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub(crate) enum Codec {
+ /// Little-endian storage copied as the column holds it
+ Le,
+ Date,
+ Time,
+ /// `timestamp` and `timestamptz`
+ Timestamp,
+ TimeTz,
+ Uuid,
+ Name,
+ /// Body bytes as String: text, bytea, json, cstring
+ Bytes,
+ Numeric,
+ Jsonb,
+ Inet,
+ Cidr,
+ MacAddr,
+ MacAddr8,
+ Tid,
+ Interval,
+ /// PostGIS point rendered as WKT, typoutput yields HEXEWKB
+ Wkt,
+ /// No local decode: typoutput text lands as String, stored bytes fail
+ Pending,
+}
+
+impl Codec {
+ pub(crate) fn choose(att: &RelAttr) -> Self {
+ use crate::schema::{
+ BOOLOID, BPCHAROID, BYTEAOID, CHAROID, CIDOID, CIDROID, DATEOID, FLOAT4OID, FLOAT8OID,
+ INETOID, INT2OID, INT4OID, INT8OID, INTERVALOID, JSONBOID, JSONOID, MACADDR8OID,
+ MACADDROID, MONEYOID, NAMEOID, NUMERICOID, OID8OID, OIDOID, PG_LSNOID, TEXTOID, TIDOID,
+ TIMEOID, TIMESTAMPOID, TIMESTAMPTZOID, TIMETZOID, UUIDOID, VARCHAROID, XID8OID, XIDOID,
+ };
+ if matches!(att.type_name.as_str(), "geography" | "geometry") {
+ return Self::Wkt;
+ }
+ if att.type_len == -2 {
+ return Self::Bytes;
+ }
+ match att.type_oid {
+ BOOLOID | CHAROID | INT2OID | INT4OID | INT8OID | MONEYOID | FLOAT4OID | FLOAT8OID
+ | OIDOID | XIDOID | CIDOID | OID8OID | XID8OID | PG_LSNOID => Self::Le,
+ DATEOID => Self::Date,
+ TIMEOID => Self::Time,
+ TIMESTAMPOID | TIMESTAMPTZOID => Self::Timestamp,
+ TIMETZOID => Self::TimeTz,
+ UUIDOID => Self::Uuid,
+ NAMEOID => Self::Name,
+ BYTEAOID | TEXTOID | VARCHAROID | BPCHAROID | JSONOID => Self::Bytes,
+ NUMERICOID => Self::Numeric,
+ JSONBOID => Self::Jsonb,
+ INETOID => Self::Inet,
+ CIDROID => Self::Cidr,
+ MACADDROID => Self::MacAddr,
+ MACADDR8OID => Self::MacAddr8,
+ TIDOID => Self::Tid,
+ INTERVALOID => Self::Interval,
+ _ => Self::Pending,
+ }
+ }
}
/// Values replacing non-finite numerics or temporal infinities, `None` rejects
#[derive(Clone, Debug, Default, PartialEq)]
pub(crate) struct NonFinite {
- nan: Option,
- pos_inf: Option,
- neg_inf: Option,
+ nan: Option,
+ pos_inf: Option,
+ neg_inf: Option,
+}
+
+/// Substitute cell, encoded for its column
+#[derive(Clone, Debug, PartialEq)]
+enum Fill {
+ Null,
+ Le(SmallVec<[u8; 32]>),
}
impl NonFinite {
/// Resolve settings against Decimal or temporal destination once per plan
fn compile(rule: &ColumnRule, ast: &TypeAst) -> Result {
- use crate::decode::codecs::NumericKind;
let nullable = ast.view().kind() == Some(Kind::Nullable);
let inner = types::strip_nullable(ast.view());
let decimal = decimal_wire_of(ast);
@@ -1456,19 +1543,19 @@ impl NonFinite {
let cell = |key: &str, sub: &Option| {
let text = match sub {
None | Some(Substitute::Reject) => return Ok(None),
- Some(Substitute::Null) if nullable => return Ok(Some(ColumnValue::Null)),
+ Some(Substitute::Null) if nullable => return Ok(Some(Fill::Null)),
Some(Substitute::Null) => {
return Err(format!("{key} = \"null\" needs a Nullable destination"));
}
- Some(_) if decimal.is_none() => {
- return Err(format!(
- "{key}: temporal infinity supports only reject or null"
- ));
- }
Some(Substitute::Min) => format!("-{bound}"),
Some(Substitute::Max) => bound.clone(),
Some(Substitute::Literal(lit)) => lit.clone(),
};
+ let Some(decimal) = decimal else {
+ return Err(format!(
+ "{key}: temporal infinity supports only reject or null"
+ ));
+ };
let (int, frac) = crate::column_rules::decimal_parts(&text).unwrap_or_default();
if frac.len() > scale && frac[scale..].bytes().any(|b| b != b'0') {
return Err(format!(
@@ -1480,7 +1567,11 @@ impl NonFinite {
"{key} = {text:?}: exceeds Decimal({precision}, {scale})"
));
}
- Ok(Some(ColumnValue::Numeric(NumericKind::Finite(text))))
+ let scaled = decimal_text_to_scaled_le(&text, decimal.scale, decimal.width)
+ .map_err(|e| format!("{key} = {text:?}: {e}"))?;
+ Ok(Some(Fill::Le(SmallVec::from_slice(
+ &scaled[..decimal.width.bytes()],
+ ))))
};
Ok(Self {
// Temporal values have no NaN
@@ -1494,21 +1585,29 @@ impl NonFinite {
})
}
- fn substitute<'a>(&'a self, v: &'a ColumnValue) -> &'a ColumnValue {
- use crate::decode::codecs::NumericKind;
- let sub = match v {
- ColumnValue::Numeric(NumericKind::NaN) => &self.nan,
- ColumnValue::Numeric(NumericKind::PInf)
- | ColumnValue::Date(i32::MAX)
- | ColumnValue::Timestamp(i64::MAX)
- | ColumnValue::TimestampTz(i64::MAX) => &self.pos_inf,
- ColumnValue::Numeric(NumericKind::NInf)
- | ColumnValue::Date(i32::MIN)
- | ColumnValue::Timestamp(i64::MIN)
- | ColumnValue::TimestampTz(i64::MIN) => &self.neg_inf,
- _ => return v,
+ /// Substitute for a non-finite `body`, `None` to encode it as is
+ fn fill(&self, codec: Codec, body: &[u8]) -> Option<&Fill> {
+ use crate::decode::codecs::Special;
+ let sub = match codec {
+ Codec::Numeric => match parse_numeric(body) {
+ Ok(Numeric::Special(Special::NaN)) => &self.nan,
+ Ok(Numeric::Special(Special::PInf)) => &self.pos_inf,
+ Ok(Numeric::Special(Special::NInf)) => &self.neg_inf,
+ _ => return None,
+ },
+ Codec::Date => match i32::from_le_bytes(body.try_into().ok()?) {
+ i32::MAX => &self.pos_inf,
+ i32::MIN => &self.neg_inf,
+ _ => return None,
+ },
+ Codec::Timestamp => match i64::from_le_bytes(body.try_into().ok()?) {
+ i64::MAX => &self.pos_inf,
+ i64::MIN => &self.neg_inf,
+ _ => return None,
+ },
+ _ => return None,
};
- sub.as_ref().unwrap_or(v)
+ sub.as_ref()
}
}
@@ -1668,6 +1767,7 @@ impl TablePlan {
ast,
decimal,
non_finite: NonFinite::default(),
+ source: None,
};
let settings = att
.map(|a| column_rules.settings(owner, &a.name))
@@ -1686,6 +1786,7 @@ impl TablePlan {
ast: oast,
decimal,
non_finite: NonFinite::default(),
+ source: None,
};
} else {
tracing::warn!(
@@ -1731,6 +1832,21 @@ impl TablePlan {
if plan.encoding != ColumnEncoding::Local && plan.non_finite != NonFinite::default() {
return Err(config_err("substitution needs local encoding"));
}
+ // Attnums past the descriptor still source patched values
+ plan.source = usize::try_from(c.src_attnum - 1)
+ .ok()
+ .and_then(|index| match att {
+ Some(a) => Some(SourceColumn {
+ index,
+ codec: Codec::choose(a),
+ type_oid: a.type_oid,
+ }),
+ None => (index >= rel.attributes.len()).then_some(SourceColumn {
+ index,
+ codec: Codec::Le,
+ type_oid: 0,
+ }),
+ });
columns.push(plan);
col_sql.push(quote_ident(&c.target_name));
}
@@ -1742,6 +1858,7 @@ impl TablePlan {
datetime64_scale: 0,
non_finite: NonFinite::default(),
encoding: ColumnEncoding::Local,
+ source: None,
};
let synth_lsn = mk(&system.lsn, "UInt64");
let synth_xid = mk(&system.xid, "UInt32");
@@ -1786,6 +1903,10 @@ pub(crate) enum Append {
Full,
}
+/// One row's mapped datums, inline up to wide tables so encoding a row
+/// does not allocate
+type RowCells<'a> = SmallVec<[DatumRef<'a>; 64]>;
+
/// Per-table per-xact accumulator, one block buffer per CH column.
pub(crate) struct TableEncoder {
pub(crate) plan: TablePlan,
@@ -1797,6 +1918,8 @@ pub(crate) struct TableEncoder {
oracle_cells: Vec