Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 25 additions & 12 deletions architecture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,25 +37,38 @@ planning committed transactions

## Commit pipeline

![Commit pipeline: reorder places rows inline, one batcher feeds a conversion pool and ClickHouse inserters, and acknowledgements advance a contiguous watermark](workers.svg)
![Commit pipeline: reorder plans column chunks, one batcher feeds a conversion and sealing pool, inserters send prepared blocks, and acknowledgements advance a contiguous watermark](workers.svg)

`BufferingDecoderSink` and `ReorderSink` share one record-queue worker
`ReorderSink` plans each commit and places its rows onto the batcher in order
Batcher seals per-table batches. A separate conversion pool resolves PostgreSQL
types before inserters send complete Native blocks. Conversion concurrency
matches oracle capacity; `[ch].inserter_pool_size` sizes the inserter pool
`ReorderSink` plans each commit and places it onto the batcher in order.
Planning routes, validates and detoasts rows but leaves them WAL bytes; each
outbox flush hands the rows of many commits to one encode job on the runtime,
which writes per-table column chunks. Batcher awaits jobs in order, merges
chunks into Native blocks and groups blocks into per-table INSERTs. A
separate conversion pool resolves PostgreSQL types, then serializes each block
once and keeps prepared bytes until its INSERT is durable. Compression follows
`[ch] compression`; inserters rebuild blocks only when a connection rejects
prepared bytes. Conversion concurrency matches oracle capacity;
`[ch].inserter_pool_size` sizes inserter pool
Each inserter owns a ClickHouse connection and takes any ready batch

Sequence numbers identify work slices, not necessarily whole transactions
Only a commit's final slice publishes its LSN, after all earlier work finishes
Bounded queues and a shared payload budget limit work in flight; transaction
and plan data can spill to disk

Queue worker runs on a dedicated thread and receives batches with heap decode
started ahead of dispatch. It checks decoded results against descriptor history
and decodes again if needed. Commits drain only their own transaction trees.
Consecutive small commits can share row chunks while retaining separate
acknowledgements
and plan data can spill to disk. Each tail's output pool holds batches from
batcher to acknowledgement and reserves seal workspace apart from them, so
conversion finishes however much output waits on inserters

Queue worker runs on a dedicated thread. A dispatcher scans each batch ahead of
it to start TOAST store reads for values updated rows keep. Ordinary heap
records stay WAL bytes under their record-time descriptor until commit;
planning reads tuples in place, resolved external values patching them.
Records resolved from stash at commit carry the same bytes under their resolved
descriptor. Compressed inline values, numeric, JSONB, and fast defaults can
need row scratch. Commits drain only their own
transaction trees. Consecutive small commits can share column blocks while
retaining separate acknowledgements. COPY and backup loads carry tuple bytes or
COPY fields and encode their own column chunks

## Design boundaries

Expand Down
2 changes: 1 addition & 1 deletion architecture/overview.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
22 changes: 19 additions & 3 deletions architecture/values.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,18 @@ acknowledgement alone is insufficient because restart could still reread a
pre-drop row. Rewrite barriers handle changed physical generations without
unconditionally wiping history needed by earlier reads

## Value forms

Values stay in forms PostgreSQL produced until they become ClickHouse Native
column bytes: stored datums read in place from WAL, page images and TOAST
chunks, or `typoutput` text from SQL-read fast defaults and COPY casts. COPY's
binary send forms convert to stored datums on arrival where a local codec reads
the type; types only PostgreSQL converts keep their send bytes for its
`typreceive`, falling back to text casts without binary I/O. Each destination
column compiles one conversion from its source type, so there is no decoded
intermediate value. Detoasted values hold their stored bytes, so resident
accounting counts what was fetched rather than a rendering

## Convert unfamiliar types

After decompression, values outside local codec set are sent to PostgreSQL
Expand All @@ -41,9 +53,13 @@ columns for destination insertion. Batch conversion avoids a database round trip
per row and keeps type semantics in PostgreSQL

Conversion workers run between batcher and inserters, so PostgreSQL conversion
overlaps ClickHouse insertion. Batches carrying only local columns pass through
without a PostgreSQL request. Inserters combine local and converted columns into
complete Native blocks and retain them across insert retries
overlaps ClickHouse insertion. Blocks carrying only local columns pass through
without a PostgreSQL request. Conversion workers combine local and converted
columns into complete Native blocks, serialize and compress each once, and
release column buffers. Inserters send prepared bytes directly. A connection
rejecting prepared bytes, such as one using different compression, gets one
block decoded and rebuilt at a time. Retries resend every block of an INSERT;
prepared blocks remain owned until every required destination acknowledges

Conversion failure stops batch with column and row context. Module pins output
settings to make conversion reproducible. Greenfield bootstrap uses a temporary
Expand Down
27 changes: 14 additions & 13 deletions architecture/values.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions architecture/workers.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 11 additions & 9 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,19 +32,21 @@ cargo bench --bench spool_read -- --plan-only --planner --mapped --rows 1000 --p
cargo bench --bench spool_read -- --plan-only --planner --mapped --rows 200000 --payload-bytes 256 --relations 8 --control-every 10000
```

Use `--planner` to run `Planner::plan_batch` before replay and place replayed rows
in batches. Use `--mapped` to assign frozen routes. By default, heaps have no
mapping: planning counts and discards them without writing them to spool.
Use `--planner` to run `Planner::plan_batch` over drained slices built before
timing starts. Rows enter as WAL insert records and encode from tuple views;
add `--cells` to plan decoded rows through the cell adapter instead. Use
`--mapped` to assign frozen routes over an int4 and a text column. By default,
heaps have no mapping: planning counts and discards them without encoding.
Use `--relations` to cycle through relation descriptors, one per row.
Use `--control-every` to insert unrelated schema controls and clear cached routes.
Use `--plan-only` to skip deferred spool measurements. Plans spill to disk when
they exceed default 1 MiB memory limit.
encoded chunks exceed default 1 MiB memory limit.

Read `plan_total` for combined write and replay time per sample, including row
placement with `--planner`. For plans in memory, byte counts estimate memory use;
for spilled plans, they report encoded file size. Route caching measures descriptor
identity checks and reuse of cached references. Measurements exclude mapping
resolution, WAL draining, TOAST I/O, control execution, and PostgreSQL/ClickHouse I/O.
Read `plan_total` for combined write and replay time per sample. For plans in
memory, byte counts report allocated chunk bytes; for spilled plans, encoded
file size. Route caching measures descriptor identity checks and reuse of cached
references. Measurements exclude mapping resolution, WAL draining, TOAST I/O,
control execution, and PostgreSQL/ClickHouse I/O.

## pgbench workload

Expand Down
Loading
Loading