Skip to content

Native encoding - #305

Draft
serprex wants to merge 20 commits into
mainfrom
native-encoding
Draft

serprex wants to merge 20 commits into
mainfrom
native-encoding

Conversation

@serprex

@serprex serprex commented Nov 7, 2026

Copy link
Copy Markdown
Member

serprex and others added 19 commits November 18, 2026 04:18
Wrap mimalloc in a counting global allocator so xact_spill, spool_read and
bootstrap_pump report allocation count, retained bytes and peak live bytes
per stage. Plan and deferred replay compare every row's values; bootstrap
pump compares a digest of decoded values against written pages and drain
asserts routed row count.

Baseline (more-types, defaults unless noted):
  xact_spill --buffer-bytes 1048576: 950k rows/s, buffer peak 1.62 MB,
    23.6k allocs per 7812-row xact, drain 21.1k allocs
  spool_read --plan-only --planner --mapped --relations 8
    --control-every 10000 --rows 200000: plan 400k allocs, replay+place
    400k allocs, replay peak 2.13 MB
  bootstrap_pump: pump 18.5M tuples/s, 822k allocs, peak 15.8 MiB;
    drain 14.2M tuples/s, 829k allocs, peak 3.5 MiB

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split heap tuple walking from owned cell construction. One generic walker
feeds either DecodedTuple cells or a TupleView of DatumRefs borrowing the
record: fixed bytes, inline or compressed varlena bodies, external
pointers, SQL NULL, absent and missing attributes. Multi-insert tuples no
longer copy into a synthesized header buffer.

Encoding policy now runs over ValueRef, a borrowed mirror of ColumnValue,
through one shared cell core. append_routed adapts owned cells to it; a
test-only append_view feeds it from datums, materializing only fast
defaults, compressed bodies, numeric, jsonb and PostGIS renders. Each
column plan records its source attribute and compiles a Direct choice:
copy little-endian bytes only when source type and target storage kind
match exactly, else decode through shared policy.

Production ownership is unchanged. Parity tests compare blocks and errors
from both paths across more-types additions, float bits, epochs, numeric
and jsonb, compressed and external values, defaults, dropped and absent
attributes, overrides, substitutes and oracle columns; every heap decoder
test also checks views build equal cells.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covered user-heap records no longer decode into ColumnValue rows at record
time. HeapInput keeps writer xid, op, LSN, tuple count, main_data and block
0 registered data in one allocation; images and other blocks drop since
wal_level=logical keeps tuple data registered. DescribedInput pins the
record-time descriptor version as DescribedHeap does and shares the spill
descriptor dictionary under new TAG_INPUT frames (spill v10).

Record time still walks each tuple without building cells, so malformed
records fail and per-op decoder counts move there as before. Commit drain
decodes inputs through the cell adapter into the existing pending-heap
queue; unresolved catalog stash keeps its RawRecord path and verdicts.
TOAST relations and runtime-config tables still decode at record time.

Predecoded row graphs are gone. The dispatcher prescan borrows record
bytes to start TOAST read-ahead for kept values and record toast births,
and RecordSink::on_records drops its predecoded slice.

xact_spill now feeds real insert records: 15.8k buffer allocations per
7812-row transaction against 23.6k for decoded rows; decode cost moves
into drain until planning encodes directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Commit drain hands planning undecoded inputs (DrainHeap) instead of
decoding them into cells. Planner views each input's tuple images in
place and encodes them straight into per-table TableEncoder chunks;
inputs with external TOAST pointers, truncates, stash-resolved records
and directly buffered heaps go through the cell adapter and detoast as
before. Placement moves into planning: append-only routes drop deletes,
PostGIS cells render, key changes split into a delete marker at lsn-1
plus the new row, with views comparing stored bytes and falling back to
owned values so NaN and representation differences decide as before.

Plan spool v3 stores encoded chunks: segments seal at the drain batch
row/byte limits and before every control or truncate, spill to frames of
serialized column buffers, and replay as one seq each. Encoding errors
now fail the plan before transaction effects under reason "encode".
Batcher takes PlacedSegment messages and merges chunks into open
batches, rebasing string offsets and recomputing the oracle frame size;
a layout change seals the open batch first. Row placement (decode.rs)
is gone; bootstrap still sends cell rows until it encodes directly.

Tuple views store up to 32 datums inline and inputs view without building
an XLogRecord, so planning allocates nothing per row on the view path.
spool_read --planner --mapped --relations 8 --control-every 10000
--rows 200000 plans in ~0.10s with 39.8k allocations either way and
checks encoded rows against a cell-encoded reference plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batcher separates Native block size from INSERT size: an open INSERT seals
blocks at 1,048,576 rows, 16 MiB of column bytes, or when the oracle
request would overflow, while row, byte and deadline triggers still bound
the INSERT. Resolvers resolve each block, serialize it once through
clickhouse-c BlockBuilder::write, compress it with the configured
[ch] compression, drop slabs and oracle output, and shrink the output
permit to stored bytes. Bodies that do not shrink stay raw.

Inserters send every block of a batch as its own Data packet under one
query, opening one block at a time into a per-inserter scratch buffer;
retries resend all blocks in order and acknowledge sequence counts only
after EndOfStream from every destination. This is the bounded
decompression intermediate step: clickhouse-c-rs 0.2.2 has no send API
for prepared payloads.

New counters: walshadow_resident_seal_seconds_total and resident
raw/stored bytes, whose ratio is achieved compression.
inserter_encode_seconds_total now times decompress and decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Architecture now describes commits planned from WAL bytes into column
chunks, blocks merged across commits and grouped per INSERT, and resolved
blocks kept compressed until acknowledgement. The native encoding plan
keeps only unfinished work: direct encoding for COPY and heap-page loads,
detoast without cells, adopting prepared payload sends, memory bounds
across owners, and end-to-end acceptance against the more-types baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records resolved from the stash at commit now drain as raw record bytes
plus their resolved descriptor, the same shape live records already use

Planner and WAL replay encode both straight from tuple views, so the
owned-cell drain, spill, plan and replay branches go away
Sink and stash gather more from record headers instead of walking every tuple.
WAL replay now rejects partial images itself

Tuple views keep datums in Vec rather than 32-slot inline SmallVec,
so per-row copies shrink and plan futures stay small enough to await
without boxing. The inline-only planner fast path goes away
fall back to text otherwise

Avoid text parsing when reading column defaults from PostgreSQL

Wait for shadow recovery before connecting COPY tests to conversion worker

Clarify TOAST accounting comments to describe retained raw bytes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant