Repository navigation
Native encoding - #305
Draft
serprex wants to merge 20 commits into
Draft
Native encoding#305serprex wants to merge 20 commits into
serprex wants to merge 20 commits into
Conversation
serprex
marked this pull request as draft
November 7, 2026 05:06
This was referenced Nov 7, 2026
serprex
force-pushed
the
native-encoding
branch
5 times, most recently
from
November 10, 2026 14:12
ad8db5a to
c75ac62
Compare
Wrap mimalloc in a counting global allocator so xact_spill, spool_read and
bootstrap_pump report allocation count, retained bytes and peak live bytes
per stage. Plan and deferred replay compare every row's values; bootstrap
pump compares a digest of decoded values against written pages and drain
asserts routed row count.
Baseline (more-types, defaults unless noted):
xact_spill --buffer-bytes 1048576: 950k rows/s, buffer peak 1.62 MB,
23.6k allocs per 7812-row xact, drain 21.1k allocs
spool_read --plan-only --planner --mapped --relations 8
--control-every 10000 --rows 200000: plan 400k allocs, replay+place
400k allocs, replay peak 2.13 MB
bootstrap_pump: pump 18.5M tuples/s, 822k allocs, peak 15.8 MiB;
drain 14.2M tuples/s, 829k allocs, peak 3.5 MiB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split heap tuple walking from owned cell construction. One generic walker feeds either DecodedTuple cells or a TupleView of DatumRefs borrowing the record: fixed bytes, inline or compressed varlena bodies, external pointers, SQL NULL, absent and missing attributes. Multi-insert tuples no longer copy into a synthesized header buffer. Encoding policy now runs over ValueRef, a borrowed mirror of ColumnValue, through one shared cell core. append_routed adapts owned cells to it; a test-only append_view feeds it from datums, materializing only fast defaults, compressed bodies, numeric, jsonb and PostGIS renders. Each column plan records its source attribute and compiles a Direct choice: copy little-endian bytes only when source type and target storage kind match exactly, else decode through shared policy. Production ownership is unchanged. Parity tests compare blocks and errors from both paths across more-types additions, float bits, epochs, numeric and jsonb, compressed and external values, defaults, dropped and absent attributes, overrides, substitutes and oracle columns; every heap decoder test also checks views build equal cells. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covered user-heap records no longer decode into ColumnValue rows at record time. HeapInput keeps writer xid, op, LSN, tuple count, main_data and block 0 registered data in one allocation; images and other blocks drop since wal_level=logical keeps tuple data registered. DescribedInput pins the record-time descriptor version as DescribedHeap does and shares the spill descriptor dictionary under new TAG_INPUT frames (spill v10). Record time still walks each tuple without building cells, so malformed records fail and per-op decoder counts move there as before. Commit drain decodes inputs through the cell adapter into the existing pending-heap queue; unresolved catalog stash keeps its RawRecord path and verdicts. TOAST relations and runtime-config tables still decode at record time. Predecoded row graphs are gone. The dispatcher prescan borrows record bytes to start TOAST read-ahead for kept values and record toast births, and RecordSink::on_records drops its predecoded slice. xact_spill now feeds real insert records: 15.8k buffer allocations per 7812-row transaction against 23.6k for decoded rows; decode cost moves into drain until planning encodes directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Commit drain hands planning undecoded inputs (DrainHeap) instead of decoding them into cells. Planner views each input's tuple images in place and encodes them straight into per-table TableEncoder chunks; inputs with external TOAST pointers, truncates, stash-resolved records and directly buffered heaps go through the cell adapter and detoast as before. Placement moves into planning: append-only routes drop deletes, PostGIS cells render, key changes split into a delete marker at lsn-1 plus the new row, with views comparing stored bytes and falling back to owned values so NaN and representation differences decide as before. Plan spool v3 stores encoded chunks: segments seal at the drain batch row/byte limits and before every control or truncate, spill to frames of serialized column buffers, and replay as one seq each. Encoding errors now fail the plan before transaction effects under reason "encode". Batcher takes PlacedSegment messages and merges chunks into open batches, rebasing string offsets and recomputing the oracle frame size; a layout change seals the open batch first. Row placement (decode.rs) is gone; bootstrap still sends cell rows until it encodes directly. Tuple views store up to 32 datums inline and inputs view without building an XLogRecord, so planning allocates nothing per row on the view path. spool_read --planner --mapped --relations 8 --control-every 10000 --rows 200000 plans in ~0.10s with 39.8k allocations either way and checks encoded rows against a cell-encoded reference plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batcher separates Native block size from INSERT size: an open INSERT seals blocks at 1,048,576 rows, 16 MiB of column bytes, or when the oracle request would overflow, while row, byte and deadline triggers still bound the INSERT. Resolvers resolve each block, serialize it once through clickhouse-c BlockBuilder::write, compress it with the configured [ch] compression, drop slabs and oracle output, and shrink the output permit to stored bytes. Bodies that do not shrink stay raw. Inserters send every block of a batch as its own Data packet under one query, opening one block at a time into a per-inserter scratch buffer; retries resend all blocks in order and acknowledge sequence counts only after EndOfStream from every destination. This is the bounded decompression intermediate step: clickhouse-c-rs 0.2.2 has no send API for prepared payloads. New counters: walshadow_resident_seal_seconds_total and resident raw/stored bytes, whose ratio is achieved compression. inserter_encode_seconds_total now times decompress and decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Architecture now describes commits planned from WAL bytes into column chunks, blocks merged across commits and grouped per INSERT, and resolved blocks kept compressed until acknowledgement. The native encoding plan keeps only unfinished work: direct encoding for COPY and heap-page loads, detoast without cells, adopting prepared payload sends, memory bounds across owners, and end-to-end acceptance against the more-types baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records resolved from the stash at commit now drain as raw record bytes plus their resolved descriptor, the same shape live records already use Planner and WAL replay encode both straight from tuple views, so the owned-cell drain, spill, plan and replay branches go away
Sink and stash gather more from record headers instead of walking every tuple. WAL replay now rejects partial images itself Tuple views keep datums in Vec rather than 32-slot inline SmallVec, so per-row copies shrink and plan futures stay small enough to await without boxing. The inline-only planner fast path goes away
fall back to text otherwise Avoid text parsing when reading column defaults from PostgreSQL Wait for shadow recovery before connecting COPY tests to conversion worker Clarify TOAST accounting comments to describe retained raw bytes
serprex
force-pushed
the
native-encoding
branch
from
November 18, 2026 06:22
c75ac62 to
9f77ac5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
built on #304
requires ClickHouse/clickhouse-c-rs/pull/5