[POC for #9934] parquet: add ReverseSerializedPageReader

[zhuqi-lucas]

Which issue does this PR close?

Tracking issue: #9934 — parquet: Support reverse page iteration for reverse streaming with filter pushdown.

This PR is a proof-of-concept (Phase 1) for the building block proposed in that issue. Its purpose is to ground the design discussion in real code, demonstrate feasibility, and collect feedback on the API shape before any final API commitment.

Rationale for this change

DataFusion has merged a row-group-level reverse scan (apache/datafusion#18817) for ORDER BY DESC LIMIT N queries on ascending-sorted Parquet files. The current granularity (~128 MB per row group) means high first-batch latency and high peak memory; for WHERE filter ORDER BY DESC LIMIT N, where RowSelection cannot pre-select matching rows, the reverse stream is the only correct strategy.

A page-level reverse iterator is the missing primitive. The key insight (per #9934) is to separate two distinct concepts:

	Possible?
Reverse-decoding rows within a single page	❌ No (RLE / dict / delta / bit-packing are forward streams)
Iterating page traversal order in reverse, decoding each page forward	✅ Yes (via OffsetIndex.page_locations)

This PR implements the second.

What changes are included in this PR?

A small extension to the existing SerializedPageReader::Pages mode plus a builder flag, instead of a separate reader type. The earlier revision of this PR introduced a standalone ReverseSerializedPageReader; per @alamb's review, the two readers were ~95% duplicate (only pop_front vs pop_back differed), so they have been collapsed into a single code path.

let reader = SerializedPageReader::new(
    chunk_reader, meta, total_rows, Some(page_locations),
)?
.with_reverse_pages(true);

for page in reader {
    // Dictionary page (if any) is emitted first; data pages are then
    // emitted from the last `PageLocation` to the first. Each page is
    // decoded forward — only page-traversal order is reversed.
}

Implementation details:

• SerializedPageReaderState::Pages gains a reverse: bool field.
• with_reverse_pages(true) flips the flag and seeds page_index to len - 1, so the page ordinal stays correct (this matters for encryption AAD — reverse mode emits the back-most page first, which has ordinal N - 1 in the original forward order).
• get_next_page / peek_next_page / skip_next_page / peek_next_page_offset branch pop_front ↔ pop_back, front ↔ back, and += 1 ↔ saturating_sub(1) under the flag.
• The dictionary page is still emitted first in both modes (data pages depend on the loaded dictionary).
• The standalone parquet::file::reverse_serialized_reader module is gone (−1080 lines deleted).
• with_reverse_pages is marked #[doc(hidden)] with a # Stability note: the surface is intentionally experimental until the design from parquet: Support reverse page iteration for reverse streaming with filter pushdown #9934 is settled.

Encryption now works for reverse iteration too, since the crypto context is shared with the forward path.

Limitations (Phase 1 POC, intentional)

• peek_next_page does not populate num_rows precisely (the current implementation tracks a coarse value that is correct in forward mode and reversed accurately in reverse mode).
• Reverse iteration only takes effect when an OffsetIndex is available; in Values mode the flag is a no-op.

Test coverage (21 tests)

Located in parquet::file::serialized_reader::reverse_pages_tests. All 21 tests run on the unified SerializedPageReader::with_reverse_pages(true) API.

Category	Tests
Page buffer / metadata equivalence	UNCOMPRESSED / SNAPPY / GZIP / ZSTD; V1 + V2 pages; with / without dict
Value-level decode equivalence (via ColumnReaderImpl)	INT32 / INT64 / BYTE_ARRAY; with / without dict; V1 + V2
NULL handling	OPTIONAL columns, V1 + V2, ~30% nulls
State machine	peek_next_page over full lifecycle; skip_next_page with / without dict; empty-chunk peek
Edge cases	Single-data-page chunks; 50k-row stress (> 20 pages)
API behavior	with_reverse_pages is a no-op in Values mode

The page-level tests verify byte-identical buffers between forward and reverse output. The decode-level tests further verify that, after page-boundary slicing and re-arrangement, ColumnReaderImpl produces identical values and def_levels.

Benchmarks

Two criterion bench groups in parquet/benches/reverse_page_reader.rs. All numbers below are --quick, Apple M-series, 100 000 INT32 rows, 98 data pages, no dictionary.

1. Per-page cost parity (reverse_page_reader/{codec})

The empirical claim of Phase 1 — reverse iteration adds no per-page overhead vs the forward path.

Bench	Forward	Reverse
uncompressed / drain (full chunk)	9.73 µs	9.34 µs
uncompressed / first_page latency	164 ns	156 ns
snappy / drain (full chunk)	21.5 µs	21.2 µs
snappy / first_page latency	313 ns	283 ns

All within measurement noise.

2. Time-to-first-N reversed values (time_to_first_n_reversed_values/...)

Compares the existing row-group-level reverse strategy used by DataFusion (forward-decode the entire chunk, reverse the value buffer, take n) against the page-level reverse strategy enabled by this PR.

The bench has two page-level functions:

• page_reverse — single trailing reverse, correct only for n <= PAGE_ROW_COUNT_LIMIT (one page worth of rows). Used for the small-n cases below.
• page_reverse_cross — per-page reverse + append, correct for any n. This is the algorithmic shape Phase 2 (RecordBatchReader integration) will use: per-page reverse + emit, no cross-page accumulation.

n	pages needed	row_group_sim time	page_reverse(_cross) time	Speedup
10	1	26.7 µs	565 ns	~47x
100	1	26.7 µs	565 ns	~47x
1 024	1	26.7 µs	472 ns	~57x
2 000	2	26.8 µs	770 ns	~35x
10 000	10	26.6 µs	2.85 µs	~9.3x
50 000	~50	26.7 µs	14.5 µs	~1.8x

Speedup follows total_pages / pages_needed_for_n and converges to ~1x as n approaches the full chunk — the expected shape for a primitive that saves work proportional to the unread tail.

3. Peak buffer memory

Time savings come with a corresponding memory reduction. The row-group-reverse strategy must materialize the entire chunk before applying the trailing reverse + truncate; page-level reverse only needs to hold enough pages to satisfy n.

For the bench fixture (100 000 INT32, 98 pages, 1 024 rows / page):

n	row_group_sim peak buffer	page_reverse_cross peak buffer	Memory ratio
10	100 000 × 4 B = 400 KB	1 page × 4 KB = 4 KB	100x less
1 024	400 KB	4 KB	100x less
10 000	400 KB	10 pages × 4 KB = ~40 KB	~10x less
50 000	400 KB	~50 pages × 4 KB = ~200 KB	~2x less

Real-world numbers track the same ratio at a much larger absolute scale. The default Parquet writer produces ~128 MB row groups with ~1 MB pages; under DataFusion's existing row-group-reverse strategy, peak buffer is ~128 MB per row group. Page-level reverse with a small n (the typical TopK case) needs ~1 MB, a ~128x reduction. This is the structural memory benefit highlighted in #9934, and it scales independently of the per-page-cost-parity result above.

Note: the in-crate bench accumulates values into a Vec rather than streaming; in a true Phase 2 RecordBatchReader integration each page is decoded → reversed → emitted as a batch → dropped, so peak buffer becomes O(1 page) regardless of n. The numbers above represent the algorithm's structural ceiling; the bench's page_reverse_cross is an O(n)-buffer simplification of it.

Caveats

• The bench drives the reader from in-memory Bytes, so I/O latency is not modeled. On real S3 / object-store backends, both readers issue one byte-range read per page; the page-reverse advantage holds and is amplified by network round-trips dominating decode cost.
• The bench has no filter. With RowFilter pushdown, both strategies pay filter-evaluation cost; the page-reverse advantage compounds because the row-group strategy filter-evaluates pages it ultimately discards. This is the killer Phase 3 use case from parquet: Support reverse page iteration for reverse streaming with filter pushdown #9934 (WHERE filter ORDER BY DESC LIMIT N).

Why this is an RFC

The remaining work for end-to-end usefulness sits above this primitive:

• Phase 2: a ParquetRecordBatchReader mode that aligns batch boundaries with page boundaries and reverses each page's RecordBatch after decoding, so the final stream is fully reversed for any n. The bench's page_reverse_cross function shows the algorithmic shape.
• Phase 3: filter + limit early termination — the killer use case from parquet: Support reverse page iteration for reverse streaming with filter pushdown #9934.

Specific feedback I'd appreciate:

1. API stability gate: is #[doc(hidden)] + a # Stability note enough, or would you prefer an explicit experimental cargo feature?
2. peek_next_page semantics: precise num_rows in reverse mode is symmetric to the forward computation but uses back instead of front; current implementation already does this. Want it explicitly tested as a separate case?
3. Phase 2 entry point: should Phase 2 add a ParquetRecordBatchReaderBuilder::with_reverse_pages(true) flag (mirroring this PR's lower-level addition), or should reverse iteration be exposed only through a higher-level with_order(Asc | Desc) style?

Are there any user-facing changes?

A new (#[doc(hidden)]) builder method SerializedPageReader::with_reverse_pages. No existing APIs are changed. Encryption is supported in reverse mode through the shared crypto context.

Verification

cargo build -p parquet
cargo check -p parquet --all-features --tests
cargo test  -p parquet --lib reverse_pages_tests   # 21 passed
cargo test  -p parquet --lib                       # 1136 passed
cargo clippy -p parquet --all-targets -- -D warnings
cargo fmt -p parquet --check
cargo bench -p parquet --bench reverse_page_reader -- --quick

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a proof-of-concept Parquet page reader that iterates page traversal order in reverse (dictionary first, then data pages from last to first) using OffsetIndex, to enable reverse streaming primitives discussed in #9934.

Changes:

• Introduces ReverseSerializedPageReader<R: ChunkReader> implementing PageReader + Iterator<Item = Result<Page>>
• Wires the new module into parquet::file and adds an extensive test suite validating forward/reverse equivalence at page- and value-level

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
parquet/src/file/reverse_serialized_reader.rs	Implements the reverse page-order reader and adds targeted tests for correctness and state-machine behavior
parquet/src/file/mod.rs	Exposes the new module via pub mod reverse_serialized_reader;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zhuqi-lucas]

cc @alamb @adriangb

[alamb]

THanks @zhuqi-lucas -- I can see why reversing just the row groups still leaves additional potential improvments

I am worried that this type of code will add lots of complexity into the parquet reader (baasically now we need to maintain two code patsh -- the forward reader and the reverse reader)

Is there any way to reuse the existing page reader, but add some sort of abtraction that defines what pages are read in what order? At the moment it just blasts down the PageIndex in order -- but maybe we could provide an alternate OffsetIdex or something that was reversed and use the same reader path 🤔

[zhuqi-lucas]

THanks @zhuqi-lucas -- I can see why reversing just the row groups still leaves additional potential improvments

I am worried that this type of code will add lots of complexity into the parquet reader (baasically now we need to maintain two code patsh -- the forward reader and the reverse reader)

Is there any way to reuse the existing page reader, but add some sort of abtraction that defines what pages are read in what order? At the moment it just blasts down the PageIndex in order -- but maybe we could provide an alternate OffsetIdex or something that was reversed and use the same reader path 🤔

Thank you @alamb , you are right, most of it was just pop_back vs pop_front. Updated in latest PR:

SerializedPageReader::new(...)?.with_reverse_pages(true)

Flag flips iteration direction in Pages mode and seeds page_index = len - 1 so encryption AAD stays correct. Dict still emitted first either way. Standalone module gone, ~190 lines net smaller, encryption works in reverse now too.

Tried the pre-reversed-Vec route first but the existing dict inference uses chunk_start..page_locations[0].offset, which breaks once you reverse — went with the flag instead. All 21 tests migrated, bench numbers unchanged.