Repository navigation
Conversation
Add filter_column_chunks_byte_ranges, materialize_filter_columns, payload_column_chunks_byte_ranges, materialize_payload_columns, filter_row_groups_with_bloom_filters and filter_row_groups_with_dictionary_pages so the multi-file reader matches the single-file HybridScanReader.
fetch_byte_ranges_to_device now accepts source_indices to fetch the byte ranges returned by the HybridScanMultiFile methods from a multi-source SourceInfo in a single call, instead of requiring one source.
Bloom filter byte ranges cover the complete filter, header included, and the hybrid scan readers require header-stripped, 32-byte aligned bitsets. Expose fetch_bloom_filters_to_device, which produces them, and use real fetches for the multi-file two-pass test.
HybridScanPassMode selects whether a hybrid scan fetches and materializes all columns at once (SINGLE_PASS) or the filter columns and then the payload columns of the surviving rows (TWO_PASS, the default). It is configured with ParquetOptions.pass_mode or CUDF_POLARS__PARQUET_OPTIONS__PASS_MODE.
Hybrid scan tasks reading from remote sources no longer serialize their I/O submissions, which lets a task overlap its requests with those of other concurrent tasks. Add ParquetOptions.hybrid_scan_interleave_remote_io to opt out.
Parquet scan tasks that cover several files now use the multi-file hybrid scan reader when a predicate can be pushed down to the parquet reader, instead of only the tasks that read a single file. Hive partition values that differ between the files of a task are recovered from the source index column.
Parquet scan tasks read through the same ParquetScanTask whether or not the scan is split, and the hybrid reader already runs on a task that covers a whole file. The special case in scan_partition_plan predates that and only changed the plan flavor of every single-file scan.
Eagerly parsing HybridScanMetadata for every prefetched footer is wasted work for scans without a pushed down predicate and for tasks that read several files, which use the multi-file reader. Decide per scan from hybrid_scan_eligible instead of from use_hybrid_scan alone.
Leave prefetch_file_metadata unspecified in ParquetOptions and resolve it from use_hybrid_scan when the streaming executor decides whether to prefetch. The in-memory executor then only rejects an explicit request to prefetch, and use_hybrid_scan no longer fails validation when prefetching is disabled, since tasks fetch the footers they need when none were prefetched.
Pass the ParquetOptions to the hybrid scan readers rather than one argument per option, and build their reader options with use_jit_filter like the default parquet reader does.
Fetch bloom filters with fetch_bloom_filters_to_device instead of copying their byte ranges. The raw ranges include the filter headers and are not aligned, so the readers found no matching row groups and dropped every row of a file written with bloom filters.
Matt711
added
feature request
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
3 tasks done
3 tasks done
Contributor
|
FYI: #25076 adds a check to Until this PR switches cudf-polars to |
This was referenced Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Tasks that cover several files now use
HybridScanMultiFile. AddsHybridScanPassMode(TWO_PASSdefault,SINGLE_PASS) and interleaved remote I/O (from #23895 and #23974, without the automatic pass selection). Also fixes bloom filter pruning and removes what blocked making hybrid scan the default (see the issue for the list). The default is not changed.This builds on the pylibcudf changes in #24853, so the first three commits are that PR.
closes #24851
Part of #24852.
This was written with an agent. I read through the diff.
Checklist
Part of #22752.