Repository navigation
Conversation
Add filter_column_chunks_byte_ranges, materialize_filter_columns, payload_column_chunks_byte_ranges, materialize_payload_columns, filter_row_groups_with_bloom_filters and filter_row_groups_with_dictionary_pages so the multi-file reader matches the single-file HybridScanReader.
fetch_byte_ranges_to_device now accepts source_indices to fetch the byte ranges returned by the HybridScanMultiFile methods from a multi-source SourceInfo in a single call, instead of requiring one source.
Bloom filter byte ranges cover the complete filter, header included, and the hybrid scan readers require header-stripped, 32-byte aligned bitsets. Expose fetch_bloom_filters_to_device, which produces them, and use real fetches for the multi-file two-pass test.
Matt711
added
Python
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
3 tasks done
Member
Author
|
/ok to test b5e49db |
3 tasks done
Keep the bloom fetch's Python resource and stream alive with its returned views. Accept the Sequence inputs advertised by the multifile scan APIs, and cover tuples alongside lists. Pass the selected stream through test copies and keep host inputs alive until their asynchronous transfers complete. Co-Authored-By: Codex (GPT-6) <noreply@openai.com>
Keep upstream's host-byte retention fixes in the multifile tests alongside the sequence and bloom-buffer lifetime coverage. Preserve the payload-page setup API rename from main. Co-Authored-By: Codex (GPT-6) <noreply@openai.com>
This was referenced Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Expose the remaining two-pass APIs on
pylibcudf.io.experimental.HybridScanMultiFile: filter and payload column chunk ranges, materialization, and row-group filtering with dictionary pages and bloom filters. This lets callers read filter columns first, build a row mask, and then read the surviving payload rows across multiple files. Part of #24852; used by #24854.fetch_byte_ranges_to_deviceacceptssource_indicesto fetch ranges from several sources in one call. Addfetch_bloom_filters_to_deviceto strip the headers and return aligned bitsets for bloom filtering. Returned views keep their buffers, memory resource, and stream alive.The span inputs accept lists and tuples, matching their
Sequence[Span]annotations. Tests cover the two-pass workflow, dictionary and bloom pruning, buffer ownership, and default and custom streams. Host buffers used by asynchronous test copies stay alive until the copies complete.Verification
Built libcudf and pylibcudf from this branch with CUDA 12.9.86 and GCC 14.4.0. Tested on a Tesla V100 with driver 535.161.08, Python 3.14, and PyArrow 25.
After merging
upstream/main, rantests/io/test_experimental_hybrid_scan.py,tests/io/test_experimental_hybrid_scan_multifile.py, andtests/io/test_parquet_io_utils.pyfrompython/pylibcudf.Checklist
Tools: Codex
Part of #22752.