Optimize parquet row filter auto strategy with adaptive fallback

[hhhizzz]

Which issue does this PR close?

• Closes [Parquet] Better heuristics to pick between RowSelection and Mask filter representation #8846
• Part of [EPIC] Faster performance for parquet predicate evaluation for non selective filters #7456
• Addresses Filter pushdown selectivity threshold #9591
• Related to Parquet decoding/pushdown performance improvements #9589
• Follow-up to [Parquet]Performance Degradation with RowFilter on Unsorted Columns due to Fragmented ReadPlan #8565

Rationale for this change

RowFilter can be much slower than a full scan for fragmented selections, especially when page indexes provide little pruning and the resulting ReadPlan contains many tiny select/skip runs. #8565 shows an extreme case where predicate pushdown is around 10x slower than scanning and filtering afterwards.

This PR improves the RowSelectionPolicy::Auto path so it can make better strategy decisions for fragmented row selections and avoid continuing with row-filter pushdown when the observed shape suggests the pushdown path is unlikely to help.

The main goal is to reduce the performance cliff without changing explicit Mask / Selectors behavior.

While working on the Auto strategy, this PR also fixes a correctness issue in the Mask execution path. With sparse page-loaded ranges, a mask-backed read plan could previously attempt to consume selected rows outside the loaded ranges and fail during decoding. This made Mask risky for some page-index / fragmented-selection cases.

This PR makes the loaded row ranges explicit in the read plan and adds coverage for sparse loaded-range execution. After this change, using Mask should no longer hit this known failure mode.

Design / implementation notes

This PR started from the observation that row-level predicate pushdown can hit a performance cliff when the resulting RowSelection is highly fragmented. In that shape, selector-backed execution spends a lot of time repeatedly skipping and reading tiny row runs, so it can be slower than a full decode followed by filtering.

A simple "prefer Mask more often" rule is not sufficient, though. When page-level pruning is involved, the in-memory column chunks may be sparse: some pages were never loaded because the current selection skipped them. The old code avoided this by forcing selectors in cases where a mask could try to decode rows from pages that were not present. This PR makes that distinction explicit:

• Auto remains conservative when page pruning has produced sparse loaded ranges.
• An explicit RowSelectionPolicy::Mask no longer assumes the loaded column data is dense. It now tracks loaded row ranges and uses a sparse mask cursor, so future users of Mask do not hit missing-page / invalid-offset failures.

The other part of the change is runtime fallback. Static selection heuristics only see the final RowSelection shape; they do not know whether predicate pushdown is actually saving work for this file/query. The push decoder now observes early row-group selection shape and can switch later row groups to decode once and apply the predicate after decode when pushdown is unlikely to pay off, for example high-selectivity/no-pruning cases or fragmented moderate/high-selectivity cases.

What changes are included in this PR?

• Adds structured RowSelection shape analysis and strategy decision metrics.
• Improves RowSelectionPolicy::Auto so it can choose between mask and selectors using selection shape and loaded page ranges.
• Adds sparse loaded-range tracking from page offsets so fragmented page-loaded selections can avoid expensive mask execution.
• Adds adaptive post-filter fallback for Auto:
  • observes the first row group;
  • keeps pushdown for sparse/low-selectivity cases where it is still useful;
  • switches later row groups to post-filter for high-selectivity or fragmented moderate/high-selectivity cases.
• Preserves correctness around fallback:
  • does not reuse sparse predicate chunks when rebuilding a base/full read;
  • disables post-filter fallback for try_next_reader handoff paths;
  • avoids evaluating ArrowPredicate twice for the same current row group when caller-provided row selection is present.
• Simplifies fallback trigger metrics to shape-based reasons only, avoiding unused timing/cache fields.
• Replaces loaded range intersection with a linear two-pointer merge.
• Extends arrow_reader_row_filter benchmarks to cover strategy-sensitive cases.
• Fixes a known Mask correctness failure with sparse loaded page ranges.
• Preserves explicit Mask behavior while ensuring sparse loaded ranges are tracked so mask execution does not read outside available page ranges.

Are these changes tested?

Yes.

Unit / integration validation:

• cargo fmt -p parquet -- --check
• git diff --check
• cargo test -p parquet --lib arrow::push_decoder
  • 38 passed
• cargo test -p parquet --lib arrow::arrow_reader::read_plan
  • 28 passed
• cargo test -p parquet --lib arrow::arrow_reader::selection
  • 25 passed
• cargo test -p parquet --lib
  • 1132 passed

New focused tests cover:

• Auto fallback with caller-provided row selection does not evaluate the current row group predicate twice.
• try_next_reader does not use post-filter fallback in normal reader handoff mode.
• Sparse/current-row fallback does not reuse incompatible predicate chunks.
• Explicit Mask execution remains correct with sparse loaded page ranges.
• Auto switches away from Mask when sparse loaded ranges make selector execution safer.
• Multi-column loaded row-range intersection remains correct after switching to a linear merge.
• Auto strategy decisions for sparse loaded ranges and expensive fragmented output.

Benchmark evidence from arrow_reader_row_filter comparing origin/main vs this branch:

case	origin/main	this branch	change
utf8View != '', all columns	7.91 ms	5.52 ms	+43.2%
int64 > 90, all columns	7.29 ms	5.34 ms	+36.5%
int64 > 90, exclude filter column	6.97 ms	5.23 ms	+33.3%
utf8View != '', exclude filter column	6.77 ms	5.53 ms	+22.5%
float64 > 99.0, exclude filter column	4.94 ms	4.38 ms	+12.9%

Across the 16 async arrow_reader_row_filter cases run, geometric mean speedup was about 1.08x. The worst observed regression in that run was about -3.3%.

Are there any user-facing changes?

No intended breaking API changes.

RowSelectionPolicy::Auto may choose different internal execution strategies than before. Explicit Mask and Selectors policies remain available for callers that want fixed behavior.