Safely ignore Parquet fields with unimplemented Thrift types by etseidl · Pull Request #9974 · apache/arrow-rs

etseidl · 2026-05-14T17:00:14Z

Which issue does this PR close?

Closes Future proof Parquet Thrift parser #9973.

Rationale for this change

The thrift decoder should be able to skip fields with unimplemented thrift types set, map, and uuid.

What changes are included in this PR?

Flesh out the thrift enums and add code to the skip function to handle the above types.

Are these changes tested?

Yes, test is added

Are there any user-facing changes?

No, changes are to internal APIs

etseidl · 2026-05-14T17:01:35Z

run benchmark metadata

adriangbot · 2026-05-14T17:03:47Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4452846276-95-jn748 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing forward_proof_skip (63d82b7) to 1ffd202 (merge-base) diff
BENCH_NAME=metadata
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench metadata
BENCH_FILTER=
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-05-14T17:09:39Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                               forward_proof_skip                     main
-----                                               ------------------                     ----
decode metadata (wide) with schema                  1.00     27.6±0.41ms        ? ?/sec    1.00     27.7±0.49ms        ? ?/sec
decode metadata (wide) with skip PES                1.00     27.0±0.33ms        ? ?/sec    1.02     27.5±0.37ms        ? ?/sec
decode metadata (wide) with skip all stats          1.01     29.1±0.32ms        ? ?/sec    1.00     28.8±0.39ms        ? ?/sec
decode metadata (wide) with skip column stats       1.01     28.5±0.39ms        ? ?/sec    1.00     28.3±0.42ms        ? ?/sec
decode metadata (wide) with skip size stats         1.00     30.9±0.42ms        ? ?/sec    1.01     31.2±0.43ms        ? ?/sec
decode metadata (wide) with stats mask              1.00     27.0±0.36ms        ? ?/sec    1.01     27.3±0.44ms        ? ?/sec
decode metadata with schema                         1.00      4.0±0.02µs        ? ?/sec    1.02      4.0±0.02µs        ? ?/sec
decode metadata with skip PES                       1.00      6.9±0.03µs        ? ?/sec    1.00      6.9±0.02µs        ? ?/sec
decode metadata with skip column stats              1.00      6.7±0.11µs        ? ?/sec    1.00      6.8±0.03µs        ? ?/sec
decode metadata with stats mask                     1.01      6.9±0.07µs        ? ?/sec    1.00      6.8±0.04µs        ? ?/sec
decode parquet metadata                             1.00      7.1±0.04µs        ? ?/sec    1.00      7.1±0.02µs        ? ?/sec
decode parquet metadata (wide)                      1.00     29.6±0.41ms        ? ?/sec    1.01     29.8±0.47ms        ? ?/sec
decode parquet metadata no path_in_schema (wide)    1.00     28.9±0.41ms        ? ?/sec    1.01     29.1±0.47ms        ? ?/sec
decode parquet metadata w/ size stats (wide)        1.00     36.1±0.38ms        ? ?/sec    1.02     36.6±0.39ms        ? ?/sec
open(default)                                       1.00      7.7±0.02µs        ? ?/sec    1.00      7.7±0.01µs        ? ?/sec
open(page index)                                    1.00    132.2±0.10µs        ? ?/sec    1.00    132.4±0.14µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	160.0s
Peak memory	4.3 GiB
Avg memory	4.2 GiB
CPU user	154.3s
CPU sys	0.8s
Peak spill	0 B

branch

Metric	Value
Wall time	155.0s
Peak memory	4.2 GiB
Avg memory	4.2 GiB
CPU user	153.2s
CPU sys	0.2s
Peak spill	0 B

File an issue against this benchmark runner

etseidl · 2026-05-14T17:28:44Z

-    type Error = ThriftProtocolError;
-    fn try_from(value: ElementType) -> std::result::Result<Self, Self::Error> {
+impl From<ElementType> for FieldType {
+    fn from(value: ElementType) -> Self {


This conversion can now be infallible

etseidl · 2026-05-14T17:29:02Z

            FieldType::Double => self.skip_bytes(8).map(|_| ()),
            FieldType::Binary => self.skip_binary().map(|_| ()),
            FieldType::Struct => {
-                let mut last_field_id = 0i16;


small drive-by improvement

I found this confusing at first, it works because skipping does not use the field id.

etseidl · 2026-05-14T17:31:36Z

                Ok(())
            }
-            // no list or map types in parquet format
+            FieldType::Map => {


A map begins with a varint size. If size is 0 it's empty. Otherwise, the next byte encodes the key and value types with 4 bits each, and then encodes size key/value pairs.

maybe it would help to add a link to the relevant part of the thrift encoding spec: https://github.com/apache/thrift/blob/master/doc/specs/thrift-compact-protocol.md#map

alamb

Makes sense to me -- thanks @etseidl

alamb · 2026-05-14T20:07:18Z

                Ok(())
            }
-            // no list or map types in parquet format
+            FieldType::Map => {


maybe it would help to add a link to the relevant part of the thrift encoding spec: https://github.com/apache/thrift/blob/master/doc/specs/thrift-compact-protocol.md#map

alamb · 2026-05-14T20:11:19Z

                Ok(())
            }
-            FieldType::List => {
+            // lists and sets are encoded the same


A link to the spec might help: https://github.com/apache/thrift/blob/master/doc/specs/thrift-compact-protocol.md#list-and-set

jhorstmann · 2026-05-15T09:04:36Z

Looks good to me!

etseidl added 2 commits May 13, 2026 20:29

add more types to skip

c59bc40

add test

63d82b7

github-actions Bot added the parquet Changes to the parquet crate label May 14, 2026

etseidl commented May 14, 2026

View reviewed changes

alamb approved these changes May 14, 2026

View reviewed changes

add links to relevant parts of the thrift spec

6bc43d9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Safely ignore Parquet fields with unimplemented Thrift types#9974

Safely ignore Parquet fields with unimplemented Thrift types#9974
etseidl wants to merge 3 commits into
apache:mainfrom
etseidl:forward_proof_skip

etseidl commented May 14, 2026

Uh oh!

etseidl commented May 14, 2026

Uh oh!

adriangbot commented May 14, 2026

Uh oh!

adriangbot commented May 14, 2026

Uh oh!

etseidl May 14, 2026

Uh oh!

etseidl May 14, 2026

Uh oh!

jhorstmann May 15, 2026

Uh oh!

etseidl May 14, 2026

Uh oh!

alamb May 14, 2026

Uh oh!

alamb left a comment

Uh oh!

alamb May 14, 2026

Uh oh!

alamb May 14, 2026

Uh oh!

jhorstmann commented May 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

etseidl commented May 14, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

etseidl commented May 14, 2026

Uh oh!

adriangbot commented May 14, 2026

Uh oh!

adriangbot commented May 14, 2026

Uh oh!

etseidl May 14, 2026

Choose a reason for hiding this comment

Uh oh!

etseidl May 14, 2026

Choose a reason for hiding this comment

Uh oh!

jhorstmann May 15, 2026

Choose a reason for hiding this comment

Uh oh!

etseidl May 14, 2026

Choose a reason for hiding this comment

Uh oh!

alamb May 14, 2026

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb May 14, 2026

Choose a reason for hiding this comment

Uh oh!

alamb May 14, 2026

Choose a reason for hiding this comment

Uh oh!

jhorstmann commented May 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants