GH-49470: [C++][Gandiva] Fix crashes in substring_index and truncate with extreme integer values by dmitry-chirkov-dremio · Pull Request #49471 · apache/arrow

dmitry-chirkov-dremio · 2026-03-09T02:45:32Z

Rationale for this change

Two Gandiva functions crash when called with extreme integer parameter values:

substring_index(VARCHAR, VARCHAR, INT) crashes with SIGBUS when count is INT_MIN
truncate(BIGINT, INT) crashes with SIGSEGV when scale is INT_MAX or INT_MIN

What changes are included in this PR?

substring_index fix (gdv_string_function_stubs.cc):

Replace abs(cnt) with safe int64_t computation to avoid undefined behavior when cnt == INT_MIN

truncate fix (precompiled/extended_math_ops.cc):

Return input unchanged for positive scales (no-op for integers)
Return 0 for scales < -38 to prevent out-of-bounds access in GetScaleMultiplier

Are these changes tested?

Yes. Added coverage for INT_MAX/INT_MIN values in gdv_function_stubs_test.cc and extended_math_ops_test.cc.

Are there any user-facing changes?

No.

This PR contains a "Critical Fix". These changes fix crashes caused by:

abs(INT_MIN) triggering undefined behavior (integer overflow) in substring_index
Out-of-bounds array access in GetScaleMultiplier when truncate receives extreme scale values

GitHub Issue: [C++][Gandiva] Functions substring_index and truncate crash with extreme integer values #49470

…ncate with extreme integer values

github-actions · 2026-03-09T02:45:56Z

⚠️ GitHub issue #49470 has been automatically assigned in GitHub to PR creator.

zanmato1984

The failing CI seems unrelated .

+1

conbench-apache-arrow · 2026-03-16T04:20:50Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit d6ce56e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.

jonkeane · 2026-03-22T14:27:03Z

@github-actions crossbow submit test-r-linux-sanitizers

github-actions · 2026-03-22T14:29:10Z

Revision: 8b17dff

Submitted crossbow builds: ursacomputing/crossbow @ actions-90c6287765

Task	Status
test-r-linux-sanitizers

…ncate with extreme integer values (apache#49471) ### Rationale for this change Two Gandiva functions crash when called with extreme integer parameter values: 1. `substring_index(VARCHAR, VARCHAR, INT)` crashes with SIGBUS when count is `INT_MIN` 2. `truncate(BIGINT, INT)` crashes with SIGSEGV when scale is `INT_MAX` or `INT_MIN` ### What changes are included in this PR? **substring_index fix** (`gdv_string_function_stubs.cc`): - Replace `abs(cnt)` with safe `int64_t` computation to avoid undefined behavior when `cnt == INT_MIN` **truncate fix** (`precompiled/extended_math_ops.cc`): - Return input unchanged for positive scales (no-op for integers) - Return 0 for scales < -38 to prevent out-of-bounds access in `GetScaleMultiplier` ### Are these changes tested? Yes. Added coverage for `INT_MAX`/`INT_MIN` values in `gdv_function_stubs_test.cc` and `extended_math_ops_test.cc`. ### Are there any user-facing changes? No. **This PR contains a "Critical Fix".** These changes fix crashes caused by: - `abs(INT_MIN)` triggering undefined behavior (integer overflow) in `substring_index` - Out-of-bounds array access in `GetScaleMultiplier` when `truncate` receives extreme scale values * GitHub Issue: apache#49470 Authored-by: Dmitry Chirkov <dmitry.chirkov@dremio.com> Signed-off-by: Rossi Sun <zanmato1984@gmail.com>

…eGH-49421 Backport several upstream fixes. (#136) * apacheGH-49420: [C++][Gandiva] Fix castVARCHAR memory allocation and len<=0 handling (apache#49421) ### Rationale for this change The `castVARCHAR` functions in Gandiva have memory allocation inefficiencies and missing edge case handling. See apacheGH-49420 for details. ### What changes are included in this PR? **Functional fixes:** - `bool`: Remove unused 5-byte arena allocation; return string literal directly - `int32`/`int64`: Add handling for `len=0` (return empty string) and `len<0` (set error) **Memory allocation optimizations:** - `int32`/`int64`: Allocate fixed small buffer (11/20 bytes) directly in arena, use optimized digit-pair conversion writing right-to-left, then `memmove` to align. Returns `min(len, actual_size)` bytes. - `date64`: Allocate only `min(len, 10)` bytes upfront (output is always "YYYY-MM-DD") - `float32`/`float64`: Allocate only `min(len, 24)` bytes upfront (max output length) **Code cleanup:** - Extract common code into helper macros to reduce duplication ### Are these changes tested? Yes. Added tests for `len=0` and `len<0` edge cases for int64, date64, float32, float64, and bool types. All existing Gandiva tests pass. Adhoc performance benchmarking was performed both via direct expression evaluation as well as via query execution via Dremio. ### Are there any user-facing changes? No. Users will see reduced memory usage and proper error messages for invalid len parameter values. Note: Error messages for negative `len` remain different between precompiled ("Output buffer length can't be negative") and interpreted ("Buffer length cannot be negative") code paths, preserving existing behavior. * GitHub Issue: apache#49420 Authored-by: Dmitry Chirkov <dmitry.chirkov@dremio.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> * apacheGH-49438: [C++][Gandiva] Optimize LPAD/RPAD functions (apache#49439) ### Rationale for this change The `lpad_utf8_int32_utf8` and `rpad_utf8_int32_utf8` functions have performance inefficiency and a potential memory safety issue: 1. **Performance**: Single-byte fills iterate character-by-character when `memset` would suffice. Multi-byte fills use O(n) iterations instead of O(log n) with a doubling strategy. 2. **Memory safety**: When the fill string is longer than the padding space needed, the code could write more bytes than allocated. Fixed preventatively. ### What changes are included in this PR? 1. **Memory safety fix**: Use `std::min(fill_text_len, total_fill_bytes)` for the initial copy to prevent overflow 2. **Fast path**: Add single-byte fill optimization using `memset` 3. **General path**: Replace character-by-character loop with doubling strategy for multi-byte fills 4. **Tests**: Add comprehensive tests for the new code paths ### Are these changes tested? Yes. Added tests covering: - Large UTF-8 fill characters (4-byte emoji, 3-byte Chinese) - Single-byte fill boundaries (1 char and 65536 char padding) - Content verification for fill patterns - Doubling strategy boundaries - Partial fill scenarios (fill text longer than padding needed) ### Are there any user-facing changes? No. * GitHub Issue: apache#49438 Authored-by: Dmitry Chirkov <dmitry.chirkov@dremio.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> * apacheGH-49441: [C++][Gandiva] Add rand_integer function (apache#49442) ### Rationale for this change Add `rand_integer` function to Gandiva to generate random integers, complementing the existing `rand`/`random` functions that generate random doubles. This provides native integer random number generation and offers a more efficient alternative to `CAST(rand() * range AS INT)`. ### What changes are included in this PR? - Add `RandomIntegerGeneratorHolder` class following the existing `RandomGeneratorHolder` pattern - Implement three function signatures: - `rand_integer()` → int32 in range [INT32_MIN, INT32_MAX] - `rand_integer(int32 range)` → int32 in range [0, range-1] - `rand_integer(int32 min, int32 max)` → int32 in range [min, max] inclusive - Add parameter validation (range > 0, min <= max) at expression compilation time - Add 8 unit tests covering all signatures and edge cases - Use `std::uniform_int_distribution<int32_t>` with Mersenne Twister engine ### Are these changes tested? Yes, added 8 unit tests in `random_generator_holder_test.cc`: - `NoParams` - verifies full int32 range - `WithRange` - verifies [0, range-1] bounds - `WithMinMax` - verifies [min, max] inclusive bounds - `WithNegativeMinMax` - verifies negative range handling - `InvalidRangeZero` - verifies range=0 is rejected - `InvalidRangeNegative` - verifies negative range is rejected - `InvalidMinGreaterThanMax` - verifies min > max is rejected - `NullRangeDefaultsToOne` - verifies null parameter handling ### Are there any user-facing changes? Yes, this adds a new `rand_integer` function to Gandiva with three signatures as described above. * GitHub Issue: apache#49441 * apacheGH-49454: [C++][Gandiva] Fix castVARCHAR_timestamp for pre-epoch timestamps (apache#49455) ### Rationale for this change apacheGH-49454 castVARCHAR_timestamp_int64 produces negative milliseconds for pre-epoch timestamps ### What changes are included in this PR? Fixed `castVARCHAR_timestamp_int64` to correctly handle pre-epoch timestamps (before 1970-01-01). The issue was that using `in % MILLIS_IN_SEC` on negative timestamps produces negative milliseconds, resulting in output like `"0107-10-17 12:20:03.-10"`. ### Are these changes tested? Yes, added 4 new test cases covering pre-epoch timestamps with milliseconds ### Are there any user-facing changes? **This PR contains a "Critical Fix".** This fixes a bug that caused **incorrect data to be produced** when casting pre-epoch timestamps to VARCHAR in Gandiva. Previously, timestamps before 1970-01-01 with non-zero milliseconds would produce invalid output with negative millisecond values (e.g., `"0107-10-17 12:20:03.-10"` instead of `"0107-10-17 12:20:03.900"`). * GitHub Issue: apache#49454 Authored-by: Dmitry Chirkov <dmitry.chirkov@dremio.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> * apacheGH-49470: [C++][Gandiva] Fix crashes in substring_index and truncate with extreme integer values (apache#49471) ### Rationale for this change Two Gandiva functions crash when called with extreme integer parameter values: 1. `substring_index(VARCHAR, VARCHAR, INT)` crashes with SIGBUS when count is `INT_MIN` 2. `truncate(BIGINT, INT)` crashes with SIGSEGV when scale is `INT_MAX` or `INT_MIN` ### What changes are included in this PR? **substring_index fix** (`gdv_string_function_stubs.cc`): - Replace `abs(cnt)` with safe `int64_t` computation to avoid undefined behavior when `cnt == INT_MIN` **truncate fix** (`precompiled/extended_math_ops.cc`): - Return input unchanged for positive scales (no-op for integers) - Return 0 for scales < -38 to prevent out-of-bounds access in `GetScaleMultiplier` ### Are these changes tested? Yes. Added coverage for `INT_MAX`/`INT_MIN` values in `gdv_function_stubs_test.cc` and `extended_math_ops_test.cc`. ### Are there any user-facing changes? No. **This PR contains a "Critical Fix".** These changes fix crashes caused by: - `abs(INT_MIN)` triggering undefined behavior (integer overflow) in `substring_index` - Out-of-bounds array access in `GetScaleMultiplier` when `truncate` receives extreme scale values * GitHub Issue: apache#49470 Authored-by: Dmitry Chirkov <dmitry.chirkov@dremio.com> Signed-off-by: Rossi Sun <zanmato1984@gmail.com> --------- Signed-off-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Rossi Sun <zanmato1984@gmail.com> Co-authored-by: Dmitry Chirkov <dmitry.chirkov@dremio.com>

apacheGH-49470: [C++][Gandiva] Fix crashes in substring_index and tru…

752ce05

…ncate with extreme integer values

github-actions Bot added Component: C++ Component: Gandiva awaiting review Awaiting review labels Mar 9, 2026

lriggs reviewed Mar 9, 2026

View reviewed changes

Comment thread cpp/src/gandiva/gdv_string_function_stubs.cc

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Mar 9, 2026

lriggs reviewed Mar 11, 2026

View reviewed changes

Comment thread cpp/src/gandiva/gdv_string_function_stubs.cc Outdated

Use int64_t consistently to avoid overflow in edge cases

8b17dff

zanmato1984 approved these changes Mar 13, 2026

View reviewed changes

lriggs approved these changes Mar 14, 2026

View reviewed changes

zanmato1984 merged commit d6ce56e into apache:main Mar 15, 2026
53 of 54 checks passed

zanmato1984 removed the awaiting committer review Awaiting committer review label Mar 15, 2026

zanmato1984 mentioned this pull request Mar 15, 2026

[C++][Gandiva] Functions substring_index and truncate crash with extreme integer values #49470

Closed

dmitry-chirkov-dremio mentioned this pull request Apr 17, 2026

[C++][Gandiva] Some string functions may be overflowed #49753

Open

lriggs mentioned this pull request Apr 23, 2026

GH-49471, GH-49455, GH-49442, GH-49439, GH-49421 Backport several upstream fixes. dremio/arrow#136

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-49470: [C++][Gandiva] Fix crashes in substring_index and truncate with extreme integer values#49471

GH-49470: [C++][Gandiva] Fix crashes in substring_index and truncate with extreme integer values#49471
zanmato1984 merged 2 commits intoapache:mainfrom
dmitry-chirkov-dremio:fix-gandiva-extreme-int-crashes

dmitry-chirkov-dremio commented Mar 9, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Mar 9, 2026

Uh oh!

Uh oh!

Uh oh!

zanmato1984 left a comment •

edited

Loading

Uh oh!

Uh oh!

conbench-apache-arrow Bot commented Mar 16, 2026

Uh oh!

jonkeane commented Mar 22, 2026

Uh oh!

github-actions Bot commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

dmitry-chirkov-dremio commented Mar 9, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Mar 9, 2026

Uh oh!

Uh oh!

Uh oh!

zanmato1984 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

conbench-apache-arrow Bot commented Mar 16, 2026

Uh oh!

jonkeane commented Mar 22, 2026

Uh oh!

github-actions Bot commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dmitry-chirkov-dremio commented Mar 9, 2026 •

edited by github-actions Bot

Loading

zanmato1984 left a comment •

edited

Loading