Fix 500s from in, between, case and coalesce on object and nested fields

[cnoramut]

Description

where f in ('x', 'y'), where f between 'a' and 'z' and eval x = case(host = 'a', f else 'z') return 500 when f is an object or nested field, with Cannot cast "java.lang.String" to "java.util.Map". coalesce(f, 'z') returns 500 with a ClassCastException once a row has f missing.

Root cause. Calcite 1.42 added two branches to SqlTypeUtil.canCastFrom that accept a string as the source of a cast to MAP or ARRAY.

} else if (fromType.getFamily() == SqlTypeFamily.CHARACTER
    || fromType.getSqlTypeName() == SqlTypeName.NULL) {
  // Cast from NULL or string to map is legal
  return true;
}

leastRestrictiveByCast relies on that check, so a string and an object or nested field now get the container as their common type. visitIn, visitBetween and visitCase reject a query only when no common type exists. They now cast the string to a map, and the generated code fails to compile. EnhancedCoalesceFunction takes its return type from the same call, so it declares a map column and then puts the string in it.

Fix. OpenSearchTypeFactory.leastRestrictive returns null when a string type meets an object or array type, which is the answer Calcite 1.41 gave. The three guards return their existing 400s again, and coalesce falls back to the string column it already returns for coalesce(f, 1).

Before, source=objs | where f in ('x', 'y') | fields host, HTTP 500

"reason":  "Internal error while compiling the query plan."
"details": "Line 3, Column 87: Cannot cast \"java.lang.String\" to \"java.util.Map\""
"type":    "RuntimeException"

After, the same query, HTTP 400

"context": {"stage": "analyzing", ...}
"details": "In expression types are incompatible: fields type STRUCT, values type [STRING, STRING]"
"type":    "SemanticCheckException"

Notes

1. Test posture - OpenSearchTypeFactoryTest pins the type answers, and issues/5856.yml runs each query end to end. The error cases match the exception type, the analyzing stage and the message as well as the status, because on the assertion-enabled test nodes the nested in case already returns 400 without the fix. Without the fix, 3 unit tests and 7 of the 8 YAML cases fail, and the keyword control passes.
2. Why the type factory and not the three call sites - coalesce never goes through CalciteRexNodeVisitor. A stricter SqlTypeMappingRule cannot help, since the new branches return before the rule is read. The same method already special-cases VARBINARY. An ANY operand is left to Calcite, which keeps ANY as the common type.
3. Behaviour change against 3.8 and 3.9 - Both releases ship Calcite 1.42, where coalesce(obj, 'z') returns an object column while no row needs the fallback and array(obj, 'z') returns an array that mvindex then fails on with 500. Both 200s arrived with the Calcite 1.42 bump in Bump Apache Calcite 1.41.0 -> 1.42.0 (CVE-2026-46718) #5619, and each is one step from a 500, the coalesce column as soon as a row lacks obj. This branch returns the answer main gave before the bump for both, shown in the table below.

These queries ran on 3a837689c, the last main commit before the Calcite 1.42 bump in #5619, on main at 779bdf5d2, and on this branch, each built from source with Calcite enabled. This branch returns a byte-identical response to the first column on every row. The bump commit 6a07689a7 gives the same status codes as 779bdf5d2.

Query on an object or nested field	3a837689c, before #5619	779bdf5d2	This branch
where obj in ('x', 'y')	400	500	400
where obj between 'a' and 'z'	400	500	400
case(host = 'a', obj else 'z')	400	500	400
where items in ('x', 'y'), nested	400	400 AssertionError	400
coalesce(obj, 'z'), a row lacks obj	200 string	500	200 string
coalesce(obj, 'z'), every row has obj	200 string	200 struct	200 string
coalesce(items, 'z'), a row lacks items	200 string	500	200 string
coalesce(obj, 1)	200 string	200 string	200 string
array(obj, 'z')	400	200 array	400
array(obj, 'z') then mvindex(x, 1)	400	500	400
where host in ('a', 'c'), keyword control	200	200	200

The nested in row hits an engine assertion first on these assertion-enabled nodes and returns 400 AssertionError on main. Without assertions the same query returns 500 on main, reproduced at 866dafcbd.

4. Not covered - lookup with append where a string field and an object field share a name still returns 500. It now fails while analyzing with Cannot infer return type for COALESCE, which by source reading is also what 1.41 did. union of an object with a string returns 500 from SchemaUnifier, which casts by type name without asking leastRestrictive, so this change does not touch it.

Related Issues

Resolves #5856

Testing

Measured on a local single-node main at 866dafcbd against the same index before and after, with Calcite and pushdown enabled. Across 105 queries, every changed line is one of the 500s turning into a 400 or a 200, a change listed in notes 3 and 4, or an array call that already returned 400 and now fails while analyzing. The keyword, long and temporal in and between paths, mvappend, append, multisearch, if, ifnull and an ANY branch are byte-identical.

Suite	Result
:core:test, :ppl:test, :opensearch:test	9271 tests, 0 failures, 39 skipped
12 related Calcite IT classes, including CalcitePPLLookupIT and CalcitePPLEnhancedCoalesceIT	285 tests, 0 failures
:integ-test:yamlRestTest	255 tests, 0 failures

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Explicitly handle null container type The method should handle null values from getContainerTypeName(t) more explicitly before using it in the boolean OR operation. While the current code works, explicitly checking for non-null would make the intent clearer and prevent potential issues if the method behavior changes. core/src/main/java/org/opensearch/sql/calcite/utils/OpenSearchTypeFactory.java [504-516] private static boolean mixesVarcharWithContainer(List<RelDataType> types) { boolean hasVarchar = false; boolean hasContainer = false; for (RelDataType t : types) { SqlTypeName name = t.getSqlTypeName(); if (name == SqlTypeName.ANY) { return false; } hasVarchar |= name == SqlTypeName.VARCHAR || name == SqlTypeName.CHAR; - hasContainer |= getContainerTypeName(t) != null; + String containerType = getContainerTypeName(t); + hasContainer |= containerType != null; } return hasVarchar && hasContainer; } Suggestion importance[1-10]: 3 __ Why: The suggestion improves code clarity by extracting getContainerTypeName(t) into a variable, making the null check more explicit. However, the existing code already handles null correctly since null != null is false, so this is a minor readability improvement with no functional impact.	Low

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.83%. Comparing base (0026279) to head (d0063db).
⚠️ Report is 1 commits behind head on main.

❌ Your project check has failed because the head coverage (63.83%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5870      +/-   ##
============================================
+ Coverage     63.80%   63.83%   +0.03%     
- Complexity     9070     9084      +14     
============================================
  Files           962      962              
  Lines         40989    41001      +12     
  Branches       4660     4666       +6     
============================================
+ Hits          26152    26173      +21     
+ Misses        13960    13951       -9     
  Partials        877      877              

Flag	Coverage Δ
sql-engine	63.83% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.