Skip to content

feature - implement RFC 020 nested scalar functions#46

Draft
dannymeijer wants to merge 1 commit into
feature/34-rfc017-aggregate-modifiersfrom
feature/37-rfc020-nested-data
Draft

feature - implement RFC 020 nested scalar functions#46
dannymeijer wants to merge 1 commit into
feature/34-rfc017-aggregate-modifiersfrom
feature/37-rfc020-nested-data

Conversation

@dannymeijer
Copy link
Copy Markdown
Collaborator

Summary

Implements RFC 020 nested data functions as scalar, registry-backed InQL helpers for arrays, maps, and named structs. The branch keeps relation-cardinality-changing generator behavior reserved for RFC 021 while adding concrete Substrait mappings and DataFusion-backed execution coverage for nested scalar projections.

Stack order: merge PR #44 first, then PR #45, then this PR.

Type of change

  • Bug fix
  • New feature
  • Refactor / maintenance
  • Documentation
  • CI / tooling
  • RFC (adds/updates docs/rfcs/*)

Area(s)

Select the primary areas touched (labels sync from checked lines when the triage workflow runs):

  • Package & tests
  • Specification (RFCs)
  • Documentation
  • Automation & repo config
  • Other

Key details

  • User-facing behavior: Authors can build and inspect nested scalar values with helpers such as array(...), cardinality(...), array_contains(...), arrays_overlap(...), element_at(...), array_flatten(...), map_from_arrays(...), map_extract(...), map_contains_key(...), map_keys(...), map_values(...), map_entries(...), and named_struct(...).
  • Internals: Adds a src/functions/nested/ helper family using declaration-side registry metadata, stable Substrait function anchors, concrete extension mappings where supported, and a documented map_contains_key(...) rewrite over existing scalar applications.
  • Risks: Nested function semantics depend on backend nested UDF support. The PR documents one-based indexing and current recoverable element_at(...) adapter behavior instead of pretending a strict/try split exists yet.

Testing / verification

  • make ci (or make fmt-check, make build, make test)
  • Manual verification described below

Manual verification notes:

  • make fmt
  • incan test tests/test_nested_data_functions.incn
  • incan test tests/test_function_registry.incn
  • incan test tests/test_substrait_plan.incn
  • incan test tests/test_session_projection.incn
  • make fmt-check
  • make test-style
  • make registry-metadata (78 helpers)
  • make build
  • make test (127 passed)
  • make pre-commit
  • make smoke-consumer

Docs impact

  • No docs changes needed
  • Docs updated
  • Docs follow Divio intent (tutorial/how-to/reference/explanation) where applicable

If docs updated:

  • Link(s): docs/language/reference/functions/nested.md, docs/language/reference/functions/index.md, docs/release_notes/v0_1.md, docs/rfcs/020_nested_data_functions.md

Checklist

  • I kept public docs user-focused and moved internals to contributing docs when appropriate
  • I avoided duplicating canonical install/run instructions in multiple places
  • I added/updated tests where it materially reduces regressions

Closes #37

@incan-triage-bot incan-triage-bot Bot added documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs labels May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant