Skip to content

feature - implement RFC 021 generator foundations (#38)#47

Draft
dannymeijer wants to merge 1 commit into
feature/37-rfc020-nested-datafrom
feature/38-rfc021-generators
Draft

feature - implement RFC 021 generator foundations (#38)#47
dannymeijer wants to merge 1 commit into
feature/37-rfc020-nested-datafrom
feature/38-rfc021-generators

Conversation

@dannymeijer
Copy link
Copy Markdown
Collaborator

Summary

Adds the first RFC 021 generator/table-valued function foundation on top of the RFC020 branch. This slice introduces registry-backed generator applications for the explode family, keeps generators distinct from scalar expressions and aggregate measures, and lowers them through the current Substrait ExtensionSingleRel relation-extension boundary.

Refs #38.

Stack order: merge #44, then #45, then #46, then this PR.

Type of change

  • Bug fix
  • New feature
  • Refactor / maintenance
  • Documentation
  • CI / tooling
  • RFC (adds/updates docs/rfcs/*)

Area(s)

  • Package & tests
  • Specification (RFCs)
  • Documentation
  • Automation & repo config
  • Other

Key details

  • User-facing behavior: Adds generate(...) plus explode(...), explode_outer(...), posexplode(...), and posexplode_outer(...) helper applications with required output aliases. The existing zero-arg DataSet.explode() remains as a compatibility marker.
  • Internals: Adds GeneratorApplication, generator registry metadata, Prism Generate nodes, generator-aware output-column inference, relation-extension lowering, and unique relation extension URN anchors when multiple extension relations appear in one plan.
  • Risks: This is an RFC021 foundation slice, not the full RFC. inline, inline_outer, stack, portable table-valued flatten, query-block syntax, and DataFusion table-valued execution remain future work.

Testing / verification

  • make ci (or make fmt-check, make build, make test)
  • Manual verification described below

Manual verification notes:

  • incan test tests/test_generator_functions.incn
  • incan test src/generator_builders.incn
  • incan test src/functions/generators
  • incan test tests/test_prism.incn
  • incan test tests/test_dataset.incn
  • incan test tests/test_function_registry.incn
  • incan test tests/test_substrait_plan.incn
  • make fmt
  • make fmt-check
  • make test-style
  • make registry-metadata
  • make build
  • make test
  • make smoke-consumer
  • make pre-commit

Docs impact

  • No docs changes needed
  • Docs updated
  • Docs follow Divio intent (tutorial/how-to/reference/explanation) where applicable

If docs updated:

  • Link(s): docs/language/reference/functions/generators.md, docs/language/reference/dataset_methods.md, docs/rfcs/021_generator_table_functions.md, docs/release_notes/v0_1.md

Checklist

  • I kept public docs user-focused and moved internals to contributing docs when appropriate
  • I avoided duplicating canonical install/run instructions in multiple places
  • I added/updated tests where it materially reduces regressions

@incan-triage-bot incan-triage-bot Bot added documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs labels May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant