Skip to content

feature - implement RFC 021 generator table functions (#38)#47

Merged
dannymeijer merged 6 commits into
mainfrom
feature/38-rfc021-generators
May 27, 2026
Merged

feature - implement RFC 021 generator table functions (#38)#47
dannymeijer merged 6 commits into
mainfrom
feature/38-rfc021-generators

Conversation

@dannymeijer
Copy link
Copy Markdown
Collaborator

@dannymeijer dannymeijer commented May 25, 2026

Summary

Implements RFC 021 generator and table-valued functions after RFC 020 has merged into main. This PR adds the full portable generator surface for explode, explode_outer, posexplode, posexplode_outer, inline, inline_outer, flatten, and stack, keeping generators distinct from scalar expressions and aggregate measures while lowering them through Substrait relation-extension metadata.

Refs #38.

Stack order: #44, #45, and #46 are merged; this PR targets current main directly.

Type of change

  • Bug fix
  • New feature
  • Refactor / maintenance
  • Documentation
  • CI / tooling
  • RFC (adds/updates docs/rfcs/*)

Area(s)

  • Package & tests
  • Specification (RFCs)
  • Documentation
  • Automation & repo config
  • Other

Key details

  • User-facing behavior: Adds generate(...) plus generator helpers for explode(...), explode_outer(...), posexplode(...), posexplode_outer(...), inline(...), inline_outer(...), portable table-valued flatten(...), and stack(...). Generated aliases are explicit and validated. The zero-arg DataSet.explode() remains a lower-level extension-boundary operation rather than the RFC 021 generator surface.
  • Internals: Adds GeneratorApplication, generator registry metadata, Prism Generate nodes, payload-backed Substrait relation-extension lowering, generator-aware output-column inference, and DataFusion execution for root and composed generator plans.
  • DataFusion bridge: Uses Substrait as the semantic boundary, then materializes generator extension relations into execution-only temporary tables where the stock DataFusion Substrait consumer cannot consume InQL generator extensions directly.
  • RFC status: Marks RFC 021 implemented. Query-block syntax itself is owned by RFC 003; when that surface lands, generator clauses should lower to this GeneratorApplication model.

Testing / verification

  • make fmt INCAN=/Users/danny/Development/encero/incan/target/debug/incan
  • incan test tests/test_session_generators.incn (10 passed)
  • incan test tests (143 passed)
  • make fmt-check INCAN=/Users/danny/Development/encero/incan/target/debug/incan
  • make test-style INCAN=/Users/danny/Development/encero/incan/target/debug/incan
  • make registry-metadata INCAN=/Users/danny/Development/encero/incan/target/debug/incan (87 helpers)
  • make build INCAN=/Users/danny/Development/encero/incan/target/debug/incan
  • make test INCAN=/Users/danny/Development/encero/incan/target/debug/incan (143 passed)
  • make smoke-consumer INCAN=/Users/danny/Development/encero/incan/target/debug/incan
  • make pre-commit INCAN=/Users/danny/Development/encero/incan/target/debug/incan (143 passed)
  • git diff --check
  • Redundant URI accessor sweep: no remaining *_extension_uri() helper call sites in src, tests, or docs

Docs impact

  • No docs changes needed
  • Docs updated
  • Docs follow Divio intent (tutorial/how-to/reference/explanation) where applicable

If docs updated:

  • Link(s): docs/language/reference/functions/generators.md, docs/language/reference/functions/nested.md, docs/language/reference/dataset_methods.md, docs/rfcs/021_generator_table_functions.md, docs/release_notes/v0_1.md

Checklist

  • I kept public docs user-focused and moved internals to contributing docs when appropriate
  • I avoided duplicating canonical install/run instructions in multiple places
  • I added/updated tests where it materially reduces regressions

@incan-triage-bot incan-triage-bot Bot added documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs labels May 25, 2026
@dannymeijer dannymeijer force-pushed the feature/37-rfc020-nested-data branch from 76cdb21 to 3a03dd1 Compare May 27, 2026 13:35
@dannymeijer dannymeijer changed the base branch from feature/37-rfc020-nested-data to main May 27, 2026 15:12
@dannymeijer dannymeijer force-pushed the feature/38-rfc021-generators branch from bbe913e to 959e82e Compare May 27, 2026 15:30
@dannymeijer dannymeijer changed the title feature - implement RFC 021 generator foundations (#38) feature - implement RFC 021 generator table functions (#38) May 27, 2026
@dannymeijer dannymeijer marked this pull request as ready for review May 27, 2026 18:55
@dannymeijer dannymeijer merged commit 0b13bc6 into main May 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation package Library source, tests, incan.toml specification docs/rfcs/ normative RFCs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant