Skip to content

perf: Elimiate SortExec on generate_series()#22238

Open
2010YOUY01 wants to merge 3 commits into
apache:mainfrom
2010YOUY01:generate-series-output-order
Open

perf: Elimiate SortExec on generate_series()#22238
2010YOUY01 wants to merge 3 commits into
apache:mainfrom
2010YOUY01:generate-series-output-order

Conversation

@2010YOUY01
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • Closes #.

Rationale for this change

What changes are included in this PR?

Eliminates SortExec on generate_series() by exposing the existing sort order property in the data source ExecutionPlan, the existing optimizer heuristics will remove it automatically.

Demo in datafusion-cli:

main

DataFusion CLI v53.1.0
> explain select * from generate_series(1,100) as t1(v1) order by v1;
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │          SortExec         │ |
|               | │    --------------------   │ |
|               | │    v1@0 ASC NULLS LAST    │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       ProjectionExec      │ |
|               | │    --------------------   │ |
|               | │         v1: value         │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       LazyMemoryExec      │ |
|               | │    --------------------   │ |
|               | │     batch_generators:     │ |
|               | │ generate_series: start=1, │ |
|               | │  end=100, batch_size=8192 │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.
Elapsed 0.006 seconds.

PR

DataFusion CLI v53.1.0
> explain select * from generate_series(1,100) as t1(v1) order by v1;

+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │       ProjectionExec      │ |
|               | │    --------------------   │ |
|               | │         v1: value         │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       LazyMemoryExec      │ |
|               | │    --------------------   │ |
|               | │     batch_generators:     │ |
|               | │ generate_series: start=1, │ |
|               | │  end=100, batch_size=8192 │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+

Are these changes tested?

slt

Are there any user-facing changes?

@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels May 16, 2026
/// Now only `Int64` argument type is supported
fn output_ordering(&self, schema: &Schema) -> Option<PhysicalSortExpr> {
let step = match &self.args {
GenSeriesArgs::Int64Args { step, .. } => *step,
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if time type allows step like 1 year -13 month, so I've kept them outside the scope of this PR for simplicity.

Copy link
Copy Markdown
Contributor

@Dandandan Dandandan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants