Skip to content

perf(shim): prune unused columns from resolved scans - #147

Open
ohaibbq wants to merge 1 commit into
mainfrom
perf/prune-unused-columns
Open

ohaibbq wants to merge 1 commit into
mainfrom
perf/prune-unused-columns

Conversation

@ohaibbq

@ohaibbq ohaibbq commented Nov 17, 2026

Copy link
Copy Markdown
Member

Turns on the analyzer's prune_unused_columns option, so a resolved scan lists only the columns the statement references. Without it, a window function carried every column of its input table through SQLite's sort, whether or not anything read them.

No issue is linked; there isn't one for this yet.

What BigQuery does

BigQuery returns the same rows either way. This is a performance change, with one correctness fix that pruning exposed.

What the emulator did

shim/analyzer_shim.cc left prune_unused_columns at its default of false, so every ResolvedTableScan and every scan above it listed all of the table's columns. Most scans tolerate that because SQLite flattens them. A window scan cannot be flattened, and SQLite does not drop unused result columns from one, so all of them went through the sorter.

On a 55-column, 200,000-row table, measured through Db.query with the two shims run back to back:

Query Before After
SELECT MAX(r) FROM (SELECT SUM(amount) OVER (PARTITION BY user_id ORDER BY ts) AS r FROM t) 841 ms 303 ms
Window over SELECT *, outer query reads one column 639 ms 631 ms
gsql-bench corpus unchanged unchanged

The second row is the limit of this change. The analyzer counts every column a SELECT * expands to as referenced, so the deduplicating shape (SELECT * ... QUALIFY ROW_NUMBER() ... = 1) is not pruned. That needs the transformer to emit only what the parent scan consumes, and is left for a follow-up.

Where to look

  • buildGroupingSetsQuery in scan_aggregate.zig. With pruning on, an aggregate scan's output can omit its grouping columns, which this function assumed were always present. 18 cases in googlesql/compliance/grouping_sets_queries failed with "ORDER BY term does not match any column in the result set" or a syntax error on an empty select list. It now orders the UNION ALL by the grouping columns the output holds, in output order, and gives a set with no output column a placeholder column to yield its rows in.
  • Row order of grouping-set results. The ORDER BY terms used to come from a hash map's iteration order and now follow the output columns. BigQuery promises no order here, and the specs that cover it are unordered.

Spec cases

testdata/specs/zigquery/syntax/query/group_by.zon:

  • rollup selecting none of its grouping columns
  • rollup selecting only its second grouping column
  • grouping sets with a constant select list
  • group by with a constant select list

The first three fail without the transformer fix once pruning is on. The fourth passed already and is there as coverage.

testdata/spec_baseline.txt gains those four, six grouping-sets-with-subquery compliance cases that now pass (and leave spec_known_failures.txt), and four analytic_option_rejections cases that already passed on main and were missing from the baseline.

Checks

  • zig build test (597/597)
  • zig build test-server (not run locally)
  • zig build spec -- --strict (5,427 PASS, no regressions; 5,417 on the same tree with the unmodified shim)
  • zig fmt --check build.zig entrypoints tools
  • ephor reports zero findings

The shim edit changes the CI shim cache key, so the shim job rebuilds on all four platforms.

🤖 Generated with Claude Code

The analyzer listed every column of a table on each scan, whatever the
statement referenced, so a window function carried all of them through its
sort. With pruning on, a running sum over a 55-column, 200k-row table runs in
303 ms instead of 841 ms. Queries that name every column through SELECT * are
unchanged.

A pruned aggregate scan can omit its grouping columns, which the ROLLUP, CUBE
and GROUPING SETS transformer assumed were always in the output. It now orders
by the grouping columns the output holds and gives a set with no output column
one to yield its rows in.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant