Repository navigation
Conversation
The analyzer listed every column of a table on each scan, whatever the statement referenced, so a window function carried all of them through its sort. With pruning on, a running sum over a 55-column, 200k-row table runs in 303 ms instead of 841 ms. Queries that name every column through SELECT * are unchanged. A pruned aggregate scan can omit its grouping columns, which the ROLLUP, CUBE and GROUPING SETS transformer assumed were always in the output. It now orders by the grouping columns the output holds and gives a set with no output column one to yield its rows in. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns on the analyzer's
prune_unused_columnsoption, so a resolved scan lists only the columns the statement references. Without it, a window function carried every column of its input table through SQLite's sort, whether or not anything read them.No issue is linked; there isn't one for this yet.
What BigQuery does
BigQuery returns the same rows either way. This is a performance change, with one correctness fix that pruning exposed.
What the emulator did
shim/analyzer_shim.ccleftprune_unused_columnsat its default offalse, so everyResolvedTableScanand every scan above it listed all of the table's columns. Most scans tolerate that because SQLite flattens them. A window scan cannot be flattened, and SQLite does not drop unused result columns from one, so all of them went through the sorter.On a 55-column, 200,000-row table, measured through
Db.querywith the two shims run back to back:SELECT MAX(r) FROM (SELECT SUM(amount) OVER (PARTITION BY user_id ORDER BY ts) AS r FROM t)SELECT *, outer query reads one columngsql-benchcorpusThe second row is the limit of this change. The analyzer counts every column a
SELECT *expands to as referenced, so the deduplicating shape (SELECT * ... QUALIFY ROW_NUMBER() ... = 1) is not pruned. That needs the transformer to emit only what the parent scan consumes, and is left for a follow-up.Where to look
buildGroupingSetsQueryinscan_aggregate.zig. With pruning on, an aggregate scan's output can omit its grouping columns, which this function assumed were always present. 18 cases ingooglesql/compliance/grouping_sets_queriesfailed with "ORDER BY term does not match any column in the result set" or a syntax error on an empty select list. It now orders theUNION ALLby the grouping columns the output holds, in output order, and gives a set with no output column a placeholder column to yield its rows in.Spec cases
testdata/specs/zigquery/syntax/query/group_by.zon:The first three fail without the transformer fix once pruning is on. The fourth passed already and is there as coverage.
testdata/spec_baseline.txtgains those four, six grouping-sets-with-subquery compliance cases that now pass (and leavespec_known_failures.txt), and fouranalytic_option_rejectionscases that already passed onmainand were missing from the baseline.Checks
zig build test(597/597)zig build test-server(not run locally)zig build spec -- --strict(5,427 PASS, no regressions; 5,417 on the same tree with the unmodified shim)zig fmt --check build.zig entrypoints toolsThe shim edit changes the CI shim cache key, so the shim job rebuilds on all four platforms.
🤖 Generated with Claude Code