Skip to content

Fix 41 analytic option rejections - #136

Merged
ohaibbq merged 7 commits into
Recidiviz:mainfrom
Datamagician10:fix-41-analytic-option-rejections
Nov 13, 2026
Merged

ohaibbq merged 7 commits into
Recidiviz:mainfrom
Datamagician10:fix-41-analytic-option-rejections

Conversation

@Datamagician10

@Datamagician10 Datamagician10 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #41

What BigQuery does

What the emulator did

Spec cases

Checks

  • zig build test
  • zig build test-server
  • zig build spec -- --strict
  • zig fmt --check build.zig entrypoints tools
  • ephor reports zero findings

…s analytic functions (Recidiviz#41)

BigQuery refuses to plan STRING_AGG(DISTINCT x) OVER () and
ARRAY_AGG(x IGNORE NULLS) OVER (); this engine returned a value for
both. The rejection is per-function on the analytic path only: the
GROUP BY aggregates keep DISTINCT and IGNORE NULLS, and other
analytic aggregates (MIN, SUM, COUNT) keep DISTINCT. RESPECT NULLS
emits no marker on the analytic path, so an explicit RESPECT NULLS
cannot be distinguished from the default and is left alone.
Direct done() tests: STRING_AGG with DISTINCT errors with
BigQuery's message, and the same frame without DISTINCT still
aggregates duplicates; ARRAY_AGG with IGNORE NULLS errors, and
without it the NULL stays in the array. SQL-level coverage via the
new spec cases is left to CI: no system sqlite3 here, so the
window.zig TestEnv suite cannot link locally.
Two error cases (STRING_AGG DISTINCT, ARRAY_AGG IGNORE NULLS over
a window, with BigQuery's messages as expect_error substrings) plus
controls showing both functions still work over a window without
the rejected option. New cases are not in spec_baseline.txt;
regenerate with --update-baseline once they pass in CI.

@ohaibbq ohaibbq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that these checks should be done at analyze/transform time, not at query runtime. A map of the odd duckling functions to which options they reject can live in just one place rather than be scattered across each window function's implementation

https://github.com/Recidiviz/zigquery/blob/main/src/internal/transformer/scan_analytic.zig

…iviz#41)

Review feedback on PR Recidiviz#136: the STRING_AGG DISTINCT and ARRAY_AGG IGNORE NULLS rejections sat in each window function's runtime done(). They now live in one map in transformer/scan_analytic.zig (function -> rejected option pseudo-arg -> BigQuery message) and are checked while the analytic scan is transformed, so a rejected query fails translation with BigQuery's message and never reaches the runtime. The fn_window done() implementations keep honoring the options if a call does reach them; their unit test asserts that instead of the rejection. New transformer tests cover both rejections at translate time plus controls (plain STRING_AGG / ARRAY_AGG, ARRAY_AGG DISTINCT, SUM DISTINCT).

Checks: zig build test 592/592, zig build test-server pass, spec-runner analytic_option_rejections 4/4 PASS, zig fmt --check clean.
@Datamagician10

Datamagician10 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Done — moved the checks to transform time.

@ohaibbq ohaibbq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks a lot better! Could you have your claude follow the comment styles laid our in AGENTS.md?

Thank you!

Comment thread src/internal/function/fn_window.zig Resolved
Comment thread src/internal/function/fn_window.zig Resolved
Comment thread src/internal/function/fn_window.zig Resolved
Follow-up on the Recidiviz#136 review: fn_window.zig drops out of the change entirely (the window .zon specs cover that behavior), the scan_analytic tests call checkAnalyticOptionRejections directly instead of translating whole queries, and the comments state the rule briefly per AGENTS.md.

Checks: zig build test, zig build test-server, spec-runner analytic_option_rejections 4/4 PASS, zig fmt clean.

@ohaibbq ohaibbq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

beauty

ohaibbq and others added 2 commits November 12, 2026 22:54
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ohaibbq
ohaibbq enabled auto-merge (squash) November 13, 2026 18:36
@ohaibbq
ohaibbq merged commit 1d709df into Recidiviz:main Nov 13, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: window STRING_AGG(DISTINCT) and ARRAY_AGG(IGNORE NULLS) are accepted; BigQuery rejects them

2 participants