Repository navigation
fix(parser,ast): ClickHouse parametric aggregates, FROM table functions and their rendering (#605, #606, #607) - #628
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Claude finished @ajitpratap0's task in 20s —— View job Review of PR #628 (ClickHouse parametric aggregates, FROM table functions, rendering)I read the non-test source diff. I did not build the project or run the tests. I did not read the new test files in full. Overall: The changes are small and well scoped. The error messages are clear. Each fix is gated tightly to the dialect it targets, and the PR description is unusually thorough. I found no blocking bugs. What looks good
Points to consider
Performance and security
TestsThe PR description reports regression tests that fail on the parent commit and pass on HEAD, plus a clean race gate. Coverage of the new paths looks proportionate. I would add one parser test for Verdict: Approve with minor, non-blocking suggestions (points 1 and 4). |
…metric aggregates and allow subscripts on call results (#605) In the ClickHouse fn(params)(args) path the parser moved the first paren group into Parameters but left its DISTINCT/ORDER BY on the FunctionCall, so uniqExact(DISTINCT x)(y) rendered as uniqExact(x)(DISTINCT y), which does not re-parse. The second group also had no DISTINCT handling, and a [...] subscript after any function call was rejected. - ClickHouse fn(p)(DISTINCT a) now parses, with Distinct applying to the argument group, and round-trips through SQL() and FormatAST. - DISTINCT or ORDER BY in the parameter group, and a third paren group (f(1)(2)(3)), are rejected with E2004 errors that point at the offending token instead of being silently rewritten or failing later with E2002. - A [...] subscript or slice directly after an identifier-named function call (groupArray(x)[1], quantiles(0.5, 0.9)(x)[1], array_agg(x)[1]) parses as ArraySubscriptExpression/ArraySliceExpression over the call, mirroring the existing identifier[...] handling. SQL Server is unaffected since [x] tokenizes as a bracket-quoted identifier there. The parametric-argument logic moves to clickhouse_parametric.go so the window.go change stays limited to parseFunctionCall. No AST, exported API or pool changes. Fixes #605 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
…e, file, url, ...) (#606) ClickHouse table functions in the FROM list were rejected with E2002 ("expected statement, got LPAREN") because supportsTableFunction() only enabled function-call table references for Snowflake, BigQuery and PostgreSQL. - Add ClickHouse to supportsTableFunction(), so numbers(10), numbers(10) AS n, remote('host', db, tbl), t, numbers(3), file(...), s3(...), cluster(...), generateRandom(...) etc. parse into a TableReference with TableFunc set. - Add isClickHouseKeywordTableFunction() and a ClickHouse-only branch in parseFromTableReference for table functions whose names tokenize as keywords (url, merge, values, null) when directly followed by '('. parseQualifiedName rejected these with "expected identifier, got URL". - Document table functions in the ClickHouse section of docs/SQL_COMPATIBILITY.md. Other dialects are unchanged, and so is parseJoinedTableRef (table functions on the right side of a JOIN). Rendering of TableFunc arguments in SQL()/formatter is fixed by #607, the next commit in this chain, which also adds the ClickHouse round-trip cases. Regression tests: TestClickHouseFromTableFunctions_Issue606 (14 cases, asserting AST shape) and TestClickHouseInsertSelectFromTableFunction_Issue606. Fixes #606 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
SQL() and FormatAST wrote only TableReference.Name and dropped the TableFunc arguments: generate_series(1,10) AS g became "generate_series g" and numbers(10) became "numbers". Both renderers now emit the FunctionCall when TableFunc is set (PostgreSQL, Snowflake, BigQuery, ClickHouse). Named-argument text (FLATTEN(input => x)) depends on #600; the CLI's own renderer is replaced by #599. Fixes #607 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
…pt parsing (#605) The SQLite tokenizer emits LBRACKET for bracket-quoted identifiers, so the new call-result subscript rule turned SELECT count(*) [cnt] into count(*)[cnt]: the alias was lost and the output is not valid SQLite. The rule now skips SQLite, restoring the pre-#605 behaviour there, and a control test pins it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
F2 (transform): ReplaceTable lost renames on function-call table references because rendering now takes the name from TableReference.TableFunc. replaceTableInFrom/replaceTableInJoins now go through renameTableRef, which also renames TableFunc when its name matches; the arguments are kept (FROM my_udtf(1) -> FROM my_udtf_v2(1)). Documented on ReplaceTable; tests cover parsed PostgreSQL and ClickHouse FROM refs, JOIN refs, and a TableFunc whose name differs. F3 (security): the AST scanner now checks TableReference.TableFunc in FROM and on the right side of JOINs: ClickHouse table functions that reach files, URLs, object stores, other servers or programs (url, file, remote, s3, mysql, postgresql, executable, ...) are reported as OUT_OF_BAND, and the call arguments go through the dangerous-function checks. It also walks ArraySubscriptExpression and ArraySliceExpression, so sleep(5)[1] and pg_sleep(5)[1:2] are no longer hidden. ScanSQL gains a FROM/JOIN <table function>( pattern for the same functions. Tests cover detection and no false positives for numbers(), generate_series(), unnest() and groupArray(x)[1]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
1156be8 to
508b90a
Compare
|
Claude finished @ajitpratap0's task in 24s —— View job Review of PR #628 (ClickHouse parametric aggregates, FROM table functions, rendering)I read the non-test source diff (parser, ast, formatter, security scanner, transform). I did not build the project or run the tests. I did not read the new test files in full. Overall: The changes are well scoped and the dialect gating is tight. I found no blocking bugs. The earlier automated review did not cover the What looks good
Points to consider
Performance and security
TestsThe reported regression tests fail on the parent commit and pass on HEAD, and the race gate is clean. I'd add the following:
Verdict: Approve with minor, non-blocking suggestions (points 1 and 3). |
SQL Server tokenizes [cnt] as a bracket-quoted identifier, so count(*) [cnt] must stay an aliased call and never become a subscript. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
|
Merge train (batch 3): rebased onto main; head 200644c.
|
|
Claude finished @ajitpratap0's task in 20s —— View job Review of PR #628 (ClickHouse parametric aggregates, FROM table functions, rendering)I read the source diffs for the parser, ast, formatter, security and transform changes, and the full diffs of Overall: The changes are well scoped, dialect gating is tight, and I found no blocking bugs. Two points are worth a look before merge: a possible duplicate scanner finding (#1) and a false-positive risk in the scanner regex (#2). Good
Points to consider
Performance and security
TestsThe PR reports regression tests that fail on the parent and pass on HEAD, plus a clean race gate. I'd add:
Verdict: Approve with minor, non-blocking suggestions (#1, #3, #4). |
…d costs - #628: SQLite still rejects count(*) [cnt]; only SQL Server keeps it. - #631: bare FORMAT / FORMAT <name> rules apply after SETTINGS. - Performance: record the measured costs of the batch-3 correctness fixes (Identifier and TableReference growth, quoting in SQL(), lineStarts reallocation, sorted Extract results). - #632: note the lıke, WITH CHECK OPTION and MINUS rendering changes. - Fix the [Unreleased] compare link (v1.14.0...HEAD) and add [1.14.0]. - Keep each section a single tight list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
…xes; refresh performance baselines (#566) (#598) * docs: consolidate CHANGELOG for the perf audit and batch-2 fixes; refresh performance baselines (#566) CHANGELOG [Unreleased] now covers every PR merged since v1.14.0, with a Behavior changes section for the observable changes in #597, #595, #584, #585, #588, #581, #582/#570, #517/#518 and #511, plus Added, Changed, Fixed, Performance, Security and Internal entries. The two existing entries (#526 and the tokenizer-pool dialect fix) are kept verbatim. performance_baselines.json is re-measured on an Apple M4 with BenchmarkPerformanceBaseline -count=10 -benchmem (median). Baselines are set at about 2.5x the M4 median rather than 1.5x: the CI Benchmark job on ubuntu-latest measures these queries about 1.5-3.2x slower than M4 (last 9 Tests runs on main, 0715bb9..61ca6c8), so 1.5x with the unchanged tolerances would fail CI's slowest runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve * docs(changelog): add batch-3 follow-up fixes (#599-#620) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve * docs(changelog): record #632 render-injection hardening and #624 quote guards Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve * docs(changelog): correct batch-3 entries after review; record measured costs - #628: SQLite still rejects count(*) [cnt]; only SQL Server keeps it. - #631: bare FORMAT / FORMAT <name> rules apply after SETTINGS. - Performance: record the measured costs of the batch-3 correctness fixes (Identifier and TableReference growth, quoting in SQL(), lineStarts reallocation, sorted Extract results). - #632: note the lıke, WITH CHECK OPTION and MINUS rendering changes. - Fix the [Unreleased] compare link (v1.14.0...HEAD) and add [1.14.0]. - Keep each section a single tight list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve * docs(changelog): describe the release as source-compatible with runtime behaviour changes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
#605 parser/clickhouse: parametric aggregates move DISTINCT to the wrong group and reject chained calls or subscripts
In the ClickHouse fn(params)(args) path, the parser now keeps DISTINCT with the argument group: quantile(0.5)(DISTINCT x) parses and round-trips through SQL() and FormatAST in compact and readable styles. DISTINCT or ORDER BY in the parameter group (uniqExact(DISTINCT x)(y), groupArray(DISTINCT 3)(x)) and a third paren group (f(1)(2)(3)) now fail with E2004 errors that point at the offending token. Before, they were either silently moved into SQL that did not re-parse, or failed with a misleading E2002. A [...] subscript or slice directly after an identifier-named function call (groupArray(x)[1], quantiles(0.5, 0.9)(x)[1], PostgreSQL array_agg(x)[1]) now parses as ArraySubscriptExpression or ArraySliceExpression over the call and round-trips. SQL Server's count(*) [cnt] still renders as AS cnt. The new regression tests fail on 61ca6c8 and pass on this commit, and the full race gate, go vet and golangci-lint are clean. Deferred, with sound reasons:
gosqlx formatsilently corrupts many expressions; switch the CLI to pkg/formatter #599.Reviewer note: in SQLite,
count(*) [cnt]now parses as a subscript instead of erroring, because SQLite bracket identifiers tokenize as LBRACKET. This matches the existing identifier behaviour but should be gated or followed up.#606 parser/clickhouse: table functions in FROM (numbers(10), remote(...)) fail to parse
Fixes #606. ClickHouse table functions in FROM (numbers(10), numbers(10) AS n, remote(...),
t, numbers(3), file, s3, cluster, generateRandom, plus the keyword-named url, merge, values and null) now parse into a TableReference with TableFunc set, instead of failing with E2002 'expected statement, got LPAREN' or 'expected identifier, got URL'. The fix adds DialectClickHouse to supportsTableFunction(). It also adds a ClickHouse-only branch for keyword-tokenized function names, which applies only when the name is directly followed by '('. Parse behaviour for every other dialect is unchanged; a parent-vs-HEAD probe across mysql, mariadb, sqlite, sqlserver, oracle, generic and postgresql showed no differences. Verified with the regression tests (they fail on the parent at the issue's exact positions and pass on HEAD), the full race gate, go vet and golangci-lint (0 issues). Deferred, with sound reasons: (1) rendering of TableFunc arguments, sonumbers(10)still renders asnumbers; this is #607, the next commit in this chain, which must add ClickHouse round-trip cases; (2) table functions on the right side of a JOIN (t JOIN numbers(5) AS n), a separate parseJoinedTableRef path that affects other dialects and overlaps the from-joins chain; (3) enabling table functions in the generic dialect; (4) the other pre-existing ClickHouse gaps listed in the issue: view(SELECT ...), url as a column, INSERT INTO FUNCTION, FINAL/SETTINGS dropped on render, and GLOBAL JOIN.#607 ast/formatter: SQL() and FormatAST drop table-function call arguments in FROM (generate_series(1,10) -> generate_series)
Fixes #607.
ast.writeTableRef(used by SQL()) andformatter.tableRefSQL(used by FormatAST) now render the parsedTableReference.TableFunccall. Before, they wrote onlyTableReference.Name, sogenerate_series(1,10) AS gcame out asgenerate_series gand the query's meaning changed. This affects PostgreSQL generate_series/unnest/json_each, Snowflake IDENTIFIER/TABLE(...)/FLATTEN, BigQuery UNNEST and ClickHouse numbers/remote/url/merge/values/null/file/cluster. Verified with new regression tests that fail on the parent commit and pass on HEAD, byte-identical round trips through SQL() and the Compact formatter in all four dialects, and the full -race suite, vet and golangci-lint. Deferred:FLATTEN(input => t.arr)needs named-argument rendering from ast/formatter: SQL() and pkg/formatter drop IGNORE NULLS, named arguments, ON DUPLICATE KEY UPDATE and INSERT OUTPUT; parser discards SELECT ... SETTINGS #600. Until then it rendersFLATTEN(input); the test asserts the full form automatically once=>appears.gosqlx formatpath still drops the arguments until cli:gosqlx formatsilently corrupts many expressions; switch the CLI to pkg/formatter #599 moves it onto pkg/formatter.Parser gaps found while probing, to file as follow-ups:
UNNEST([1,2,3])fails with E2001.WITH ORDINALITYand column-alias lists after a table function.outer => TRUE.Commits (4), one implementation commit per issue plus test-only follow-ups:
13 files changed, 747 insertions(+), 26 deletions(-)Test Plan
CGO_ENABLED=1 go test -race -timeout 300s $(go list ./... | grep -v /cbinding)),go vet,golangci-lint, re-ran the reproduction cases on HEAD, and confirmed the new tests fail on the parent commit.benchstat -count=6before/after on the same machine.#605 evidence (validator)
#605 race gate: CGO_ENABLED=1 go test -race -timeout 300s $(go list ./... | grep -v /cbinding) passed with EXIT 0. 44 packages reported ok, with no FAIL and no DATA RACE. Output is in scratchpad/b3/605/race.txt. Targeted run on HEAD: TestClickHouseParametric_DistinctInArgs_605, TestClickHouseParametric_RejectsModifiersInParams_605, TestFunctionCallSubscript_605, TestFormatAST_ClickHouseParametricAggregates and the existing TestClickHouseParametricAggregates, _ASTShape and _Render (#482/#531) all PASS.
#606 evidence (validator)
#606 race gate: PASS. Ran
CGO_ENABLED=1 go test -race -timeout 300s $(go list ./... | grep -v /cbinding)on HEAD a63c92b and it exited 0. All 44 test packages reported ok and none failed. pkg/sql/parser (9.27s), pkg/gosqlx (3.25s) and pkg/formatter (1.94s) ran fresh, not from cache. Log: /private/tmp/claude-501/-Users-ajitpratapsingh-dev-GoSQLX/b99cd3cf-cb92-4294-a0db-d6a7c9a86253/scratchpad/b3/606/race.txt#607 evidence (validator)
#607 race gate: PASS. Ran
CGO_ENABLED=1 go test -race -timeout 300s $(go list ./... | grep -v /cbinding)on HEAD 24183bd and it exited 0. 44 packages reported ok with no FAIL lines, including pkg/formatter (4.6s), pkg/gosqlx (8.6s), pkg/sql/ast (6.5s) and pkg/sql/parser (19.1s). There were no flakes, so nothing was re-run. Log: /private/tmp/claude-501/-Users-ajitpratapsingh-dev-GoSQLX/b99cd3cf-cb92-4294-a0db-d6a7c9a86253/scratchpad/b3/607/race.txtBreaking Changes
Notes
gosqlx formatsilently corrupts many expressions; switch the CLI to pkg/formatter #599-schema/postgres: loader misreports index columns and key flags (column order, INCLUDE, expression and partitioned indexes, read-only roles) #620); later PRs in the train are rebased onto this one after it merges.main.Fixes #605
Fixes #606
Fixes #607
Follow-up to #566
🤖 Generated with Claude Code
https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve