Repository navigation
fix(cli): render gosqlx format through pkg/formatter and refuse non-round-tripping output (#599) - #624
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Claude finished @ajitpratap0's task in 25s —— View job Review of PR #624 (
|
gosqlx format silently corrupts many expressions; switch the CLI to pkg/formatter (#599)9c5a127 to
701763d
Compare
F1 (must-fix, regression): gosqlx format rewrote a non-executing EXPLAIN
into executable SQL. The parser splits "EXPLAIN VERBOSE|ANALYZE <stmt>"
into DescribeStatement{TableName: "VERBOSE"} plus <stmt>; the old CLI
renderer refused DescribeStatement, but rendering through pkg/formatter
accepted it, and the SQL() round-trip check is equally lossy, so
`format -i` turned "EXPLAIN VERBOSE DELETE FROM users;" into
"DESCRIBE VERBOSE; DELETE FROM users" with exit 0 (same for UPDATE,
DROP, TRUNCATE, INSERT, EXPLAIN ANALYZE and MySQL EXPLAIN ANALYZE).
formatSQL now refuses, before rendering, DescribeStatement (drops
EXPLAIN options, the explained query and DESCRIBE object kinds) and
UnsupportedStatement (rendered as a comment), restoring the base
"unsupported statement type" error and leaving input unchanged. As a
general guard it also refuses any parse with more statements than
semicolon-separated segments in the input, since a statement split by
the parser cannot be detected by re-parsing the output.
The underlying parser bug (EXPLAIN VERBOSE/ANALYZE splitting into two
statements, DescribeStatement losing options and object kind) still
needs its own issue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
TestFormatCommand_JoinUsing_Issue603 and TestFormatCommand_InsertUpdateForms landed on main while #624 was open and pinned the old hand-written CLI renderer's layout. With the CLI now rendering through pkg/formatter, a lone SELECT * stays on one line and table aliases render without AS (u v), the same layout as the rest of the #624 corpus. USING (...) and UPDATE ... FROM lists still round-trip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1rUqknXo3RVxa3jGASwDq
|
Rebased onto main (1be0808, includes #622, #623, #628–#631) to clear conflicts.
Note for whoever owns |
|
Claude finished @ajitpratap0's task in 24s —— View job Review of PR #624 (
|
…ound-tripping output (#599) `gosqlx format` used its own 1,476-line renderer in cmd/gosqlx/cmd/sql_formatter.go. Any expression type it did not handle fell back to TokenLiteral(), and formatInsert dropped every clause after VALUES/SELECT. The command exited 0 with SQL of a different meaning, or with SQL that does not parse (CAST -> `CAST`, `a = ANY(...)` -> `ANY`, tuples -> `TUPLE`, `arr[1]` -> `[]`, ROLLUP/CUBE/GROUPING SETS, WITHIN GROUP, `EXISTS (SELECT 1 UNION SELECT 2)` -> `EXISTS ()`, ON CONFLICT/RETURNING dropped, PIVOT dropped, lossy ALTER TABLE). - sql_formatter.go is now a thin adapter. SQLFormatter, NewSQLFormatter, FormatterOptions and Format keep their signatures. Each statement is rendered with pkg/formatter.FormatStatement, with indent, compact and keyword case mapped to ast.FormatOptions. AddSemicolon is off and the CLI keeps its "; " / ";\n\n" separators. FormatterOptions gains an additive Dialect field. AlignColumns is retained but ignored. All parallel rendering code is deleted. - formatSQL now verifies its output. It re-parses the rendering with the same dialect and requires the same statement count and a case-insensitive equal SQL(). Otherwise it returns a "formatted output does not re-parse / changes the query" error, so stdout, -o and -i never receive corrupted SQL and --check counts the file as failed. Lowercase-keyword output is verified via its uppercase twin, because the parser rejects some lowercase keyword sequences. - pkg/formatter renders *ast.AlterStatement ALTER TABLE operations (ADD COLUMN, ADD CONSTRAINT, DROP COLUMN/CONSTRAINT [CASCADE], RENAME TO, RENAME COLUMN, ALTER COLUMN) in a new render_alter.go. Before, it rendered a bare "ALTER". Non-table ALTER keeps its previous output. render.go gains one switch case. Tests: - TestFormatSQL_Issue599 covers every reproduction in the issue (compact output must match exactly; readable output must round-trip and be idempotent). Rows that pkg/formatter itself still loses skip with the blocking issue and self-heal when the fix lands. - TestFormatSQL_CorpusRoundTrip formats all 134 parseable testdata/**/*.sql files in three modes, re-parses them, compares SQL() and checks idempotence. 8 documented known gaps may error but never emit changed SQL. - TestFormatStatement_AlterTable* cover the new ALTER rendering. Blocked on library fixes (these now match pkg/formatter and are fixed automatically when they land): - #600: named args, IGNORE NULLS, SETTINGS, ON DUPLICATE KEY UPDATE, OUTPUT - #603: JOIN USING - #604: WITH (NOLOCK) - #601: NOT EXISTS (now an explicit error) - #611: quoted aliases and identifiers (now an explicit error) Follow-ups to file: - CROSS/OUTER APPLY rendering - MATCH ... AGAINST rendering - Oracle CONNECT BY rendering - Snowflake QUALIFY rendering - backslash literals in dialects that keep them literal - ALTER ROLE/POLICY/CONNECTOR rendering - --no-uppercase flag has no effect - parser case-sensitivity for lowercase `returning` / `cross join lateral` CLI behaviour change: readable output layout now follows pkg/formatter, and output that would not round-trip is now an error with a non-zero exit instead of exit 0. Fixes #599 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
… errors (#599) formatSQL wrapped its errors with "formatting failed:" and both format command paths wrapped them again, so users saw "formatting failed: formatting failed: ...". formatSQL now uses step-specific wrapping and returns the self-describing round-trip errors unchanged; the command keeps the single prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
F1 (must-fix, regression): gosqlx format rewrote a non-executing EXPLAIN
into executable SQL. The parser splits "EXPLAIN VERBOSE|ANALYZE <stmt>"
into DescribeStatement{TableName: "VERBOSE"} plus <stmt>; the old CLI
renderer refused DescribeStatement, but rendering through pkg/formatter
accepted it, and the SQL() round-trip check is equally lossy, so
`format -i` turned "EXPLAIN VERBOSE DELETE FROM users;" into
"DESCRIBE VERBOSE; DELETE FROM users" with exit 0 (same for UPDATE,
DROP, TRUNCATE, INSERT, EXPLAIN ANALYZE and MySQL EXPLAIN ANALYZE).
formatSQL now refuses, before rendering, DescribeStatement (drops
EXPLAIN options, the explained query and DESCRIBE object kinds) and
UnsupportedStatement (rendered as a comment), restoring the base
"unsupported statement type" error and leaving input unchanged. As a
general guard it also refuses any parse with more statements than
semicolon-separated segments in the input, since a statement split by
the parser cannot be detected by re-parsing the output.
The underlying parser bug (EXPLAIN VERBOSE/ANALYZE splitting into two
statements, DescribeStatement losing options and object kind) still
needs its own issue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
F2: the format round-trip check passed trivially for statements without a SQL() method (ALTER, sequences), since AST.SQL() skips them and "" equals "". verifyFormattedSQL now also compares a canonical pkg/formatter rendering of every statement, and compares the quoted tokens of the input and output: each quoted identifier must stay quoted or mean the same identifier bare (ASCII word, not a keyword for the dialect, and unchanged by the dialect's case folding), so `ADD COLUMN "order"`, "Mixed" and MySQL `select` are refused instead of written bare with exit 0. This also refuses SELECT "Mixed" and corpus files gorm_joins.sql (alias "CompanyName"), oracle/14_pivot.sql and 15_unpivot.sql (PIVOT IN 'North' rendered as the identifier North), which were previously written changed; they are listed as known gaps. F3 (guard only): every input string literal value must appear in the output, so backslash literals in dialects where backslash is not an escape are refused with an explicit "changes string literal" error. The renderer still re-escapes backslashes MySQL-style; making LiteralValue rendering dialect-aware needs the dialect threaded into pkg/sql/ast SQL() and is left as a follow-up. F5: UnaryExpression.SQL() wrote nested unary minus as "--", which starts a line comment; it now writes "- -1". Tests in pkg/sql/ast and pkg/formatter. F6: DESCRIBE/DESC were already refused by checkFormattable (previous commit); added Snowflake DESCRIBE STAGE / VIEW / DESC TABLE regression tests. F7: doc.go, formatter.go and sql_formatter.go comments now describe the check as a comparison of the input's parse with the output's parse, and state that what the parser drops (comments, skipped clauses) is not detected. F9: format --check no longer prints "All N files are properly formatted" when some files failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
…test TestFormatCommand_JoinUsing_Issue603 (#603) pinned the old CLI renderer's readable layout ("SELECT\n *\nFROM ..."). #599 routes `gosqlx format` through pkg/formatter, whose readable layout keeps a single-item select list on the SELECT line ("SELECT *\nFROM ..."). Only the readable expectation changes; the USING assertions (no "ON" rendering, re-parsed Using columns, nil Condition) are unchanged. Integration fix for #603 x #599; port into whichever of the two lands second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
With #601 (NOT EXISTS renders as a prefix NOT) and #599 (gosqlx format via pkg/formatter) both applied, corpus/tpch/q21_suppliers_kept_waiting.sql and mysql/19_not_exists.sql round-trip in every mode, so they no longer belong in corpusFormatKnownGaps. Keeping them would let a regression of The ent_queries.sql reason now names what is still missing after #611: the quoted qualifier in "groups"."id" (Identifier.Table quoting was deferred by #611). Integration fix for #599 x #601/#611; port into whichever lands second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
…sts with #632/#623 Integration with #632 (now on main), which quotes reserved-word names on render: - ALTER TABLE ... ADD COLUMN "order" now formats correctly, so the guard test expects the quoted output instead of a refusal. - In MySQL/MariaDB a backtick keyword such as `select` would render as "select", which MySQL reads as a string unless ANSI_QUOTES is set. The new checkMySQLDoubleQuotes refuses output that adds double-quoted tokens the input did not have, keeping the CLI's previous safe refusal. Integration with #623: pkg/formatter writes table aliases without AS, so the UPDATE ... FROM list expectation becomes "u v". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
701763d to
50cb701
Compare
|
Claude finished @ajitpratap0's task in 31s —— View job Review of PR #624 (
|
Review follow-ups for the rebase integration:
- checkMySQLDoubleQuotes compared only counts, so a double-quoted MySQL
string written bare could cancel out an identifier newly written in ANSI
quotes. It now requires the output to keep exactly the input's
double-quoted tokens, which also refuses WHERE name = "bob" -> name = bob
(a string silently becoming a column reference). Input that uses "..." as
a name where MySQL cannot accept a string ("users"."id", FROM "users") is
ANSI_QUOTES-style and is not checked; no MySQL/MariaDB corpus file is
refused.
- TestSQLFormatter_NotExistsDoesNotPanic: the #601 Skipf is unreachable now
that #632 is on main; fail instead.
- The corpus round-trip test now fails when a known-gap file starts to
format, so the list cannot go stale.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
|
Claude finished @ajitpratap0's task in 21s —— View job Review of PR #624 (
|
|
Merge train (batch 3): #624 (last code PR of the train)
|
…e guards Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
…xes; refresh performance baselines (#566) (#598) * docs: consolidate CHANGELOG for the perf audit and batch-2 fixes; refresh performance baselines (#566) CHANGELOG [Unreleased] now covers every PR merged since v1.14.0, with a Behavior changes section for the observable changes in #597, #595, #584, #585, #588, #581, #582/#570, #517/#518 and #511, plus Added, Changed, Fixed, Performance, Security and Internal entries. The two existing entries (#526 and the tokenizer-pool dialect fix) are kept verbatim. performance_baselines.json is re-measured on an Apple M4 with BenchmarkPerformanceBaseline -count=10 -benchmem (median). Baselines are set at about 2.5x the M4 median rather than 1.5x: the CI Benchmark job on ubuntu-latest measures these queries about 1.5-3.2x slower than M4 (last 9 Tests runs on main, 0715bb9..61ca6c8), so 1.5x with the unchanged tolerances would fail CI's slowest runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve * docs(changelog): add batch-3 follow-up fixes (#599-#620) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve * docs(changelog): record #632 render-injection hardening and #624 quote guards Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve * docs(changelog): correct batch-3 entries after review; record measured costs - #628: SQLite still rejects count(*) [cnt]; only SQL Server keeps it. - #631: bare FORMAT / FORMAT <name> rules apply after SETTINGS. - Performance: record the measured costs of the batch-3 correctness fixes (Identifier and TableReference growth, quoting in SQL(), lineStarts reallocation, sorted Extract results). - #632: note the lıke, WITH CHECK OPTION and MINUS rendering changes. - Fix the [Unreleased] compare link (v1.14.0...HEAD) and add [1.14.0]. - Keep each section a single tight list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve * docs(changelog): describe the release as source-compatible with runtime behaviour changes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
#599 cli:
gosqlx formatsilently corrupts many expressions; switch the CLI to pkg/formattergosqlx formatno longer has its own renderer. cmd/gosqlx/cmd/sql_formatter.go is now a thin adapter over pkg/formatter.FormatStatement, and the exported SQLFormatter, NewSQLFormatter, FormatterOptions and Format API is unchanged apart from an additive Dialect field. Before this change the CLI silently corrupted CAST,::, EXTRACT, INTERVAL, ANY/ALL, tuples, subscripts and ARRAY, ROLLUP/CUBE/GROUPING SETS, WITHIN GROUP, set-operation subqueries, ON CONFLICT, RETURNING, PIVOT and ALTER TABLE with exit 0. They now format correctly and round-trip, and pkg/formatter gains full ALTER TABLE operation rendering (one new switch case in render.go). A new safety net re-parses the formatted output with the same dialect and compares the statement count and SQL(). If they differ the command errors, and-i/-onever write corrupted SQL. On this branch alone, NOT EXISTS (#601) and quoted identifiers (#611) therefore error until the ast-render chain lands, so merge after it. Verified with a full -race gate, task test:cbinding, vet and golangci-lint (0 issues). An independent before/after CLI comparison covered all 25 scout cases. The new regression tests fail on the parent: 22 TestFormatSQL_Issue599 subtests plus the corpus and ALTER render tests. A 134-file corpus round-trip runs in 3 modes. Still open, with the CLI inheriting each fix automatically: #600, #603 and #604, plus follow-ups for MySQL-style backslash escaping in every dialect, CROSS/OUTER APPLY rendered as... APPLY JOIN, MATCH ... AGAINST, Oracle CONNECT BY, Snowflake QUALIFY, ALTER ROLE/POLICY/CONNECTOR, the no-op--no-uppercaseflag, and lowercase keywords the parser rejects. A newly observed pre-existing parser bug also needs its own issue: lowercaseany(...)is parsed as ALL.Commits (2), one implementation commit per issue plus test-only follow-ups:
gosqlx formatsilently corrupts many expressions; switch the CLI to pkg/formatter #599)gosqlx formatsilently corrupts many expressions; switch the CLI to pkg/formatter #599)9 files changed, 772 insertions(+), 1435 deletions(-)Test Plan
CGO_ENABLED=1 go test -race -timeout 300s $(go list ./... | grep -v /cbinding)),go vet,golangci-lint, re-ran the reproduction cases on HEAD, and confirmed the new tests fail on the parent commit.benchstat -count=6before/after on the same machine.#599 evidence (validator)
#599 race gate: PASS.
CGO_ENABLED=1 go test -race -timeout 300s $(go list ./... | grep -v /cbinding)on HEAD b4b76f2 exited 0. 44 packages reported ok with no FAIL, panic or DATA RACE; cmd/gosqlx/cmd (2.3s) and pkg/formatter (2.2s) ran fresh, not cached.task test:cbindingreported ok (cached). Log: /private/tmp/claude-501/-Users-ajitpratapsingh-dev-GoSQLX/b99cd3cf-cb92-4294-a0db-d6a7c9a86253/scratchpad/b3/599/race_validator.txtBreaking Changes
gosqlx formatsilently corrupts many expressions; switch the CLI to pkg/formatter #599: Go API: additive only.SQLFormatter,NewSQLFormatter(FormatterOptions),FormatterOptions{Indent, Compact, UppercaseKw, AlignColumns}and(*SQLFormatter).Format(*ast.AST) (string, error). FormatterOptions gains a newDialect stringfield. AlignColumns is retained but documented as ignored. Unexported render helpers are removed.*ast.AlterStatementALTER TABLE operations where they used to return the bare tokenALTER, so output only improves. Other ALTER types are unchanged.Observable CLI behaviour:
gosqlx formatoutput layout now matches pkg/formatter (readable or compact, CLI separators;\n\n/;, no trailing semicolon).The PR must describe both CLI changes under Breaking Changes (CLI output and exit behaviour only).
Notes
gosqlx formatsilently corrupts many expressions; switch the CLI to pkg/formatter #599-schema/postgres: loader misreports index columns and key flags (column order, INCLUDE, expression and partitioned indexes, read-only roles) #620); later PRs in the train are rebased onto this one after it merges.main.Fixes #599
Follow-up to #566
🤖 Generated with Claude Code
https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve