Skip to content

fix(cli): render gosqlx format through pkg/formatter and refuse non-round-tripping output (#599) - #624

Merged
ajitpratap0 merged 8 commits into
mainfrom
fix/cli-format-use-pkg-formatter
Oct 14, 2026
Merged

ajitpratap0 merged 8 commits into
mainfrom
fix/cli-format-use-pkg-formatter

Conversation

@ajitpratap0

Copy link
Copy Markdown
Owner

Summary

#599 cli: gosqlx format silently corrupts many expressions; switch the CLI to pkg/formatter

gosqlx format no longer has its own renderer. cmd/gosqlx/cmd/sql_formatter.go is now a thin adapter over pkg/formatter.FormatStatement, and the exported SQLFormatter, NewSQLFormatter, FormatterOptions and Format API is unchanged apart from an additive Dialect field. Before this change the CLI silently corrupted CAST, ::, EXTRACT, INTERVAL, ANY/ALL, tuples, subscripts and ARRAY, ROLLUP/CUBE/GROUPING SETS, WITHIN GROUP, set-operation subqueries, ON CONFLICT, RETURNING, PIVOT and ALTER TABLE with exit 0. They now format correctly and round-trip, and pkg/formatter gains full ALTER TABLE operation rendering (one new switch case in render.go). A new safety net re-parses the formatted output with the same dialect and compares the statement count and SQL(). If they differ the command errors, and -i/-o never write corrupted SQL. On this branch alone, NOT EXISTS (#601) and quoted identifiers (#611) therefore error until the ast-render chain lands, so merge after it. Verified with a full -race gate, task test:cbinding, vet and golangci-lint (0 issues). An independent before/after CLI comparison covered all 25 scout cases. The new regression tests fail on the parent: 22 TestFormatSQL_Issue599 subtests plus the corpus and ALTER render tests. A 134-file corpus round-trip runs in 3 modes. Still open, with the CLI inheriting each fix automatically: #600, #603 and #604, plus follow-ups for MySQL-style backslash escaping in every dialect, CROSS/OUTER APPLY rendered as ... APPLY JOIN, MATCH ... AGAINST, Oracle CONNECT BY, Snowflake QUALIFY, ALTER ROLE/POLICY/CONNECTOR, the no-op --no-uppercase flag, and lowercase keywords the parser rejects. A newly observed pre-existing parser bug also needs its own issue: lowercase any(...) is parsed as ALL.

Commits (2), one implementation commit per issue plus test-only follow-ups:

9 files changed, 772 insertions(+), 1435 deletions(-)

Test Plan

  • Each issue's reproduction cases were run against main before the fix (Haiku scout), and a failing regression test was written before the code change.
  • Independent review (Opus, or Sonnet on small chains) with a bounded fix loop.
  • Independent Opus validator ran the repository race gate (CGO_ENABLED=1 go test -race -timeout 300s $(go list ./... | grep -v /cbinding)), go vet, golangci-lint, re-ran the reproduction cases on HEAD, and confirmed the new tests fail on the parent commit.
  • Perf items: benchstat -count=6 before/after on the same machine.

#599 evidence (validator)

Not a perf item, so no benchstat. Reproduction evidence comes from two CLI builds, HEAD and its parent (git archive b4b76f2^), fed all 25 scout cases plus 12 extra probes in compact mode. Outputs are in scratchpad/b3/599/head_cases.txt and parent_cases.txt.

Parent output, showing the bug:
- `SELECT CAST, CAST FROM t`
- `SELECT EXTRACT FROM t`
- `d + INTERVAL FROM t`
- `RANGE BETWEEN INTERVAL PRECEDING`
- `WHERE ANY AND ALL`
- `WHERE TUPLE IN (TUPLE, TUPLE)`
- `SELECT [], [:], ARRAY`
- `GROUP BY ROLLUP` / `GROUP BY CUBE` / `GROUP BY GROUPING SETS`
- PERCENTILE_CONT loses its WITHIN GROUP
- `EXISTS ()` and `a IN ()`
- ON CONFLICT and RETURNING dropped
- PIVOT becomes `FROM s AS p`
- ALTER renders as `ADD c INT`, a bare `DROP`, or `...`
- `"Weird Col"` renders as `Weird Col`

All of those exit 0.

HEAD output, every case exit 0 and matching the expected text:
- `SELECT CAST(a AS INT), CAST(b AS text) FROM t`
- `EXTRACT(YEAR FROM d)`
- `d + INTERVAL '1 day'`
- `RANGE BETWEEN INTERVAL '1 day' PRECEDING AND CURRENT ROW`
- `a = ANY (SELECT x FROM u) AND b > ALL (SELECT x FROM u)`
- `(a, b) IN ((1, 2), (3, 4))`
- `arr[1], arr[1:2], ARRAY[1, 2]`
- `ROLLUP(a, b)` / `CUBE(a, b)` / `GROUPING SETS((a), (b), ())`
- `WITHIN GROUP (ORDER BY x)`
- `EXISTS (SELECT 1 UNION SELECT 2)` and `a IN (SELECT 1 UNION SELECT 2)`
- ON CONFLICT ... DO UPDATE SET a = 2 and RETURNING id are kept
- SQL Server `PIVOT (SUM(amt) FOR q IN ([Q1], [Q2])) AS p` is kept
- ALTER TABLE ADD COLUMN, DROP COLUMN ... CASCADE, RENAME TO, RENAME COLUMN, ADD CONSTRAINT ... PRIMARY KEY, DROP CONSTRAINT, ALTER COLUMN SET NOT NULL / TYPE BIGINT / DROP NOT NULL are all exact

The cases blocked on the library (#600 named argument, IGNORE NULLS, SETTINGS, ON DUPLICATE KEY UPDATE, OUTPUT; #603 USING; #604 NOLOCK) produce the same output as pkg/formatter.

NOT EXISTS and quoted aliases now exit 1 with "formatted output does not re-parse". `format -i` on NOT EXISTS input exits 1 and leaves the file byte-identical (checked 

#599 race gate: PASS. CGO_ENABLED=1 go test -race -timeout 300s $(go list ./... | grep -v /cbinding) on HEAD b4b76f2 exited 0. 44 packages reported ok with no FAIL, panic or DATA RACE; cmd/gosqlx/cmd (2.3s) and pkg/formatter (2.2s) ran fresh, not cached. task test:cbinding reported ok (cached). Log: /private/tmp/claude-501/-Users-ajitpratapsingh-dev-GoSQLX/b99cd3cf-cb92-4294-a0db-d6a7c9a86253/scratchpad/b3/599/race_validator.txt

Breaking Changes

  • cli: gosqlx format silently corrupts many expressions; switch the CLI to pkg/formatter #599: Go API: additive only.
  • cmd/gosqlx/cmd keeps SQLFormatter, NewSQLFormatter(FormatterOptions), FormatterOptions{Indent, Compact, UppercaseKw, AlignColumns} and (*SQLFormatter).Format(*ast.AST) (string, error). FormatterOptions gains a new Dialect string field. AlignColumns is retained but documented as ignored. Unexported render helpers are removed.
  • pkg/formatter: no signature changes. FormatStatement/FormatAST now render *ast.AlterStatement ALTER TABLE operations where they used to return the bare token ALTER, so output only improves. Other ALTER types are unchanged.

Observable CLI behaviour:

  • gosqlx format output layout now matches pkg/formatter (readable or compact, CLI separators ;\n\n / ; , no trailing semicolon).
  • When the formatted output would not re-parse, or would change the query, the command now returns an error (non-zero exit, file untouched) instead of exiting 0 with corrupted SQL.

The PR must describe both CLI changes under Breaking Changes (CLI output and exit behaviour only).

Notes

Fixes #599
Follow-up to #566

🤖 Generated with Claude Code

https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve

@vercel

vercel Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
gosqlx-website Ready Ready Preview Oct 14, 2026 11:50am UTC

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 25s —— View job


Review of PR #624 (gosqlx format via pkg/formatter)

I read the diff for the non-test code: sql_formatter.go, formatter.go, render_alter.go, render.go and doc.go. I did not run the tests or the linter.

Overall: this is a sound change. It replaces about 1,400 lines of duplicated renderer with a thin adapter. It also adds a re-parse safety net, so corrupted SQL is never written. The exported API stays the same, and the one addition is the Dialect field.

Strengths

  • The adapter is small, and the AlignColumns-is-ignored decision is documented.
  • verifyFormattedSQL checks statement count and SQL() after a re-parse, and the AST is released with defer, as CLAUDE.md requires.
  • Failing closed for -i and -o is the right behaviour.
  • The ALTER TABLE rendering is isolated in its own file with tests.

Issues and suggestions

  1. Case-insensitive comparison (formatter.go, verifyFormattedSQL).
    • strings.EqualFold on the whole SQL() string also ignores case differences inside string literals and identifiers.
    • A formatter bug that changed 'Abc' to 'abc' would pass verification.
    • If only some nodes keep the source keyword case, such as set operators, normalising just those would be safer. Otherwise this is a known gap, and I'd note it in a comment.
  2. Lowercase verification (formatter.go).
    • With --no-uppercase, the code verifies an uppercase rendering instead of the text that is actually written.
    • The lowercase output is therefore never re-parsed.
    • The PR acknowledges this because the parser rejects some lowercase keyword sequences. The consequence is that the lowercase path can emit text that does not re-parse (for example any(...) parsed as ALL), with no error. A short doc note on this limitation would help users.
  3. Double formatting.
    • Lowercase mode renders the tree twice, and every run does a full tokenize and parse again.
    • That is fine for a CLI, but it is worth a benchmark on large files.
  4. ALTER TABLE (render_alter.go).
    • Operation types that the parser does not produce, such as DropColumn variants beyond those handled, silently render as nothing after ALTER TABLE name. The safety net catches this in the CLI, but library users of pkg/formatter get truncated SQL.
    • Falling back to stmtSQL(a) in the default: case would be more conservative.
    • ADD COLUMN always emits COLUMN, and IF EXISTS / IF NOT EXISTS flags are not rendered. If the AST carries those flags, they would be dropped and the safety net would catch it. Please confirm the AST has no such field.
  5. Separator handling.
    • AddSemicolon = false is set, and Format appends no trailing semicolon. This is a behaviour change from before and is already listed under Breaking Changes. It is worth a mention in the release notes, since scripts may rely on the trailing ;.
  6. Merge ordering. The PR says NOT EXISTS (ast/formatter: NOT EXISTS (subquery) renders as NOT (EXISTS (...) NOT ), which does not re-parse #601) and quoted identifiers (ast/formatter: quoted identifiers that are keywords or case-sensitive lose their quotes on render (SELECT "select", CTE/view column lists, table names) #611) error on this branch alone. Please make sure the merge train enforces that order. Merging this first would be a user-visible regression, although a loud one.
  7. CHANGELOG. It is not updated here, which is intentional per the description. Please keep it on the checklist for docs: consolidate CHANGELOG for the perf audit and batch-2/batch-3 fixes; refresh performance baselines (#566) #598.

Tests: the regression and corpus round-trip tests look thorough. I'd add a test that a case-only change inside a string literal is caught by verification (relates to point 1).

Nothing here blocks the merge. Points 1 and 4 are the ones worth addressing.

@github-actions github-actions Bot added the cli label Oct 14, 2026
@ajitpratap0 ajitpratap0 changed the title cli: gosqlx format silently corrupts many expressions; switch the CLI to pkg/formatter (#599) fix(cli): render gosqlx format through pkg/formatter and refuse non-round-tripping output (#599) Oct 14, 2026
@ajitpratap0
ajitpratap0 force-pushed the fix/cli-format-use-pkg-formatter branch from 9c5a127 to 701763d Compare October 14, 2026 07:30
ajitpratap0 added a commit that referenced this pull request Oct 14, 2026
F1 (must-fix, regression): gosqlx format rewrote a non-executing EXPLAIN
into executable SQL. The parser splits "EXPLAIN VERBOSE|ANALYZE <stmt>"
into DescribeStatement{TableName: "VERBOSE"} plus <stmt>; the old CLI
renderer refused DescribeStatement, but rendering through pkg/formatter
accepted it, and the SQL() round-trip check is equally lossy, so
`format -i` turned "EXPLAIN VERBOSE DELETE FROM users;" into
"DESCRIBE VERBOSE; DELETE FROM users" with exit 0 (same for UPDATE,
DROP, TRUNCATE, INSERT, EXPLAIN ANALYZE and MySQL EXPLAIN ANALYZE).

formatSQL now refuses, before rendering, DescribeStatement (drops
EXPLAIN options, the explained query and DESCRIBE object kinds) and
UnsupportedStatement (rendered as a comment), restoring the base
"unsupported statement type" error and leaving input unchanged. As a
general guard it also refuses any parse with more statements than
semicolon-separated segments in the input, since a statement split by
the parser cannot be detected by re-parsing the output.

The underlying parser bug (EXPLAIN VERBOSE/ANALYZE splitting into two
statements, DescribeStatement losing options and object kind) still
needs its own issue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
ajitpratap0 pushed a commit that referenced this pull request Oct 14, 2026
TestFormatCommand_JoinUsing_Issue603 and TestFormatCommand_InsertUpdateForms
landed on main while #624 was open and pinned the old hand-written CLI
renderer's layout. With the CLI now rendering through pkg/formatter, a lone
SELECT * stays on one line and table aliases render without AS (u v), the
same layout as the rest of the #624 corpus. USING (...) and UPDATE ... FROM
lists still round-trip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rUqknXo3RVxa3jGASwDq
@ajitpratap0

ajitpratap0 commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner Author

Rebased onto main (1be0808, includes #622, #623, #628–#631) to clear conflicts.

Note for whoever owns GoSQLX-worktrees/fix-cli-format-use-pkg-formatter: it still has uncommitted WIP (formatter.go, ast/sql.go unary-minus fix, 4 new tests) that was NOT included because it doesn't compile yet (verifyFormattedSQL signature). Rebase it onto the new head 701763d before pushing.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 24s —— View job


Review of PR #624 (gosqlx format via pkg/formatter)

I read the diff for formatter.go, sql_formatter.go, render_alter.go and render.go. I did not run the tests or the linter. This review covers the same ground as the earlier one, and I checked its claims against the code.

Overall: this is a good change. About 1,500 lines of duplicate renderer become a thin adapter, and the re-parse safety net stops -i and -o from writing corrupted SQL.

Checked and fine

  • ALTER TABLE ... RENAME TO uses op.NewTableName.String(). NewTableName is a value ObjectName (pkg/sql/ast/alter.go:145), so there is no nil dereference.
  • AddColumn and AlterColumn guard ColumnDef != nil, and Constraint is guarded as well.
  • Both ASTs are released with defer, so the pooling rules in CLAUDE.md are followed.

Issues

  1. strings.EqualFold on the whole SQL() string (formatter.go, verifyFormattedSQL).
    • It also ignores case changes inside string literals and quoted identifiers. A formatter bug turning 'Abc' into 'abc' would pass verification. The doc comment says the fold is there for set-operator case, but the code applies it to everything.
    • Suggestion: normalise the set-operator case at the source, or compare token-wise with literals and identifiers compared exactly. If neither is done, document it as a known gap.
    • Add a test for a case-only change inside a literal.
  2. Lowercase output is never re-parsed.
    • With --no-uppercase, the check runs on the uppercase rendering, and the text that gets written is not verified. This is reasonable given the parser's lowercase gaps, but the bytes written are unchecked.
    • Suggestion: mention it in the flag help or doc.go. A cheap extra check is to tokenize the lowercase output and compare the token count with the uppercase one.
  3. writeAlterTableOperation silently emits nothing for unhandled operation types (render_alter.go).
    • There is no default:. ALTER TABLE t would render with no operation, and library users of pkg/formatter get truncated SQL. The CLI net catches this, but library callers do not benefit.
    • Suggestion: add default: return stmtSQL(a) at the statement level, or render the whole statement through the old path when the op type is unknown.
    • ADD COLUMN always prints COLUMN and drops IF [NOT] EXISTS. If AlterTableOperation has such fields, they are lost. I did not find them rendered here, so please confirm they cannot be set.
  4. Cost. In lowercase mode the tree is rendered twice, and every run does a full tokenize and parse of the output. That is acceptable for a CLI. A quick benchmark on a large file would settle it.
  5. Behaviour changes. The trailing semicolon is gone, and NOT EXISTS (ast/formatter: NOT EXISTS (subquery) renders as NOT (EXISTS (...) NOT ), which does not re-parse #601) and quoted identifiers (ast/formatter: quoted identifiers that are keywords or case-sensitive lose their quotes on render (SELECT "select", CTE/view column lists, table names) #611) now error until those PRs land. The PR description calls both out. Please make sure the merge train enforces that order, and put both items in the release notes.
  6. CHANGELOG. Not updated here, which is intentional. Keep it on the docs: consolidate CHANGELOG for the perf audit and batch-2/batch-3 fixes; refresh performance baselines (#566) #598 checklist.

Tests: the issue-599 regression tests, the corpus round-trip and the ALTER render tests give good coverage. The missing piece is the case-in-literal test from point 1.

Nothing here blocks the merge. Points 1 and 3 are the ones worth addressing.

ajitpratap0 and others added 7 commits October 14, 2026 17:11
…ound-tripping output (#599)

`gosqlx format` used its own 1,476-line renderer in
cmd/gosqlx/cmd/sql_formatter.go. Any expression type it did not handle fell
back to TokenLiteral(), and formatInsert dropped every clause after
VALUES/SELECT. The command exited 0 with SQL of a different meaning, or with
SQL that does not parse (CAST -> `CAST`, `a = ANY(...)` -> `ANY`, tuples ->
`TUPLE`, `arr[1]` -> `[]`, ROLLUP/CUBE/GROUPING SETS, WITHIN GROUP,
`EXISTS (SELECT 1 UNION SELECT 2)` -> `EXISTS ()`, ON CONFLICT/RETURNING
dropped, PIVOT dropped, lossy ALTER TABLE).

- sql_formatter.go is now a thin adapter. SQLFormatter, NewSQLFormatter,
  FormatterOptions and Format keep their signatures. Each statement is
  rendered with pkg/formatter.FormatStatement, with indent, compact and
  keyword case mapped to ast.FormatOptions. AddSemicolon is off and the CLI
  keeps its "; " / ";\n\n" separators. FormatterOptions gains an additive
  Dialect field. AlignColumns is retained but ignored. All parallel rendering
  code is deleted.
- formatSQL now verifies its output. It re-parses the rendering with the same
  dialect and requires the same statement count and a case-insensitive equal
  SQL(). Otherwise it returns a "formatted output does not re-parse / changes
  the query" error, so stdout, -o and -i never receive corrupted SQL and
  --check counts the file as failed. Lowercase-keyword output is verified via
  its uppercase twin, because the parser rejects some lowercase keyword
  sequences.
- pkg/formatter renders *ast.AlterStatement ALTER TABLE operations (ADD
  COLUMN, ADD CONSTRAINT, DROP COLUMN/CONSTRAINT [CASCADE], RENAME TO, RENAME
  COLUMN, ALTER COLUMN) in a new render_alter.go. Before, it rendered a bare
  "ALTER". Non-table ALTER keeps its previous output. render.go gains one
  switch case.

Tests:
- TestFormatSQL_Issue599 covers every reproduction in the issue (compact
  output must match exactly; readable output must round-trip and be
  idempotent). Rows that pkg/formatter itself still loses skip with the
  blocking issue and self-heal when the fix lands.
- TestFormatSQL_CorpusRoundTrip formats all 134 parseable testdata/**/*.sql
  files in three modes, re-parses them, compares SQL() and checks
  idempotence. 8 documented known gaps may error but never emit changed SQL.
- TestFormatStatement_AlterTable* cover the new ALTER rendering.

Blocked on library fixes (these now match pkg/formatter and are fixed
automatically when they land):
- #600: named args, IGNORE NULLS, SETTINGS, ON DUPLICATE KEY UPDATE, OUTPUT
- #603: JOIN USING
- #604: WITH (NOLOCK)
- #601: NOT EXISTS (now an explicit error)
- #611: quoted aliases and identifiers (now an explicit error)

Follow-ups to file:
- CROSS/OUTER APPLY rendering
- MATCH ... AGAINST rendering
- Oracle CONNECT BY rendering
- Snowflake QUALIFY rendering
- backslash literals in dialects that keep them literal
- ALTER ROLE/POLICY/CONNECTOR rendering
- --no-uppercase flag has no effect
- parser case-sensitivity for lowercase `returning` / `cross join lateral`

CLI behaviour change: readable output layout now follows pkg/formatter, and
output that would not round-trip is now an error with a non-zero exit instead
of exit 0.

Fixes #599

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
… errors (#599)

formatSQL wrapped its errors with "formatting failed:" and both format
command paths wrapped them again, so users saw
"formatting failed: formatting failed: ...". formatSQL now uses
step-specific wrapping and returns the self-describing round-trip errors
unchanged; the command keeps the single prefix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
F1 (must-fix, regression): gosqlx format rewrote a non-executing EXPLAIN
into executable SQL. The parser splits "EXPLAIN VERBOSE|ANALYZE <stmt>"
into DescribeStatement{TableName: "VERBOSE"} plus <stmt>; the old CLI
renderer refused DescribeStatement, but rendering through pkg/formatter
accepted it, and the SQL() round-trip check is equally lossy, so
`format -i` turned "EXPLAIN VERBOSE DELETE FROM users;" into
"DESCRIBE VERBOSE; DELETE FROM users" with exit 0 (same for UPDATE,
DROP, TRUNCATE, INSERT, EXPLAIN ANALYZE and MySQL EXPLAIN ANALYZE).

formatSQL now refuses, before rendering, DescribeStatement (drops
EXPLAIN options, the explained query and DESCRIBE object kinds) and
UnsupportedStatement (rendered as a comment), restoring the base
"unsupported statement type" error and leaving input unchanged. As a
general guard it also refuses any parse with more statements than
semicolon-separated segments in the input, since a statement split by
the parser cannot be detected by re-parsing the output.

The underlying parser bug (EXPLAIN VERBOSE/ANALYZE splitting into two
statements, DescribeStatement losing options and object kind) still
needs its own issue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
F2: the format round-trip check passed trivially for statements without
a SQL() method (ALTER, sequences), since AST.SQL() skips them and ""
equals "". verifyFormattedSQL now also compares a canonical
pkg/formatter rendering of every statement, and compares the quoted
tokens of the input and output: each quoted identifier must stay quoted
or mean the same identifier bare (ASCII word, not a keyword for the
dialect, and unchanged by the dialect's case folding), so
`ADD COLUMN "order"`, "Mixed" and MySQL `select` are refused instead
of written bare with exit 0. This also refuses SELECT "Mixed" and
corpus files gorm_joins.sql (alias "CompanyName"), oracle/14_pivot.sql
and 15_unpivot.sql (PIVOT IN 'North' rendered as the identifier North),
which were previously written changed; they are listed as known gaps.

F3 (guard only): every input string literal value must appear in the
output, so backslash literals in dialects where backslash is not an
escape are refused with an explicit "changes string literal" error.
The renderer still re-escapes backslashes MySQL-style; making
LiteralValue rendering dialect-aware needs the dialect threaded into
pkg/sql/ast SQL() and is left as a follow-up.

F5: UnaryExpression.SQL() wrote nested unary minus as "--", which starts
a line comment; it now writes "- -1". Tests in pkg/sql/ast and
pkg/formatter.

F6: DESCRIBE/DESC were already refused by checkFormattable (previous
commit); added Snowflake DESCRIBE STAGE / VIEW / DESC TABLE regression
tests.

F7: doc.go, formatter.go and sql_formatter.go comments now describe the
check as a comparison of the input's parse with the output's parse, and
state that what the parser drops (comments, skipped clauses) is not
detected.

F9: format --check no longer prints "All N files are properly
formatted" when some files failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
…test

TestFormatCommand_JoinUsing_Issue603 (#603) pinned the old CLI renderer's
readable layout ("SELECT\n  *\nFROM ..."). #599 routes `gosqlx format`
through pkg/formatter, whose readable layout keeps a single-item select
list on the SELECT line ("SELECT *\nFROM ..."). Only the readable
expectation changes; the USING assertions (no "ON" rendering, re-parsed
Using columns, nil Condition) are unchanged.

Integration fix for #603 x #599; port into whichever of the two lands
second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
With #601 (NOT EXISTS renders as a prefix NOT) and #599 (gosqlx format via
pkg/formatter) both applied, corpus/tpch/q21_suppliers_kept_waiting.sql
and mysql/19_not_exists.sql round-trip in every mode, so they no longer
belong in corpusFormatKnownGaps. Keeping them would let a regression of

The ent_queries.sql reason now names what is still missing after #611:
the quoted qualifier in "groups"."id" (Identifier.Table quoting was
deferred by #611).

Integration fix for #599 x #601/#611; port into whichever lands second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
…sts with #632/#623

Integration with #632 (now on main), which quotes reserved-word names on
render:
- ALTER TABLE ... ADD COLUMN "order" now formats correctly, so the guard
  test expects the quoted output instead of a refusal.
- In MySQL/MariaDB a backtick keyword such as `select` would render as
  "select", which MySQL reads as a string unless ANSI_QUOTES is set. The
  new checkMySQLDoubleQuotes refuses output that adds double-quoted tokens
  the input did not have, keeping the CLI's previous safe refusal.

Integration with #623: pkg/formatter writes table aliases without AS, so
the UPDATE ... FROM list expectation becomes "u v".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
@ajitpratap0
ajitpratap0 force-pushed the fix/cli-format-use-pkg-formatter branch from 701763d to 50cb701 Compare October 14, 2026 11:44
@github-actions github-actions Bot added the ast AST node changes label Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 31s —— View job


Review of PR #624 (gosqlx format via pkg/formatter)

I read cmd/gosqlx/cmd/formatter.go (verifyFormattedSQL) and pkg/formatter/render_alter.go on the current head. I did not run the tests or the linter.

Overall: this is a good change. It replaces about 1,500 lines of duplicate renderer with a thin adapter. The re-parse safety net keeps -i and -o from writing corrupted SQL.

Two points from the earlier automated reviews no longer apply to the current head:

  • The safety net now also compares quoted tokens, so string literals and quoted identifiers are checked. This is done in checkQuotedTokens and checkMySQLDoubleQuotes, and the comment at formatter.go:295-299 explains the split.
  • The ALTER rendering covers ALTER COLUMN and RENAME COLUMN fully.

Remaining issues

  1. Case-insensitive SQL comparison (formatter.go:330, :337).

    • strings.EqualFold runs on the whole SQL() string and on the per-statement rendering.
    • The token check covers string-literal values and quoted identifiers.
    • Case changes inside an unquoted identifier go unseen, for example Abc becoming abc. That is probably harmless in most dialects.
    • A case-only change inside a literal is covered only by the token check. Please add an explicit test for this. The fold can only hide that kind of bug if the token check misses it.
  2. --no-uppercase output is not verified directly.

    • The check runs on the form of the output that is verified, not on the bytes that get written. If that is still the case, say so in the flag help or doc.go.
    • A cheap extra check is to re-tokenize the lowercase output and compare its tokens with the uppercase tokens, folding case.
  3. Silent truncation in writeAlterTableOperation (render_alter.go:54).

    • The switch has no default:. The doc comment says unproduced types are "left empty".
    • For library users of pkg/formatter, an unhandled op.Type renders as ALTER TABLE t with nothing after it. The CLI net catches it, but library callers do not benefit.
    • Suggestion: in renderAlterStatement, fall back to stmtSQL(a) when the operation type is not handled. Alternatively, add a default: that does the same.
    • Please also confirm that AlterTableOperation has no IF [NOT] EXISTS fields. ADD COLUMN and DROP COLUMN never print them, so they would be dropped.
  4. op.NewTableName.String() (render_alter.go:93). This is fine, since the field is a value type. I'd still add one test with a schema-qualified new name.

  5. Cost. Every run tokenizes and parses the output again, and renders each statement twice more for the canonical comparison. That is fine for a CLI. A quick benchmark on a large file would settle it.

  6. Merge order and release notes.

Tests: the issue-599 regression tests, the corpus round-trip, the guard tests and the ALTER render tests give good coverage. The missing pieces are the case-in-literal test (point 1) and the lowercase verification (point 2).

Nothing here blocks the merge. Points 1 and 3 are the ones worth addressing.

Review follow-ups for the rebase integration:
- checkMySQLDoubleQuotes compared only counts, so a double-quoted MySQL
  string written bare could cancel out an identifier newly written in ANSI
  quotes. It now requires the output to keep exactly the input's
  double-quoted tokens, which also refuses WHERE name = "bob" -> name = bob
  (a string silently becoming a column reference). Input that uses "..." as
  a name where MySQL cannot accept a string ("users"."id", FROM "users") is
  ANSI_QUOTES-style and is not checked; no MySQL/MariaDB corpus file is
  refused.
- TestSQLFormatter_NotExistsDoesNotPanic: the #601 Skipf is unreachable now
  that #632 is on main; fail instead.
- The corpus round-trip test now fails when a known-gap file starts to
  format, so the list cannot go stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 21s —— View job


Review of PR #624 (gosqlx format via pkg/formatter)

I read pkg/formatter/render_alter.go and the function list of cmd/gosqlx/cmd/formatter.go on head 38b2982. I did not run the tests or the linter.

Overall: this is a good change. About 1,500 lines of duplicate renderer become a thin adapter. The re-parse safety net keeps -i and -o from writing corrupted SQL. The latest commits tighten it with quoted-token and MySQL double-quote checks, which resolves the literal and quoted-identifier concern from earlier reviews.

Remaining issues

  1. No default: in writeAlterTableOperation (render_alter.go:54).

    • The doc comment says unproduced operation types are "left empty".
    • A library caller of pkg/formatter gets ALTER TABLE t with nothing after it if an unhandled op.Type ever reaches it. The CLI net catches this, but library callers don't benefit.
    • Suggestion: fall back to stmtSQL(a) for unhandled types.
    • Please confirm that AlterTableOperation has no IF [NOT] EXISTS field. ADD COLUMN and DROP COLUMN never print one.
  2. strings.EqualFold on the whole SQL() string (formatter.go:330, :337).

    • It hides case-only changes in unquoted identifiers. That is probably harmless.
    • Literals and quoted identifiers are covered by checkQuotedTokens. Add an explicit test that a case-only change inside a literal fails verification, so the fold can't mask a later regression.
  3. --no-uppercase output is verified on its uppercase rendering, not on the bytes written.

    • A cheap extra check: re-tokenize the lowercase output and compare its tokens with the uppercase ones, folding case.
    • Otherwise, state the limitation in the flag help or doc.go.
  4. Schema-qualified RENAME TO (render_alter.go:93). op.NewTableName.String() is fine because the field is a value type. A test with a qualified new name would pin it.

  5. Cost. Each run re-tokenizes and re-parses the output and renders each statement again. That is fine for a CLI. A quick benchmark on a large file would settle it.

  6. Merge order and release notes.

Tests: the issue-599 regressions, the corpus round-trip, the guard tests and the ALTER render tests give good coverage. The missing pieces are the case-in-literal test (point 2) and lowercase verification (point 3).

Nothing here blocks the merge. Point 1 is the one worth addressing.

@ajitpratap0

Copy link
Copy Markdown
Owner Author

Merge train (batch 3): #624 (last code PR of the train)

@ajitpratap0
ajitpratap0 merged commit baf528e into main Oct 14, 2026
28 checks passed
ajitpratap0 added a commit that referenced this pull request Oct 14, 2026
…e guards

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve
ajitpratap0 added a commit that referenced this pull request Oct 14, 2026
…xes; refresh performance baselines (#566) (#598)

* docs: consolidate CHANGELOG for the perf audit and batch-2 fixes; refresh performance baselines (#566)

CHANGELOG [Unreleased] now covers every PR merged since v1.14.0, with a
Behavior changes section for the observable changes in #597, #595, #584,
#585, #588, #581, #582/#570, #517/#518 and #511, plus Added, Changed,
Fixed, Performance, Security and Internal entries. The two existing
entries (#526 and the tokenizer-pool dialect fix) are kept verbatim.

performance_baselines.json is re-measured on an Apple M4 with
BenchmarkPerformanceBaseline -count=10 -benchmem (median). Baselines are
set at about 2.5x the M4 median rather than 1.5x: the CI Benchmark job on
ubuntu-latest measures these queries about 1.5-3.2x slower than M4 (last
9 Tests runs on main, 0715bb9..61ca6c8), so 1.5x with the unchanged
tolerances would fail CI's slowest runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve

* docs(changelog): add batch-3 follow-up fixes (#599-#620)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve

* docs(changelog): record #632 render-injection hardening and #624 quote guards

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve

* docs(changelog): correct batch-3 entries after review; record measured costs

- #628: SQLite still rejects count(*) [cnt]; only SQL Server keeps it.
- #631: bare FORMAT / FORMAT <name> rules apply after SETTINGS.
- Performance: record the measured costs of the batch-3 correctness fixes
  (Identifier and TableReference growth, quoting in SQL(), lineStarts
  reallocation, sorted Extract results).
- #632: note the lıke, WITH CHECK OPTION and MINUS rendering changes.
- Fix the [Unreleased] compare link (v1.14.0...HEAD) and add [1.14.0].
- Keep each section a single tight list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve

* docs(changelog): describe the release as source-compatible with runtime behaviour changes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZSk1292jPc8ipDboNxeve

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 38b29820 Deployed Oct 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ast AST node changes cli

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cli: gosqlx format silently corrupts many expressions; switch the CLI to pkg/formatter

1 participant