From 5b500fd5c1338ec22fe8ea4c1badbdd6d35e5108 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 7 Nov 2026 19:42:25 +0000 Subject: [PATCH 1/5] Restructure README for newcomers; move detailed reference into docs/ README is now a short landing page (pitch, quick start, feature tables, a 'Good to know' summary). The full options, query, type, transaction, connection and limitation details moved unchanged in substance to docs/Options|Queries|Types|Transactions|Limitations|Generated-Code.md. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01FskpZBzDwZ1DHnoWBrmEt8 --- README.md | 444 +++++++++++++++++------------------------ docs/Architecture.md | 2 +- docs/Design.md | 2 +- docs/Generated-Code.md | 46 +++++ docs/Limitations.md | 57 ++++++ docs/Options.md | 59 ++++++ docs/Queries.md | 48 +++++ docs/Transactions.md | 96 +++++++++ docs/Types.md | 82 ++++++++ 9 files changed, 574 insertions(+), 262 deletions(-) create mode 100644 docs/Generated-Code.md create mode 100644 docs/Limitations.md create mode 100644 docs/Options.md create mode 100644 docs/Queries.md create mode 100644 docs/Transactions.md create mode 100644 docs/Types.md diff --git a/README.md b/README.md index d8273de..f789264 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,61 @@ -[![Build](https://github.com/DaredevilOSS/sqlc-gen-ruby/actions/workflows/build.yml/badge.svg?branch=main)](https://github.com/DaredevilOSS/sqlc-gen-ruby/actions/workflows/build.yml) -[![Tests](https://github.com/DaredevilOSS/sqlc-gen-ruby/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/DaredevilOSS/sqlc-gen-ruby/actions/workflows/tests.yml) +

sqlc-gen-ruby

-# sqlc-gen-ruby -A [sqlc](https://sqlc.dev) plugin that turns your SQL into Ruby methods, with RBS type signatures. -Works with `pg` (PostgreSQL), `mysql2` (MySQL) and `sqlite3` (SQLite), and Ruby 3.1 – 4.0. +

+ Write SQL. Get typed, idiomatic Ruby.
+ A sqlc plugin for PostgreSQL, MySQL and SQLite, with RBS signatures. +

-The plugin is written in pure Ruby and ships as one self-contained WebAssembly file -(the [ruby.wasm](https://github.com/ruby/ruby.wasm) interpreter with the plugin inside), so you don't need to install anything besides sqlc. +

+ Build + Tests +

+ +--- + +You write the schema and the queries: + +```sql +-- name: GetAuthor :one +SELECT * FROM authors WHERE id = $1; + +-- name: CreateAuthor :one +INSERT INTO authors (name, bio) VALUES ($1, $2) RETURNING *; +``` + +`sqlc generate` gives you plain Ruby you call like any other object: + +```ruby +queries = PgCodegen::QuerySql.connect({ dbname: 'app', host: 'localhost' }, size: 5) # a connection pool + +author = queries.create_author(name: 'Seuss', bio: nil) # => # +queries.get_author(id: author.id) # the row, or nil +queries.get_author!(id: 42) # raises QuerySql::NotFound + +queries.transaction do |tx| # COMMIT at the end, ROLLBACK if it raises + tx.create_author(name: 'a', bio: nil) +end +``` + +No ORM, no runtime dependency on the plugin, no magic: the SQL stays yours, the Ruby is generated, committed and +reviewable. + +## Why you might like it + +| | | +|---|---| +| **Three databases** | `pg`, `mysql2` and `sqlite3`, Ruby 3.1 – 4.0. | +| **Typed** | `.rbs` signatures for every model, row and method. The generated code is RuboCop- and Steep-clean. | +| **Idiomatic Ruby** | Keyword arguments, frozen `Data` rows (`Struct` below Ruby 3.2), pattern matching, doc comments from your SQL comments. | +| **Real transactions** | Nested savepoints, isolation levels, safe under `Timeout`, `Thread#kill`, fibers (`async`) and forks (Puma, Sidekiq). | +| **Everything sqlc offers** | `:one` `:many` `:exec` `:execrows` `:execlastid` `:execresult` `:batch*` `:copyfrom`, plus `sqlc.arg` / `narg` / `slice` / `embed`. | +| **Nothing to install but sqlc** | The plugin is pure Ruby shipped as one WebAssembly file ([ruby.wasm](https://github.com/ruby/ruby.wasm)). No Ruby needed to *generate*. | +| **Safe by default** | Parameters are bound (PostgreSQL / SQLite) or escaped by the driver (MySQL), and their values are never logged. | ## Quick start -**1. Point sqlc at the plugin** (`sqlc.yaml`; the URL and sha256 of each version are in its [release notes](https://github.com/DaredevilOSS/sqlc-gen-ruby/releases)): + +**1. Point sqlc at the plugin** in `sqlc.yaml`. The URL and sha256 of every version are in its +[release notes](https://github.com/DaredevilOSS/sqlc-gen-ruby/releases). + ```yaml version: "2" plugins: @@ -28,267 +74,144 @@ sql: driver: pg # or mysql2 / sqlite3 ``` -**2. Write queries** (`db/query.sql`): -```sql --- name: GetAuthor :one -SELECT * FROM authors WHERE id = $1; +**2. Write your queries** in `db/query.sql` (see above). --- name: CreateAuthor :one -INSERT INTO authors (name, bio) VALUES ($1, $2) RETURNING *; -``` +**3. Run `sqlc generate`** and `require_relative 'app/db/query_sql'`. -**3. Run `sqlc generate` and use the code:** -```ruby -require_relative 'app/db/query_sql' +Add the gems the generated code uses (`pg`, `mysql2` or `sqlite3`, and `connection_pool`), or set +`generateGemfile: true` to get a ready `Gemfile`. -queries = PgCodegen::QuerySql.connect({ dbname: 'app', host: 'localhost' }, size: 5) # a connection pool +> Want to see the output first? [docs/Examples.md](docs/Examples.md) shows the real generated code for every driver, +> built from [these schemas and queries](examples/authors). -author = queries.create_author(name: 'Seuss', bio: nil) # => # -queries.get_author(id: author.id) # the row, or nil -queries.get_author!(id: 42) # raises QuerySql::NotFound instead of returning nil +## What you get -queries.transaction do |tx| # COMMIT when the block ends, ROLLBACK if it raises - tx.create_author(name: 'a', bio: nil) -end -``` -More: [the generated code for every driver](docs/Examples.md) (it is the plugin's real output, from -[these schemas and queries](examples/authors)). +For each query file `foo.sql`, in `out:`: + +| File | Contents | +|---|---| +| `foo_sql.rb` | `FooSql`: one method per query, plus `transaction` | +| `models.rb` | one class per table, one module of constants per enum | +| `connection.rb` | shared helpers | +| `*.rbs` | type signatures for all of the above | + +How it feels to use (details in [Generated code](docs/Generated-Code.md)): + +* **Arguments** are keywords (`get_author(id:)`); missing or unknown ones raise `ArgumentError`. +* **Rows** are frozen `Data` objects (or keyword `Struct`s below Ruby 3.2) and work with pattern matching. +* **Bang finders**: every `:one` query also has `get_author!`. +* **Pooling**: `connect` builds a [`connection_pool`](https://github.com/mperham/connection_pool); or pass your own to + `QuerySql.new(pool)`. SQLite takes a `SQLite3::Database`. +* **Logging**: `connect(params, logger: Rails.logger)` logs `QuerySql#get_author (0.4ms)`, never the values. + +## Query annotations + +| Annotation | Returns | +|---|---| +| `:one` | the row or `nil` (+ a `!` finder) | +| `:many` | an `Array` of rows | +| `:exec` | `nil` | +| `:execrows` | the number of affected rows | +| `:execlastid` | the new id | +| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | +| `:batchexec` `:batchone` `:batchmany` | run the query once per element of an `Array`, on one connection | +| `:copyfrom` | bulk insert (`COPY` / `LOAD DATA` / multi-row `INSERT`) | + +Per-database notes and conflict-clause rules: [Queries](docs/Queries.md). + +## sqlc macros + +| Macro | PostgreSQL | MySQL | SQLite | +|---|:---:|:---:|:---:| +| `sqlc.arg(name)` | ✅ | ✅ | ✅ | +| `sqlc.narg(name)` (nullable, defaults to `nil`) | ✅ | ✅ | ✅ | +| `sqlc.slice(ids)` (`IN (...)` with an `Array`) | use `= ANY($1::bigint[])` | ✅ | ✅ | +| `sqlc.embed(table)` (the row holds the table's model) | ✅ | ✅ | ✅ | + +## Transactions -## Options -| Option | Default | What it does | -|---|---|---| -| `driver` | required | `pg`, `mysql2` or `sqlite3`. Must match the sqlc `engine:`. | -| `rubyVersion` | `3.3` | Oldest Ruby the code must run on, `3.1` – `4.0`. Rows are `Data` classes from 3.2 and `Struct`s below; `3` means any 3.x, so it gets what Ruby 3.1 needs (`Struct`s, the Timeout handling below 3.3). | -| `generateTypes` | `true` | Also generate `.rbs` type signatures. | -| `generateGemfile` | `false` | Also generate a `Gemfile` with the gems the code needs. | -| `moduleName` | `Codegen` | The module the code lives in, e.g. `MyApp::Db`. | -| `overrides` | none | Change the type of columns / parameters, see [below](#overrides). | -| `rename` | none | Rename tables (model class) and columns (attribute), e.g. `{authors: Writer, bio: biography}`; `schema.table` works too. | -| `exactTableNames` | `false` | Keep table names as model names (`authors` → `Authors`) instead of singularizing them (`Author`). | -| `preparedStatements` | `true` | `pg` only (an error with other drivers). `false` sends the SQL with every call instead of preparing it (needed behind pgbouncer in transaction mode). Statements dropped by `DEALLOCATE ALL` / `DISCARD ALL`, and statements PostgreSQL refuses after a migration changed the type of a column they return (`PG::FeatureNotSupported`: cached plan must not change result type), are prepared again by their next call outside a transaction (inside one, that call raises and the transaction rolls back; the next call prepares the statement again). Statements are named `Module::Class#method` (with a hash when longer than PostgreSQL's 63 bytes), so classes sharing a pool never run each other's SQL. | - -Unknown or misspelled options are errors, with a hint for near misses (`generate_types` → `generateTypes`). -Options shared by several `sql:` blocks can go in the top-level `options:` of `sqlc.yaml` under the plugin's name -(`options: {ruby: {moduleName: MyApp::Db}}`); a codegen entry's own options win, and the `overrides` of both apply. - -### Overrides -```yaml -overrides: - - column: "GetAuthor:bio" # one column of one query (:) - ruby_type: { type: "Symbol", notNull: true } - - column: "authors.bio" # a table column, in its model and in every query (.. too) - ruby_type: { type: "Symbol" } - - column: "*:bio" # a column name in every query and model - ruby_type: { type: "Pathname", notNull: false } - - db_type: "uuid" # a database type, everywhere (also the elements of uuid[] arrays) - ruby_type: { type: "String" } -``` -The first match wins in this order: `:`, `
.`, `*:`, `db_type`. -Overrides only change the RBS types; the values are whatever the driver returns. Unknown keys are errors. - -## The generated code -For each query file you get `query_sql.rb` with a `QuerySql` class (one method per query; the file name in CamelCase: -`admin_users.sql` → `AdminUsersSql` in `admin_users_sql.rb`), plus `models.rb` (one class -per table, one module of constants per enum), `connection.rb` (shared helpers) and the matching `.rbs` files. - -* **Arguments** are keywords named after the parameters (`get_author(id:)`); `sqlc.arg(name)` names them yourself. - Missing or unknown ones raise `ArgumentError`. Nullable parameters named with `sqlc.arg` / `sqlc.narg` default - to `nil`; other nullable parameters (`VALUES ($1, $2)`) must be passed, even as `nil`. -* **Rows** are frozen `Data` objects from Ruby 3.2 (with `with` and `to_h`) or mutable keyword `Struct`s below Ruby 3.2; - both support pattern matching: `case row in { id:, name: }`. -* **Bang finders:** every `:one` query also gets a `get_author!` that raises `QuerySql::NotFound` instead of returning `nil`. -* **Connections:** `QuerySql.connect(params, **pool_options)` builds a [`connection_pool`](https://github.com/mperham/connection_pool). - To use a pool you already have (or a fake in tests) pass it to `QuerySql.new(pool)`; anything with `#with` works. - For SQLite pass a `SQLite3::Database` (opened with any options, `results_as_hash: true` included): `QuerySql.new(db)`. -* **Logging:** `connect(params, logger: Rails.logger)` / `new(pool, logger:)` logs `QuerySql#get_author (0.4ms)` at - debug level for each query and transaction. Parameter values are never logged. -* **Async / fibers:** it is plain synchronous Ruby and works with the Fiber scheduler ([`async`](https://github.com/socketry/async)): - inside `Sync { ... }`, pooled `pg` and `mysql2` queries run concurrently (`mysql2` blocks the thread on Windows). MySQL parameters are quoted client-side - with `Mysql2::Client#escape` and sent with `Client#query`, not server-side prepared statements. SQLite queries run - one at a time. -* **Style:** SQL lives in frozen heredoc constants (`GET_AUTHOR_SQL`), comments above a query become its doc comment, - table / column / enum comments of the schema become doc comments of the models. The code is RuboCop-clean; where a - name from your schema breaks a RuboCop naming rule (`"UserId"`, `naïve`), that class or method gets a `rubocop:disable` - (on SQLite keep the query files ASCII, see [Limitations](#limitations)). Query names such as `HasAuthors` / `IsActive` - keep their name (`has_authors`) with `Naming/PredicatePrefix` disabled. - A result column without a usable name (`SELECT 1, now()`) is named by its position: `column_1`, `column_2`, ... - -### Transactions ```ruby queries.transaction(isolation: :serializable) do |tx| id = tx.create_author_return_id(name: 'a', bio: nil) - tx.transaction { tx.create_book(name: 'b', author_id: id) } # nested: a SAVEPOINT - :done # the block's value is returned + tx.transaction { tx.create_book(name: 'b', author_id: id) } # nested: a SAVEPOINT + :done # the block's value is returned end ``` -* Everything in the block runs on one connection. It commits when the block ends (also on `break`, `return`, `throw`) - and rolls back when it raises or the thread is killed. An interrupt (`Timeout`, `Thread#raise`, `Thread#kill`) that - arrives while `BEGIN`, `COMMIT` or `ROLLBACK` runs waits for it to finish, and a `BEGIN` / `COMMIT` that raises - itself (a stopped `Async` task) is rolled back: the connection never goes back to the pool inside a transaction. - On PostgreSQL a query interrupted by any of them (inside or outside a transaction) is cancelled at once, so the - connection goes back to the pool idle and a killed transaction rolls back without waiting for its query. - On MySQL a `Thread#kill` during a query closes that client (mysql2 cannot send `ROLLBACK` while it waits for the - query's result), so the server rolls the transaction back and the pool replaces the client. On Ruby <3.3 with timeout <0.4, `break`, `return`, and - `throw` also roll back because Timeout cannot be told apart; add `gem 'timeout', '>= 0.4'` to the generated Gemfile - to keep those exits committing; an old `Timeout` that interrupts a PostgreSQL query in a transaction still - raises `Timeout::Error` (not the `PG::InFailedSqlTransaction` of the cancelled query) after rolling back. A nested `transaction` rolls back only its own part. With `connection_pool` or - SQLite, parent-object calls from the same thread reuse the transaction connection; injected `#with` objects must do - the same. -* `tx` is a copy of the query object bound to the transaction: its queries stay in the transaction from any fiber or - thread (an `Enumerator`, an `Async` task, a `Thread`), one at a time, and raise `TransactionEnded` once the - transaction is over. Other query objects (the parent `queries`, another generated class sharing the pool) used from - another fiber or thread check out their own connection and run outside the transaction: pass `tx` instead. -* `isolation:` (`pg`, `mysql2`) takes `:read_uncommitted`, `:read_committed`, `:repeatable_read`, `:serializable`; - `mode:` (`sqlite3`) takes `:deferred`, `:immediate`, `:exclusive`. Other values raise `ArgumentError` before anything runs, and so - do `isolation:` / `mode:` on a nested `transaction` (a savepoint cannot change them). -* PostgreSQL: if a statement fails and the block rescues the error, PostgreSQL still rolls the whole transaction - back, so `transaction` raises `PG::InFailedSqlTransaction` instead of pretending it committed. In a nested - `transaction` only the savepoint is rolled back: the outer block can rescue that error and go on. -* MySQL: a deadlock (or a lock wait timeout with `innodb_rollback_on_timeout`) rolls the whole transaction back on - the server, its savepoints too. If the block rescues that error, every later query in the transaction and - `transaction` itself raise it again (a `Mysql2::Error` with the same `error_number`, its message prefixed with - `transaction rolled back:`) instead of running in autocommit mode and reporting a commit; retry the whole - `transaction`. A lock wait timeout without that setting fails only its statement: the block can rescue it and go on. -* A connection the server closed (restart, failover, killed backend) is reconnected (`pg`) or replaced in the pool - (`mysql2`) the next time it is checked out; the call that hit the dead connection still raises. Inside `transaction` - a lost `pg` connection is not reconnected: every later call in the block and `transaction` itself raise - `PG::ConnectionBad`, so nothing after the loss runs outside the transaction (the server rolled it back). A `mysql2` - client that mysql2 closed because a running query was interrupted (an `Async` task stopped, `Thread#raise`) is - replaced the same way, without failing the next call. -* SQLite: threads and fibers sharing one `SQLite3::Database` take turns; a transaction holds the database until it ends. -* `pg`: `QuerySql.connect` also takes a connection URL: `QuerySql.connect(ENV.fetch('DATABASE_URL'), size: 5)`. -* A connection shared with another library (ActiveRecord's `raw_connection`, Sequel, your own code) keeps its - settings: the generated queries pass their own type maps (`pg`; the connection's `type_map_for_results`, - `type_map_for_queries` and `field_name_type` are left as they are) or query options (`mysql2`: rows are read with - `as: :array, cast: true, cast_booleans: false`, whatever the client's or `Mysql2::Client.default_query_options`). - On SQLite `results_as_hash` and the other settings of the `SQLite3::Database` are kept too. -* Forked processes (Puma / Sidekiq / Resque workers, `Process.fork`): connections opened before the fork stay the - parent's. connection_pool's fork handling replaces the pooled ones in the child without closing the parent's - sessions, and the child lets go of every inherited connection the generated code has used (also ones another - thread had checked out at fork time, and ones the child never uses), so its exit does not end them either. A `pg` - connection used directly (not from a `ConnectionPool`) reconnects in the child; a `mysql2` client - used that way raises `Mysql2::Error::ConnectionError` in the child (open clients in the process that uses them). - SQLite: the sqlite3 gem closes writable databases a child inherits, so open the database again in the child. - On macOS, connecting from a forked child can crash in libpq's GSSAPI probe; set `gssencmode: 'disable'` (or - `PGGSSENCMODE=disable`), as usual for forking `pg` apps there. -* A `transaction` inside a transaction that another library opened on the same connection (`BEGIN`) is a savepoint - on PostgreSQL and SQLite. MySQL has no nested transactions: its `BEGIN` commits the open transaction first (see - [Limitations](#limitations)). - -## Supported features -### Query annotations -| Annotation | Returns | Notes | + +Commit on exit, rollback on exception, savepoints for nesting. `tx` stays in the transaction from any fiber or thread +and raises `TransactionEnded` once it is over. Dead connections, timeouts, killed threads, forks, deadlocks and shared +connections (ActiveRecord, Sequel) are all handled and documented in [Transactions and connections](docs/Transactions.md). + +## Options + +| Option | Default | What it does | +|---|---|---| +| `driver` | required | `pg`, `mysql2` or `sqlite3`; must match the sqlc `engine:` | +| `rubyVersion` | `3.3` | oldest Ruby the code must run on (`3.1` – `4.0`) | +| `generateTypes` | `true` | also generate `.rbs` signatures | +| `generateGemfile` | `false` | also generate a `Gemfile` with the gems the code needs | +| `moduleName` | `Codegen` | the module the code lives in, e.g. `MyApp::Db` | +| `overrides` | none | change the Ruby type of columns / parameters | +| `rename` | none | rename tables and columns, e.g. `{authors: Writer, bio: biography}` | +| `exactTableNames` | `false` | keep `authors` as `Authors` instead of singularizing to `Author` | +| `preparedStatements` | `true` | `pg` only; `false` for pgbouncer in transaction mode | + +Misspelled options are errors with a "did you mean" hint. Full reference, overrides and `sqlc.yaml`-level sharing: +[Options](docs/Options.md). + +## Types + +| SQL | Ruby | +|---|---| +| integers / floats | `Integer` / `Float` | +| `numeric`, `decimal` | `BigDecimal` | +| text, enums | `String` (enums also get constants: `ExtendedBioType::MEMOIR`) | +| dates / times | `Date` / `Time` | +| `inet`, `cidr` | `IPAddr` | +| `json`, `jsonb` (PostgreSQL) | parsed (`untyped`) | +| PostgreSQL arrays | `Array[T]` | +| nullable | `T?` | + +Types follow what each driver returns. Time-zone behavior, the exotic PostgreSQL types, MySQL `JSON`, SQLite affinity +and naming rules (keywords, acronyms, clashing columns) are in [Types and names](docs/Types.md). + +## Good to know + +The plugin is deliberately direct, so a few things work differently from what you might expect. The ones people hit +first: + +| Database | Heads-up | Do this | |---|---|---| -| `:one` | the row or `nil` | plus a bang finder (`get_author!`) | -| `:many` | `Array` of rows | | -| `:exec` | `nil` | | -| `:execrows` | number of affected rows | MySQL: rows changed, not rows matched; connect with `flags: Mysql2::Client::FOUND_ROWS` to count matched rows | -| `:execlastid` | the new id | PostgreSQL / SQLite: needs `RETURNING id`; MySQL: the auto-increment id | -| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | `last_insert_id` is `nil` on PostgreSQL (use `RETURNING`) | -| `:batchexec` / `:batchone` / `:batchmany` | `nil` / `Array` of rows or `nil` / `Array` of `Array`s of rows | take an `Array` of `Args` and run the query once per element, on one connection | -| `:copyfrom` | `nil` | bulk insert of an `Array` (or any `Enumerable`, also lazy) of `Args`; all rows or none (see below) | - -`:copyfrom` uses `COPY ... FROM STDIN` on PostgreSQL and batched multi-row `INSERT`s on SQLite. On MySQL it uses -`LOAD DATA LOCAL INFILE`, which needs `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the -server; duplicate keys and invalid values raise `Mysql2::Error` instead of being skipped. Conflict clauses are kept: -`INSERT OR REPLACE` / `OR IGNORE` / `ON CONFLICT ... DO ...` on SQLite, `INSERT IGNORE` (duplicates skipped) and -`REPLACE` on MySQL. Clauses `COPY` / `LOAD DATA` cannot express (`ON CONFLICT` / `RETURNING` on PostgreSQL, -`ON DUPLICATE KEY UPDATE` on MySQL) are generation errors; use `:batchexec` for those. -* MySQL: `LOAD DATA` names the table and columns as the `INSERT` writes them (`` `MixedCase` ``): sqlc reports MySQL - names in lowercase, but table names are case-sensitive on Linux. -* MySQL: `LOAD DATA` commits an open transaction that was started with raw SQL (`BEGIN`); call `:copyfrom` inside - `transaction` instead. - -### Macros -| Macro | PostgreSQL | MySQL | SQLite | Notes | -|---|:---:|:---:|:---:|---| -| `sqlc.arg` | ✅ | ✅ | ✅ | names a parameter (unnamed ones such as `ANY($1::BIGINT[])` become `param1`, ...) | -| `sqlc.narg` | ✅ | ✅ | ✅ | a nullable parameter (`T?`, defaults to `nil`) | -| `sqlc.slice` | 🚫 | ✅ | ✅ | `IN (sqlc.slice(ids))` takes an `Array` (an empty one: `IN` matches nothing, `NOT IN` everything). PostgreSQL: use `= ANY($1::bigint[])` with an array | -| `sqlc.embed` | ✅ | ✅ | ✅ | the row gets an attribute holding the table's model (`author`, `author2` for the same table twice); `nil` when an outer join found no row | - -🚫 = not offered by sqlc for that database. - -### Types -* Types follow what the drivers return: `Integer`, `Float`, `BigDecimal` (`numeric` / `decimal`), `String`, `Date`, `Time`, - `IPAddr`, `true` / `false`; PostgreSQL `json` / `jsonb` are parsed (`untyped`; MySQL `JSON` columns and parameters are JSON text `String`s, as - `mysql2` returns them); nullable columns and parameters are `T?`. -* PostgreSQL arrays are `Array[T]` (`Array[Array[T]]` for `T[][]`); arrays of types the pg gem does not decode - (`uuid[]`, enum arrays, ...) are `Array[String]`, also for types created after the connection was opened (a - migration run while the application is up). Geometric, range, full-text, `hstore`, `ltree`, `timetz`, `reg*` and - composite types are `String`s; domains and unknown types are `untyped`. -* MySQL: a `Time` / `DateTime` parameter (also in `:copyfrom`) keeps its instant whatever its UTC offset: it is written - in the zone `mysql2` reads `DATETIME` / `TIMESTAMP` values back in, the process's local zone, or UTC for a client - created with `database_timezone: :utc` (`connect(params.merge(database_timezone: :utc))`). The server also converts - `TIMESTAMP` columns with the session `time_zone`, so keep that equal to the client's zone. -* PostgreSQL: a `Time` or `DateTime` parameter of a `timestamp` (without time zone) column (also in `:copyfrom` and - batches) keeps its instant whatever its UTC offset: it is written in the process's local zone, the zone the pg gem - reads `timestamp` values back in (PostgreSQL itself ignores the offset of a `timestamp` value). A `DateTime` of a - `timestamptz` column keeps its instant too (the pg gem alone would send its date only); the RBS still declares - `::Time`, the type the rows return. Elements of `timestamp[]` arrays are written as given. -* A column without a time zone (PostgreSQL `timestamp`, MySQL `DATETIME`) stores the local wall-clock time, so an - instant in the hour repeated when the local zone falls back from daylight saving time (e.g. 02:30 on the last - Sunday of October in Europe/Berlin happens twice) reads back as the first of the two, one hour off for the second. - Use `timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), to keep every instant. -* A `String` passed to a PostgreSQL `json` / `jsonb` parameter is JSON text (`'{"a":1}'`): pass `'"hi"'` or - `'hi'.to_json` for a JSON string. Hashes, Arrays, numbers, `true` / `false` and `nil` are encoded for you. -* Enums are `String`s, with constants in `models.rb` (`ExtendedBioType::MEMOIR`, `ExtendedBioType::ALL`). Values that - are not valid constant names still get one: `'end'` → `END_`, `'all'` → `ALL_`, `'2x'` → `VALUE_2X`, `'a b'` → `A_B`; - values that end up with the same name get `_2`, `_3`, ... -* SQLite follows its [type affinity rules](https://www.sqlite.org/datatype3.html#determination_of_column_affinity) - (`VARCHAR(255)` → `String`, `DECIMAL(10,5)` → `Float`, `UNSIGNED BIG INT` → `Integer`); `JSON` and `UUID` columns are `String`s, - also when SQLite stored numeric-looking text as a number (`'42'` → `"42"`, in SQLite's spelling: `'1.50'` → `"1.5"`). - The generated SQLite code uses `Monitor`: if you run Steep, add `library "monitor"` to your `Steepfile`. Code - generated for a `rubyVersion` below 3.3 also uses `Timeout` (see [Transactions](#transactions)): add `library "timeout"` too. - -### Names -* Query methods and keyword arguments are snake_cased, with acronyms kept together (`MIN_ID` → `min_id`, - `HTTPServer` → `http_server`); Ruby keywords get a trailing `_` (`end_:`). Row members use the column spelling - reported by sqlc for the engine, except names that conflict with row methods, including the private ones `Data` / `Struct` call while building and copying rows (`class_`, `hash_`, `initialize_`): - PostgreSQL and SQLite keep `"camelCase"` as `:camelCase`, while MySQL reports it as `:camelcase`. A column named - `end` is the row member `:end` but the argument `end_:`. `Struct` rows (`rubyVersion` below 3.2) have more methods - than `Data` rows, so more names get the `_`: a column `count` is `:count` from 3.2 but `:count_` below - (`Struct#count` would otherwise be `Enumerable#count`); check the generated `.rbs` when changing `rubyVersion`. -* A parameter used twice is one keyword when named with `sqlc.arg`; repeated `?` / `$n` of one column become `id:`, `id_2:`. -* Models may be named like Ruby or driver constants (tables `exceptions`, `data`, `threads`, `sets`, ...): the - generated code and `.rbs` refer to those as `::Exception`, `::Data`, `::Thread`, ... so the models cannot shadow them. -* SQLite: sqlc before 1.30 (including the pinned 1.27) reports a quoted table name with its quotes, so - `CREATE TABLE "Items"` gives the model `Items` there and `Item` with sqlc 1.30 and later (PostgreSQL / MySQL give - `Item` with any version). Use `rename` or `exactTableNames` to keep the name when upgrading sqlc. -* A table `authors` next to a table `author` keeps its plural model name (`Authors`). Query names that would replace - a generated method (`Transaction`, `Initialize`, ...) are errors. - -### Limitations -* Two result columns with the same name (e.g. `a.id, b.id`) are an error: alias one (`b.id AS book_id`) or use `sqlc.embed`. -* SQLite's `NUMERIC` affinity can return an `Integer` where `Float` is declared. -* SQLite: a statement takes at most 32,766 parameters (`SQLITE_MAX_VARIABLE_NUMBER`), so a `sqlc.slice` with more - elements raises `SQLite3::SQLException: too many SQL variables`; on MySQL the limit is the server's - `max_allowed_packet`. Load larger sets into a table (`:copyfrom`) and join it instead. -* `:batchexec` / `:batchone` / `:batchmany` are not atomic on their own: wrap them in `transaction` for all-or-nothing. -* MySQL: `transaction` on a client inside a transaction that was started with raw SQL or by another library commits - that transaction first (MySQL's `BEGIN` does), so a later `ROLLBACK` no longer undoes it. Run the outer work in - `transaction` too (inner calls become savepoints), or finish the other transaction first. -* SQLite: sqlc does not see parameters in `ON CONFLICT ... DO UPDATE SET` (it leaves `sqlc.arg(...)` in the SQL or - reports fewer parameters than there are `?`), so such queries are generation errors. Use `excluded.` - (`DO UPDATE SET n = counters.n + excluded.n`) or move the value into `VALUES`. -* SQLite: `busy_timeout=` makes the sqlite3 gem wait for a locked database while holding Ruby's GVL, so other threads - and fibers stop too; `busy_handler_timeout=` waits without blocking them. -* SQLite transactions belong to the thread that opened them; fibers sharing the same database take turns, and a - transaction holds it until it ends. Only the yielded `tx` runs inside the transaction from other fibers / threads. -* SQLite: sqlc (1.27 – 1.31 at least) misplaces the SQL of a query file that contains a non-ASCII character (in a - name, literal or comment): that query and the following ones lose characters at the end (`FROM user` instead of - `FROM users`). This is a sqlc bug, not a plugin one; keep SQLite query files ASCII and pass such values as parameters. -* MySQL: sqlc lowercases the table name that `sqlc.embed` expands to (`sqlc.embed(MixedItems)` becomes - `SELECT mixeditems.itemid, ...`). MySQL table names are case-sensitive on Linux, so a mixed-case table fails with - `Unknown column 'mixeditems.itemid'`. This is a sqlc limitation the plugin cannot undo (the request only carries the - lowercased name); give the table an alias and embed that: `` SELECT sqlc.embed(m) FROM `MixedItems` AS m ``. -* Not applicable to Ruby (they exist in the C# plugin): `withAsyncSuffix` / `withCancellationToken` (use the Fiber - scheduler), `useDapper`, target framework / csproj options (`moduleName` replaces the namespace). - -# Contributing -You need Ruby 3.3 (`bundle install`), [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html) (version in -[`scripts/tool-versions.env`](scripts/tool-versions.env)), `curl` + `tar` to build the WASM, and PostgreSQL / MySQL -servers (or Docker) for the end-to-end tests. +| all | Two result columns with the same name (`a.id, b.id`) are a generation error | alias one, or use `sqlc.embed` | +| all | `:batch*` calls are not atomic | wrap them in `transaction` | +| all | `timestamp` / `DATETIME` store local wall-clock time | use `timestamptz` / `TIMESTAMP`, or run with `TZ=UTC` | +| PostgreSQL | no `sqlc.slice` | `= ANY($1::bigint[])` | +| PostgreSQL | prepared statements break behind pgbouncer (transaction mode) | `preparedStatements: false` | +| MySQL | no server-side prepared statements (client-side quoting); `:copyfrom` needs `local_infile` | enable it on client and server | +| MySQL | `sqlc.embed` lowercases table names | alias the table: `` sqlc.embed(m) FROM `MixedItems` AS m `` | +| SQLite | sqlc breaks queries in files with non-ASCII characters | keep query files ASCII, pass values as parameters | +| SQLite | `ON CONFLICT ... DO UPDATE` with `sqlc.arg` is a generation error | use `excluded.` | +| SQLite | at most 32,766 parameters per statement | `:copyfrom` into a table and join | + +Everything else, with the reasons: [Limitations](docs/Limitations.md). + +## Documentation + +| | | +|---|---| +| [Generated code](docs/Generated-Code.md) | files, rows, finders, logging, style | +| [Queries](docs/Queries.md) | annotations, `:copyfrom`, macros, arguments | +| [Transactions and connections](docs/Transactions.md) | transactions, pools, forks, Async, shared connections | +| [Types and names](docs/Types.md) | type mapping, time zones, enums, naming rules | +| [Options](docs/Options.md) | every option and `overrides` | +| [Limitations](docs/Limitations.md) | the complete list, per database | +| [Examples](docs/Examples.md) | per-driver configs and generated output | + +## Contributing + +Contributions are welcome. You need Ruby 3.3 (`bundle install`), [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html) +(version in [`scripts/tool-versions.env`](scripts/tool-versions.env)), `curl` + `tar` to build the WASM, and +PostgreSQL / MySQL servers (or Docker) for the end-to-end tests. ``` bundle exec rake test # unit + golden tests (no database needed) @@ -299,11 +222,12 @@ rake end2end_sqlite3 # end-to-end, no server needed rake end2end_pg end2end_mysql2 # end-to-end, needs servers (see docs/Testing.md) rake run_end2end_tests # all of it in Docker (docker-compose) ``` + `examples/` is the plugin's committed output and CI fails when it drifts, so regenerate it with every change to the generated code. Pull requests need both CI workflows (`build.yml`, `tests.yml`) green. -Releases are started by a maintainer (**Actions → Release → Run workflow**) and are automated from there: the full -test gate, the tag, the GitHub release with the tested `sqlc-gen-ruby.wasm` and its sha256, and a check of the published file. +Releases are started by a maintainer (**Actions → Release → Run workflow**) and are automated from there: the full test +gate, the tag, the GitHub release with the tested `sqlc-gen-ruby.wasm` and its sha256, and a check of the published file. | Doc | What | |---|---| @@ -311,5 +235,5 @@ test gate, the tag, the GitHub release with the tested `sqlc-gen-ruby.wasm` and | [docs/Testing.md](docs/Testing.md) | the test layers, CI workflows, what to run before pushing | | [docs/Releasing.md](docs/Releasing.md) | the release process and versioning rules | | [docs/Design.md](docs/Design.md) | the WASM packaging and design decisions | -| [docs/Examples.md](docs/Examples.md) | per-driver configs and links to the generated examples | +| [docs/Coverage.md](docs/Coverage.md) | every documented behavior mapped to its end-to-end test | | [CLAUDE.md](CLAUDE.md) | the same orientation condensed for AI coding assistants | diff --git a/docs/Architecture.md b/docs/Architecture.md index 721f829..5b1bc4a 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -87,5 +87,5 @@ Design decisions worth knowing (the *why* is in the comments of `drivers/base.rb `NotImplementedError`), register it in `Generator#instantiate_driver` and `Options::DRIVERS`, add `examples/authors/` + an end-to-end test file `tests/end2end_.rb`, and a service container job in `.github/workflows/tests.yml`. -**A new option:** `options.rb` (parse + default), use it where needed, document it in the README option table, and +**A new option:** `options.rb` (parse + default), use it where needed, document it in the README option table and docs/Options.md, and add a case to `test/options_test.rb` and (if it changes output) to the `codegen-options` matrix in `build.yml`. diff --git a/docs/Design.md b/docs/Design.md index 585a1e5..f66f4a0 100644 --- a/docs/Design.md +++ b/docs/Design.md @@ -45,7 +45,7 @@ transactions, nullability, `overrides`) and Ruby conventions (bang finders, logg now idiomatic Ruby rather than a copy of the C# one. ## Known limitations -See [Limitations in the README](../README.md#limitations). Design choices that look like limitations: +See [Limitations](Limitations.md). Design choices that look like limitations: * Overrides only change the RBS types; values are never converted, so the generated code stays a thin layer over the driver. * SQLite queries do not overlap: a `SQLite3::Database` is one connection, so the generated code takes one lock per database (held for a whole transaction) to keep threads and fibers from mixing their transactions. diff --git a/docs/Generated-Code.md b/docs/Generated-Code.md new file mode 100644 index 0000000..4a2af5d --- /dev/null +++ b/docs/Generated-Code.md @@ -0,0 +1,46 @@ +# The generated code + +For each query file you get `query_sql.rb` with a `QuerySql` class (one method per query; the file name in CamelCase: +`admin_users.sql` → `AdminUsersSql` in `admin_users_sql.rb`), plus: + +* `models.rb`: one class per table, one module of constants per enum, +* `connection.rb`: shared helpers, +* the matching `.rbs` files (unless `generateTypes: false`), and a `Gemfile` with `generateGemfile: true`. + +See [Examples](Examples.md) for the real output of every driver. + +## Rows + +Frozen `Data` objects from Ruby 3.2 (with `with` and `to_h`) or mutable keyword `Struct`s below Ruby 3.2; both support +pattern matching: + +```ruby +case row +in { id:, name: } then ... +end +``` + +## Arguments and finders + +* Arguments are keywords named after the parameters: see [Queries](Queries.md#arguments). +* **Bang finders:** every `:one` query also gets a `get_author!` that raises `QuerySql::NotFound` instead of returning + `nil`. + +## Logging + +`connect(params, logger: Rails.logger)` / `new(pool, logger:)` logs `QuerySql#get_author (0.4ms)` at debug level for +each query and transaction. Parameter values are never logged. + +## Style + +* SQL lives in frozen heredoc constants (`GET_AUTHOR_SQL`). +* Comments above a query become its doc comment; table / column / enum comments of the schema become doc comments of + the models. +* The code is RuboCop-clean. Where a name from your schema breaks a RuboCop naming rule (`"UserId"`, `naïve`), that + class or method gets a `rubocop:disable` (on SQLite keep the query files ASCII, see [Limitations](Limitations.md)). + Query names such as `HasAuthors` / `IsActive` keep their name (`has_authors`) with `Naming/PredicatePrefix` disabled. + +## More + +* [Transactions and connections](Transactions.md): `transaction`, pools, forks, Async. +* [Types and names](Types.md). diff --git a/docs/Limitations.md b/docs/Limitations.md new file mode 100644 index 0000000..a5f25a5 --- /dev/null +++ b/docs/Limitations.md @@ -0,0 +1,57 @@ +# Limitations + +What to know before you adopt the plugin. Most of these come from sqlc or the databases, not from the plugin. + +## All databases + +* Two result columns with the same name (e.g. `a.id, b.id`) are an error: alias one (`b.id AS book_id`) or use + `sqlc.embed`. +* `:batchexec` / `:batchone` / `:batchmany` are not atomic on their own: wrap them in `transaction` for all-or-nothing. +* Query names that would replace a generated method (`Transaction`, `Initialize`, ...) are errors. +* Large `sqlc.slice` arrays hit database limits (see SQLite and MySQL below). Load larger sets into a table + (`:copyfrom`) and join it instead. +* Timestamps without a time zone store the local wall-clock time (see [Types](Types.md#time-zones)). + +## PostgreSQL + +* `sqlc.slice` is not offered by sqlc for PostgreSQL: use `= ANY($1::bigint[])` with an array. +* `:copyfrom` cannot express `ON CONFLICT` / `RETURNING`: those are generation errors, use `:batchexec`. +* `:execresult`'s `last_insert_id` is `nil`: use `RETURNING`. +* Behind pgbouncer in transaction mode set `preparedStatements: false` ([Options](Options.md#preparedstatements-pg)). + +## MySQL + +* `transaction` on a client inside a transaction that was started with raw SQL or by another library commits that + transaction first (MySQL's `BEGIN` does), so a later `ROLLBACK` no longer undoes it. Run the outer work in + `transaction` too (inner calls become savepoints), or finish the other transaction first. +* sqlc lowercases the table name that `sqlc.embed` expands to (`sqlc.embed(MixedItems)` becomes + `SELECT mixeditems.itemid, ...`). MySQL table names are case-sensitive on Linux, so a mixed-case table fails with + `Unknown column 'mixeditems.itemid'`. This is a sqlc limitation the plugin cannot undo (the request only carries the + lowercased name); give the table an alias and embed that: `` SELECT sqlc.embed(m) FROM `MixedItems` AS m ``. +* A statement is limited by the server's `max_allowed_packet` (large `sqlc.slice` arrays). +* `:copyfrom` needs `local_infile` on client and server and commits an open raw-SQL transaction + (see [Queries](Queries.md#copyfrom)). +* No server-side prepared statements: parameters are quoted client-side. + +## SQLite + +* A statement takes at most 32,766 parameters (`SQLITE_MAX_VARIABLE_NUMBER`), so a `sqlc.slice` with more elements + raises `SQLite3::SQLException: too many SQL variables`. +* `NUMERIC` affinity can return an `Integer` where `Float` is declared. +* sqlc does not see parameters in `ON CONFLICT ... DO UPDATE SET` (it leaves `sqlc.arg(...)` in the SQL or reports + fewer parameters than there are `?`), so such queries are generation errors. Use `excluded.` + (`DO UPDATE SET n = counters.n + excluded.n`) or move the value into `VALUES`. +* sqlc (1.27 – 1.31 at least) misplaces the SQL of a query file that contains a non-ASCII character (in a name, + literal or comment): that query and the following ones lose characters at the end (`FROM user` instead of + `FROM users`). This is a sqlc bug, not a plugin one; keep SQLite query files ASCII and pass such values as + parameters. +* `busy_timeout=` makes the sqlite3 gem wait for a locked database while holding Ruby's GVL, so other threads and + fibers stop too; `busy_handler_timeout=` waits without blocking them. +* Transactions belong to the thread that opened them; fibers sharing the same database take turns, and a transaction + holds it until it ends. Only the yielded `tx` runs inside the transaction from other fibers / threads. +* sqlc before 1.30 reports quoted table names with their quotes (see [Names](Types.md#names)). + +## Not applicable to Ruby + +They exist in the C# plugin: `withAsyncSuffix` / `withCancellationToken` (use the Fiber scheduler), `useDapper`, +target framework / csproj options (`moduleName` replaces the namespace). diff --git a/docs/Options.md b/docs/Options.md new file mode 100644 index 0000000..75db913 --- /dev/null +++ b/docs/Options.md @@ -0,0 +1,59 @@ +# Options + +All options go in the `options:` of a `codegen:` entry in `sqlc.yaml`: + +```yaml +codegen: + - plugin: ruby + out: app/db + options: + driver: pg + rubyVersion: "3.3" + moduleName: MyApp::Db +``` + +Unknown or misspelled options are errors, with a hint for near misses (`generate_types` → `generateTypes`). +Options shared by several `sql:` blocks can go in the top-level `options:` of `sqlc.yaml` under the plugin's name +(`options: {ruby: {moduleName: MyApp::Db}}`); a codegen entry's own options win, and the `overrides` of both apply. + +| Option | Default | What it does | +|---|---|---| +| `driver` | required | `pg`, `mysql2` or `sqlite3`. Must match the sqlc `engine:`. | +| `rubyVersion` | `3.3` | Oldest Ruby the code must run on, `3.1` – `4.0`. Rows are `Data` classes from 3.2 and `Struct`s below; `3` means any 3.x, so it gets what Ruby 3.1 needs (`Struct`s, the Timeout handling below 3.3). | +| `generateTypes` | `true` | Also generate `.rbs` type signatures. | +| `generateGemfile` | `false` | Also generate a `Gemfile` with the gems the code needs. | +| `moduleName` | `Codegen` | The module the code lives in, e.g. `MyApp::Db`. | +| `overrides` | none | Change the type of columns / parameters, see [Overrides](#overrides). | +| `rename` | none | Rename tables (model class) and columns (attribute), e.g. `{authors: Writer, bio: biography}`; `schema.table` works too. | +| `exactTableNames` | `false` | Keep table names as model names (`authors` → `Authors`) instead of singularizing them (`Author`). | +| `preparedStatements` | `true` | `pg` only (an error with other drivers). See [below](#preparedstatements-pg). | + +## `preparedStatements` (pg) + +`false` sends the SQL with every call instead of preparing it (needed behind pgbouncer in transaction mode). + +With the default (`true`): + +* Statements dropped by `DEALLOCATE ALL` / `DISCARD ALL`, and statements PostgreSQL refuses after a migration changed + the type of a column they return (`PG::FeatureNotSupported`: cached plan must not change result type), are prepared + again by their next call outside a transaction. Inside a transaction, that call raises and the transaction rolls + back; the next call prepares the statement again. +* Statements are named `Module::Class#method` (with a hash when longer than PostgreSQL's 63 bytes), so classes sharing + a pool never run each other's SQL. + +## Overrides + +```yaml +overrides: + - column: "GetAuthor:bio" # one column of one query (:) + ruby_type: { type: "Symbol", notNull: true } + - column: "authors.bio" # a table column, in its model and in every query (.
. too) + ruby_type: { type: "Symbol" } + - column: "*:bio" # a column name in every query and model + ruby_type: { type: "Pathname", notNull: false } + - db_type: "uuid" # a database type, everywhere (also the elements of uuid[] arrays) + ruby_type: { type: "String" } +``` + +The first match wins in this order: `:`, `
.`, `*:`, `db_type`. +Overrides only change the RBS types; the values are whatever the driver returns. Unknown keys are errors. diff --git a/docs/Queries.md b/docs/Queries.md new file mode 100644 index 0000000..7a9d5e0 --- /dev/null +++ b/docs/Queries.md @@ -0,0 +1,48 @@ +# Queries + +## Annotations + +| Annotation | Returns | Notes | +|---|---|---| +| `:one` | the row or `nil` | plus a bang finder (`get_author!`) that raises `QuerySql::NotFound` instead of returning `nil` | +| `:many` | `Array` of rows | | +| `:exec` | `nil` | | +| `:execrows` | number of affected rows | MySQL: rows changed, not rows matched; connect with `flags: Mysql2::Client::FOUND_ROWS` to count matched rows | +| `:execlastid` | the new id | PostgreSQL / SQLite: needs `RETURNING id`; MySQL: the auto-increment id | +| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | `last_insert_id` is `nil` on PostgreSQL (use `RETURNING`) | +| `:batchexec` / `:batchone` / `:batchmany` | `nil` / `Array` of rows or `nil` / `Array` of `Array`s of rows | take an `Array` of `Args` and run the query once per element, on one connection. Not atomic on their own: wrap them in `transaction` for all-or-nothing | +| `:copyfrom` | `nil` | bulk insert of an `Array` (or any `Enumerable`, also lazy) of `Args`; all rows or none | + +### `:copyfrom` + +| | Implementation | +|---|---| +| PostgreSQL | `COPY ... FROM STDIN` | +| MySQL | `LOAD DATA LOCAL INFILE`: needs `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the server; duplicate keys and invalid values raise `Mysql2::Error` instead of being skipped | +| SQLite | batched multi-row `INSERT`s | + +Conflict clauses are kept: `INSERT OR REPLACE` / `OR IGNORE` / `ON CONFLICT ... DO ...` on SQLite, `INSERT IGNORE` +(duplicates skipped) and `REPLACE` on MySQL. Clauses `COPY` / `LOAD DATA` cannot express (`ON CONFLICT` / `RETURNING` +on PostgreSQL, `ON DUPLICATE KEY UPDATE` on MySQL) are generation errors; use `:batchexec` for those. + +* MySQL: `LOAD DATA` names the table and columns as the `INSERT` writes them (`` `MixedCase` ``): sqlc reports MySQL + names in lowercase, but table names are case-sensitive on Linux. +* MySQL: `LOAD DATA` commits an open transaction that was started with raw SQL (`BEGIN`); call `:copyfrom` inside + `transaction` instead. + +## Macros + +| Macro | PostgreSQL | MySQL | SQLite | Notes | +|---|:---:|:---:|:---:|---| +| `sqlc.arg` | ✅ | ✅ | ✅ | names a parameter (unnamed ones such as `ANY($1::BIGINT[])` become `param1`, ...) | +| `sqlc.narg` | ✅ | ✅ | ✅ | a nullable parameter (`T?`, defaults to `nil`) | +| `sqlc.slice` | 🚫 | ✅ | ✅ | `IN (sqlc.slice(ids))` takes an `Array` (an empty one: `IN` matches nothing, `NOT IN` everything). PostgreSQL: use `= ANY($1::bigint[])` with an array | +| `sqlc.embed` | ✅ | ✅ | ✅ | the row gets an attribute holding the table's model (`author`, `author2` for the same table twice); `nil` when an outer join found no row | + +🚫 = not offered by sqlc for that database. + +## Arguments + +Keywords named after the parameters (`get_author(id:)`); `sqlc.arg(name)` names them yourself. Missing or unknown ones +raise `ArgumentError`. Nullable parameters named with `sqlc.arg` / `sqlc.narg` default to `nil`; other nullable +parameters (`VALUES ($1, $2)`) must be passed, even as `nil`. diff --git a/docs/Transactions.md b/docs/Transactions.md new file mode 100644 index 0000000..f83fddf --- /dev/null +++ b/docs/Transactions.md @@ -0,0 +1,96 @@ +# Transactions and connections + +```ruby +queries.transaction(isolation: :serializable) do |tx| + id = tx.create_author_return_id(name: 'a', bio: nil) + tx.transaction { tx.create_book(name: 'b', author_id: id) } # nested: a SAVEPOINT + :done # the block's value is returned +end +``` + +## The basics + +* Everything in the block runs on one connection. It commits when the block ends (also on `break`, `return`, `throw`) + and rolls back when it raises or the thread is killed. A nested `transaction` is a savepoint and rolls back only its + own part. +* `isolation:` (`pg`, `mysql2`) takes `:read_uncommitted`, `:read_committed`, `:repeatable_read`, `:serializable`; + `mode:` (`sqlite3`) takes `:deferred`, `:immediate`, `:exclusive`. Other values raise `ArgumentError` before anything + runs, and so do `isolation:` / `mode:` on a nested `transaction` (a savepoint cannot change them). +* `tx` is a copy of the query object bound to the transaction: its queries stay in the transaction from any fiber or + thread (an `Enumerator`, an `Async` task, a `Thread`), one at a time, and raise `TransactionEnded` once the + transaction is over. Other query objects (the parent `queries`, another generated class sharing the pool) used from + another fiber or thread check out their own connection and run outside the transaction: pass `tx` instead. +* With `connection_pool` or SQLite, parent-object calls from the same thread reuse the transaction connection; + injected `#with` objects must do the same. +* A `transaction` inside a transaction that another library opened on the same connection (`BEGIN`) is a savepoint on + PostgreSQL and SQLite. MySQL has no nested transactions: its `BEGIN` commits the open transaction first (see + [Limitations](Limitations.md)). + +## When something goes wrong + +### Errors inside the block + +* **PostgreSQL:** if a statement fails and the block rescues the error, PostgreSQL still rolls the whole transaction + back, so `transaction` raises `PG::InFailedSqlTransaction` instead of pretending it committed. In a nested + `transaction` only the savepoint is rolled back: the outer block can rescue that error and go on. +* **MySQL:** a deadlock (or a lock wait timeout with `innodb_rollback_on_timeout`) rolls the whole transaction back on + the server, its savepoints too. If the block rescues that error, every later query in the transaction and + `transaction` itself raise it again (a `Mysql2::Error` with the same `error_number`, its message prefixed with + `transaction rolled back:`) instead of running in autocommit mode and reporting a commit; retry the whole + `transaction`. A lock wait timeout without that setting fails only its statement: the block can rescue it and go on. +* **SQLite:** threads and fibers sharing one `SQLite3::Database` take turns; a transaction holds the database until it + ends. + +### Interrupts (`Timeout`, `Thread#raise`, `Thread#kill`, a stopped `Async` task) + +* An interrupt that arrives while `BEGIN`, `COMMIT` or `ROLLBACK` runs waits for it to finish, and a `BEGIN` / + `COMMIT` that raises itself (a stopped `Async` task) is rolled back: the connection never goes back to the pool + inside a transaction. +* **PostgreSQL:** a query interrupted by any of them (inside or outside a transaction) is cancelled at once, so the + connection goes back to the pool idle and a killed transaction rolls back without waiting for its query. +* **MySQL:** a `Thread#kill` during a query closes that client (mysql2 cannot send `ROLLBACK` while it waits for the + query's result), so the server rolls the transaction back and the pool replaces the client. +* **Ruby < 3.3 with timeout < 0.4:** `break`, `return` and `throw` also roll back because Timeout cannot be told + apart; add `gem 'timeout', '>= 0.4'` to the generated Gemfile to keep those exits committing. An old `Timeout` that + interrupts a PostgreSQL query in a transaction still raises `Timeout::Error` (not the `PG::InFailedSqlTransaction` + of the cancelled query) after rolling back. + +### Lost connections + +A connection the server closed (restart, failover, killed backend) is reconnected (`pg`) or replaced in the pool +(`mysql2`) the next time it is checked out; the call that hit the dead connection still raises. Inside `transaction` a +lost `pg` connection is not reconnected: every later call in the block and `transaction` itself raise +`PG::ConnectionBad`, so nothing after the loss runs outside the transaction (the server rolled it back). A `mysql2` +client that mysql2 closed because a running query was interrupted (an `Async` task stopped, `Thread#raise`) is +replaced the same way, without failing the next call. + +## Connections + +* `QuerySql.connect(params, **pool_options)` builds a [`connection_pool`](https://github.com/mperham/connection_pool). + To use a pool you already have (or a fake in tests) pass it to `QuerySql.new(pool)`; anything with `#with` works. +* `pg`: `QuerySql.connect` also takes a connection URL: `QuerySql.connect(ENV.fetch('DATABASE_URL'), size: 5)`. +* SQLite: pass a `SQLite3::Database` (opened with any options, `results_as_hash: true` included): `QuerySql.new(db)`. + SQLite transactions belong to the thread that opened them; fibers sharing the same database take turns, and a + transaction holds it until it ends. Only the yielded `tx` runs inside the transaction from other fibers / threads. +* **Shared with another library** (ActiveRecord's `raw_connection`, Sequel, your own code): the connection keeps its + settings. The generated queries pass their own type maps (`pg`; the connection's `type_map_for_results`, + `type_map_for_queries` and `field_name_type` are left as they are) or query options (`mysql2`: rows are read with + `as: :array, cast: true, cast_booleans: false`, whatever the client's or `Mysql2::Client.default_query_options`). + On SQLite `results_as_hash` and the other settings of the `SQLite3::Database` are kept too. +* **Forked processes** (Puma / Sidekiq / Resque workers, `Process.fork`): connections opened before the fork stay the + parent's. connection_pool's fork handling replaces the pooled ones in the child without closing the parent's + sessions, and the child lets go of every inherited connection the generated code has used (also ones another thread + had checked out at fork time, and ones the child never uses), so its exit does not end them either. + * A `pg` connection used directly (not from a `ConnectionPool`) reconnects in the child; a `mysql2` client used that + way raises `Mysql2::Error::ConnectionError` in the child (open clients in the process that uses them). + * SQLite: the sqlite3 gem closes writable databases a child inherits, so open the database again in the child. + * macOS: connecting from a forked child can crash in libpq's GSSAPI probe; set `gssencmode: 'disable'` (or + `PGGSSENCMODE=disable`), as usual for forking `pg` apps there. + +## Async / fibers + +The generated code is plain synchronous Ruby and works with the Fiber scheduler +([`async`](https://github.com/socketry/async)): inside `Sync { ... }`, pooled `pg` and `mysql2` queries run +concurrently (`mysql2` blocks the thread on Windows). MySQL parameters are quoted client-side with +`Mysql2::Client#escape` and sent with `Client#query`, not server-side prepared statements. SQLite queries run one at a +time. diff --git a/docs/Types.md b/docs/Types.md new file mode 100644 index 0000000..fa1223e --- /dev/null +++ b/docs/Types.md @@ -0,0 +1,82 @@ +# Types and names + +## Types + +Types follow what the drivers return. + +| SQL | Ruby | +|---|---| +| integers, floats | `Integer`, `Float` | +| `numeric` / `decimal` | `BigDecimal` | +| text | `String` | +| dates and times | `Date`, `Time` | +| network addresses | `IPAddr` | +| booleans | `true` / `false` | +| nullable columns and parameters | `T?` | + +### PostgreSQL + +* `json` / `jsonb` are parsed (`untyped`). A `String` passed to a `json` / `jsonb` parameter is JSON text + (`'{"a":1}'`): pass `'"hi"'` or `'hi'.to_json` for a JSON string. Hashes, Arrays, numbers, `true` / `false` and `nil` + are encoded for you. +* Arrays are `Array[T]` (`Array[Array[T]]` for `T[][]`); arrays of types the pg gem does not decode (`uuid[]`, enum + arrays, ...) are `Array[String]`, also for types created after the connection was opened (a migration run while the + application is up). +* Geometric, range, full-text, `hstore`, `ltree`, `timetz`, `reg*` and composite types are `String`s; domains and + unknown types are `untyped`. +* A `Time` or `DateTime` parameter of a `timestamp` (without time zone) column (also in `:copyfrom` and batches) keeps + its instant whatever its UTC offset: it is written in the process's local zone, the zone the pg gem reads `timestamp` + values back in (PostgreSQL itself ignores the offset of a `timestamp` value). A `DateTime` of a `timestamptz` column + keeps its instant too (the pg gem alone would send its date only); the RBS still declares `::Time`, the type the + rows return. Elements of `timestamp[]` arrays are written as given. + +### MySQL + +* `JSON` columns and parameters are JSON text `String`s, as `mysql2` returns them. +* A `Time` / `DateTime` parameter (also in `:copyfrom`) keeps its instant whatever its UTC offset: it is written in + the zone `mysql2` reads `DATETIME` / `TIMESTAMP` values back in, the process's local zone, or UTC for a client + created with `database_timezone: :utc` (`connect(params.merge(database_timezone: :utc))`). The server also converts + `TIMESTAMP` columns with the session `time_zone`, so keep that equal to the client's zone. + +### SQLite + +* Follows its [type affinity rules](https://www.sqlite.org/datatype3.html#determination_of_column_affinity) + (`VARCHAR(255)` → `String`, `DECIMAL(10,5)` → `Float`, `UNSIGNED BIG INT` → `Integer`). +* `JSON` and `UUID` columns are `String`s, also when SQLite stored numeric-looking text as a number (`'42'` → `"42"`, + in SQLite's spelling: `'1.50'` → `"1.5"`). +* The generated code uses `Monitor`: if you run Steep, add `library "monitor"` to your `Steepfile`. Code generated for + a `rubyVersion` below 3.3 also uses `Timeout` (see [Transactions](Transactions.md)): add `library "timeout"` too. + +### Time zones + +A column without a time zone (PostgreSQL `timestamp`, MySQL `DATETIME`) stores the local wall-clock time, so an +instant in the hour repeated when the local zone falls back from daylight saving time (e.g. 02:30 on the last Sunday of +October in Europe/Berlin happens twice) reads back as the first of the two, one hour off for the second. Use +`timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), to keep every instant. + +### Enums + +Enums are `String`s, with constants in `models.rb` (`ExtendedBioType::MEMOIR`, `ExtendedBioType::ALL`). Values that are +not valid constant names still get one: `'end'` → `END_`, `'all'` → `ALL_`, `'2x'` → `VALUE_2X`, `'a b'` → `A_B`; values +that end up with the same name get `_2`, `_3`, ... + +## Names + +* Query methods and keyword arguments are snake_cased, with acronyms kept together (`MIN_ID` → `min_id`, + `HTTPServer` → `http_server`); Ruby keywords get a trailing `_` (`end_:`). +* Row members use the column spelling reported by sqlc for the engine, except names that conflict with row methods, + including the private ones `Data` / `Struct` call while building and copying rows (`class_`, `hash_`, `initialize_`): + PostgreSQL and SQLite keep `"camelCase"` as `:camelCase`, while MySQL reports it as `:camelcase`. A column named + `end` is the row member `:end` but the argument `end_:`. `Struct` rows (`rubyVersion` below 3.2) have more methods + than `Data` rows, so more names get the `_`: a column `count` is `:count` from 3.2 but `:count_` below + (`Struct#count` would otherwise be `Enumerable#count`); check the generated `.rbs` when changing `rubyVersion`. +* A result column without a usable name (`SELECT 1, now()`) is named by its position: `column_1`, `column_2`, ... +* A parameter used twice is one keyword when named with `sqlc.arg`; repeated `?` / `$n` of one column become `id:`, + `id_2:`. +* Models may be named like Ruby or driver constants (tables `exceptions`, `data`, `threads`, `sets`, ...): the + generated code and `.rbs` refer to those as `::Exception`, `::Data`, `::Thread`, ... so the models cannot shadow them. +* A table `authors` next to a table `author` keeps its plural model name (`Authors`). Query names that would replace a + generated method (`Transaction`, `Initialize`, ...) are errors. +* SQLite: sqlc before 1.30 (including the pinned 1.27) reports a quoted table name with its quotes, so + `CREATE TABLE "Items"` gives the model `Items` there and `Item` with sqlc 1.30 and later (PostgreSQL / MySQL give + `Item` with any version). Use `rename` or `exactTableNames` to keep the name when upgrading sqlc. From a05646c8ebbe91020e0f29805415f2a51863cc46 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 7 Nov 2026 19:48:28 +0000 Subject: [PATCH 2/5] README: lead with how it works and what is supported; limitations become a symptom-based Troubleshooting guide Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01FskpZBzDwZ1DHnoWBrmEt8 --- README.md | 240 ++++++++++++++++------------------------ docs/Contributing.md | 30 +++++ docs/Design.md | 2 +- docs/Generated-Code.md | 2 +- docs/Limitations.md | 57 ---------- docs/Transactions.md | 2 +- docs/Troubleshooting.md | 54 +++++++++ 7 files changed, 182 insertions(+), 205 deletions(-) create mode 100644 docs/Contributing.md delete mode 100644 docs/Limitations.md create mode 100644 docs/Troubleshooting.md diff --git a/README.md b/README.md index f789264..7078343 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@

sqlc-gen-ruby

- Write SQL. Get typed, idiomatic Ruby.
- A sqlc plugin for PostgreSQL, MySQL and SQLite, with RBS signatures. + Write SQL. Get typed, idiomatic Ruby.
+ A sqlc plugin for PostgreSQL, MySQL and SQLite, with RBS signatures.

@@ -12,17 +12,46 @@ --- -You write the schema and the queries: +## How it works + +**1. You write SQL** ```sql +-- schema.sql +CREATE TABLE authors ( + id BIGSERIAL PRIMARY KEY, + name text NOT NULL, + bio text +); + +-- query.sql -- name: GetAuthor :one -SELECT * FROM authors WHERE id = $1; +-- Returns the author with the given id, or nil when there is none. +SELECT * FROM authors WHERE id = $1 LIMIT 1; -- name: CreateAuthor :one INSERT INTO authors (name, bio) VALUES ($1, $2) RETURNING *; ``` -`sqlc generate` gives you plain Ruby you call like any other object: +**2. `sqlc generate` writes Ruby, and RBS to go with it** + +```ruby +# Returns the author with the given id, or nil when there is none. +def get_author(id:) + ... +end +``` +```rbs +def get_author: (id: ::Integer) -> GetAuthorRow? +def get_author!: (id: ::Integer) -> GetAuthorRow +class GetAuthorRow + attr_reader id: ::Integer + attr_reader name: ::String + attr_reader bio: ::String? +end +``` + +**3. You call it** ```ruby queries = PgCodegen::QuerySql.connect({ dbname: 'app', host: 'localhost' }, size: 5) # a connection pool @@ -36,25 +65,15 @@ queries.transaction do |tx| # COMMIT at the end, end ``` -No ORM, no runtime dependency on the plugin, no magic: the SQL stays yours, the Ruby is generated, committed and -reviewable. - -## Why you might like it - -| | | -|---|---| -| **Three databases** | `pg`, `mysql2` and `sqlite3`, Ruby 3.1 – 4.0. | -| **Typed** | `.rbs` signatures for every model, row and method. The generated code is RuboCop- and Steep-clean. | -| **Idiomatic Ruby** | Keyword arguments, frozen `Data` rows (`Struct` below Ruby 3.2), pattern matching, doc comments from your SQL comments. | -| **Real transactions** | Nested savepoints, isolation levels, safe under `Timeout`, `Thread#kill`, fibers (`async`) and forks (Puma, Sidekiq). | -| **Everything sqlc offers** | `:one` `:many` `:exec` `:execrows` `:execlastid` `:execresult` `:batch*` `:copyfrom`, plus `sqlc.arg` / `narg` / `slice` / `embed`. | -| **Nothing to install but sqlc** | The plugin is pure Ruby shipped as one WebAssembly file ([ruby.wasm](https://github.com/ruby/ruby.wasm)). No Ruby needed to *generate*. | -| **Safe by default** | Parameters are bound (PostgreSQL / SQLite) or escaped by the driver (MySQL), and their values are never logged. | +No ORM and no runtime dependency on the plugin: you get plain Ruby files that are committed and reviewable, in the +style you would have written by hand (keyword arguments, `Data` rows, pattern matching, RuboCop- and Steep-clean). +See the [full generated output](docs/Examples.md) for every driver. -## Quick start +## Install -**1. Point sqlc at the plugin** in `sqlc.yaml`. The URL and sha256 of every version are in its -[release notes](https://github.com/DaredevilOSS/sqlc-gen-ruby/releases). +The plugin is one WebAssembly file ([ruby.wasm](https://github.com/ruby/ruby.wasm) with the plugin inside): the only +thing to install is [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html). Add it to `sqlc.yaml` (the URL and +sha256 of every version are in its [release notes](https://github.com/DaredevilOSS/sqlc-gen-ruby/releases)): ```yaml version: "2" @@ -74,126 +93,83 @@ sql: driver: pg # or mysql2 / sqlite3 ``` -**2. Write your queries** in `db/query.sql` (see above). +Run `sqlc generate`, add `connection_pool` and your driver gem (`pg`, `mysql2` or `sqlite3`) to your Gemfile (or set +`generateGemfile: true` and the plugin writes one), and `require_relative 'app/db/query_sql'`. -**3. Run `sqlc generate`** and `require_relative 'app/db/query_sql'`. +## What's supported -Add the gems the generated code uses (`pg`, `mysql2` or `sqlite3`, and `connection_pool`), or set -`generateGemfile: true` to get a ready `Gemfile`. +Ruby 3.1 – 4.0 · PostgreSQL (`pg`) · MySQL (`mysql2`) · SQLite (`sqlite3`) -> Want to see the output first? [docs/Examples.md](docs/Examples.md) shows the real generated code for every driver, -> built from [these schemas and queries](examples/authors). +### Query annotations -## What you get +| Annotation | Returns | PostgreSQL | MySQL | SQLite | +|---|---|:---:|:---:|:---:| +| `:one` | the row or `nil`, plus a bang finder (`get_author!`) | ✅ | ✅ | ✅ | +| `:many` | an `Array` of rows | ✅ | ✅ | ✅ | +| `:exec` | `nil` | ✅ | ✅ | ✅ | +| `:execrows` | number of affected rows | ✅ | ✅ | ✅ | +| `:execlastid` | the new id | ✅ | ✅ | ✅ | +| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | ✅ | ✅ | ✅ | +| `:batchexec` `:batchone` `:batchmany` | run the query for every element of an `Array`, on one connection | ✅ | ✅ | ✅ | +| `:copyfrom` | bulk insert of an `Array` or any `Enumerable` | ✅ `COPY` | ✅ `LOAD DATA` | ✅ multi-row `INSERT` | -For each query file `foo.sql`, in `out:`: +### sqlc macros -| File | Contents | -|---|---| -| `foo_sql.rb` | `FooSql`: one method per query, plus `transaction` | -| `models.rb` | one class per table, one module of constants per enum | -| `connection.rb` | shared helpers | -| `*.rbs` | type signatures for all of the above | - -How it feels to use (details in [Generated code](docs/Generated-Code.md)): - -* **Arguments** are keywords (`get_author(id:)`); missing or unknown ones raise `ArgumentError`. -* **Rows** are frozen `Data` objects (or keyword `Struct`s below Ruby 3.2) and work with pattern matching. -* **Bang finders**: every `:one` query also has `get_author!`. -* **Pooling**: `connect` builds a [`connection_pool`](https://github.com/mperham/connection_pool); or pass your own to - `QuerySql.new(pool)`. SQLite takes a `SQLite3::Database`. -* **Logging**: `connect(params, logger: Rails.logger)` logs `QuerySql#get_author (0.4ms)`, never the values. +| Macro | PostgreSQL | MySQL | SQLite | +|---|:---:|:---:|:---:| +| `sqlc.arg(name)`: name a parameter | ✅ | ✅ | ✅ | +| `sqlc.narg(name)`: nullable, defaults to `nil` | ✅ | ✅ | ✅ | +| `sqlc.slice(ids)`: `IN (...)` with an `Array` | `= ANY($1::bigint[])` | ✅ | ✅ | +| `sqlc.embed(table)`: the row holds the table's model | ✅ | ✅ | ✅ | -## Query annotations +### Types -| Annotation | Returns | +| SQL | Ruby | |---|---| -| `:one` | the row or `nil` (+ a `!` finder) | -| `:many` | an `Array` of rows | -| `:exec` | `nil` | -| `:execrows` | the number of affected rows | -| `:execlastid` | the new id | -| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | -| `:batchexec` `:batchone` `:batchmany` | run the query once per element of an `Array`, on one connection | -| `:copyfrom` | bulk insert (`COPY` / `LOAD DATA` / multi-row `INSERT`) | - -Per-database notes and conflict-clause rules: [Queries](docs/Queries.md). - -## sqlc macros +| integers, floats | `Integer`, `Float` | +| `numeric`, `decimal` | `BigDecimal` | +| text, `varchar` | `String` | +| enums | `String`, plus constants (`ExtendedBioType::MEMOIR`, `ExtendedBioType::ALL`) | +| `date`, `timestamp`, `timestamptz`, `datetime` | `Date`, `Time` | +| `inet`, `cidr` | `IPAddr` | +| `json`, `jsonb` | parsed Ruby objects on PostgreSQL, JSON text on MySQL | +| arrays (PostgreSQL) | `Array[T]` | +| nullable columns and parameters | `T?` | -| Macro | PostgreSQL | MySQL | SQLite | -|---|:---:|:---:|:---:| -| `sqlc.arg(name)` | ✅ | ✅ | ✅ | -| `sqlc.narg(name)` (nullable, defaults to `nil`) | ✅ | ✅ | ✅ | -| `sqlc.slice(ids)` (`IN (...)` with an `Array`) | use `= ANY($1::bigint[])` | ✅ | ✅ | -| `sqlc.embed(table)` (the row holds the table's model) | ✅ | ✅ | ✅ | +Types follow what each driver returns; override any of them with [`overrides`](docs/Options.md#overrides). +Everything else (ranges, geometric types, composite types, SQLite affinity, time zones): [Types and names](docs/Types.md). -## Transactions +### Built for production -```ruby -queries.transaction(isolation: :serializable) do |tx| - id = tx.create_author_return_id(name: 'a', bio: nil) - tx.transaction { tx.create_book(name: 'b', author_id: id) } # nested: a SAVEPOINT - :done # the block's value is returned -end -``` +| | | +|---|---| +| **Transactions** | `transaction` with commit/rollback, nested savepoints and isolation levels (`isolation:` / `mode:`) | +| **Pooling** | `connection_pool` built for you, or bring your own pool, `SQLite3::Database` or a shared ActiveRecord / Sequel connection | +| **Resilience** | `Timeout`, `Thread#kill`, stopped `Async` tasks, lost connections, deadlocks and forked workers (Puma, Sidekiq) are handled | +| **Concurrency** | plain Ruby that works with the Fiber scheduler (`async`) | +| **Logging** | `connect(params, logger: Rails.logger)` logs `QuerySql#get_author (0.4ms)`; parameter values are never logged | +| **Docs and types** | SQL comments become doc comments; `.rbs` for every model, row and method | -Commit on exit, rollback on exception, savepoints for nesting. `tx` stays in the transaction from any fiber or thread -and raises `TransactionEnded` once it is over. Dead connections, timeouts, killed threads, forks, deadlocks and shared -connections (ActiveRecord, Sequel) are all handled and documented in [Transactions and connections](docs/Transactions.md). +Details: [Transactions and connections](docs/Transactions.md) · [Generated code](docs/Generated-Code.md) · [Queries](docs/Queries.md). ## Options +Set them under `options:` of the `codegen` entry. + | Option | Default | What it does | |---|---|---| | `driver` | required | `pg`, `mysql2` or `sqlite3`; must match the sqlc `engine:` | -| `rubyVersion` | `3.3` | oldest Ruby the code must run on (`3.1` – `4.0`) | +| `rubyVersion` | `3.3` | oldest Ruby the code must run on (`3.1` – `4.0`); `Data` rows from 3.2, `Struct` below | | `generateTypes` | `true` | also generate `.rbs` signatures | -| `generateGemfile` | `false` | also generate a `Gemfile` with the gems the code needs | +| `generateGemfile` | `false` | also generate a `Gemfile` | | `moduleName` | `Codegen` | the module the code lives in, e.g. `MyApp::Db` | | `overrides` | none | change the Ruby type of columns / parameters | | `rename` | none | rename tables and columns, e.g. `{authors: Writer, bio: biography}` | | `exactTableNames` | `false` | keep `authors` as `Authors` instead of singularizing to `Author` | | `preparedStatements` | `true` | `pg` only; `false` for pgbouncer in transaction mode | -Misspelled options are errors with a "did you mean" hint. Full reference, overrides and `sqlc.yaml`-level sharing: -[Options](docs/Options.md). - -## Types - -| SQL | Ruby | -|---|---| -| integers / floats | `Integer` / `Float` | -| `numeric`, `decimal` | `BigDecimal` | -| text, enums | `String` (enums also get constants: `ExtendedBioType::MEMOIR`) | -| dates / times | `Date` / `Time` | -| `inet`, `cidr` | `IPAddr` | -| `json`, `jsonb` (PostgreSQL) | parsed (`untyped`) | -| PostgreSQL arrays | `Array[T]` | -| nullable | `T?` | - -Types follow what each driver returns. Time-zone behavior, the exotic PostgreSQL types, MySQL `JSON`, SQLite affinity -and naming rules (keywords, acronyms, clashing columns) are in [Types and names](docs/Types.md). - -## Good to know - -The plugin is deliberately direct, so a few things work differently from what you might expect. The ones people hit -first: - -| Database | Heads-up | Do this | -|---|---|---| -| all | Two result columns with the same name (`a.id, b.id`) are a generation error | alias one, or use `sqlc.embed` | -| all | `:batch*` calls are not atomic | wrap them in `transaction` | -| all | `timestamp` / `DATETIME` store local wall-clock time | use `timestamptz` / `TIMESTAMP`, or run with `TZ=UTC` | -| PostgreSQL | no `sqlc.slice` | `= ANY($1::bigint[])` | -| PostgreSQL | prepared statements break behind pgbouncer (transaction mode) | `preparedStatements: false` | -| MySQL | no server-side prepared statements (client-side quoting); `:copyfrom` needs `local_infile` | enable it on client and server | -| MySQL | `sqlc.embed` lowercases table names | alias the table: `` sqlc.embed(m) FROM `MixedItems` AS m `` | -| SQLite | sqlc breaks queries in files with non-ASCII characters | keep query files ASCII, pass values as parameters | -| SQLite | `ON CONFLICT ... DO UPDATE` with `sqlc.arg` is a generation error | use `excluded.` | -| SQLite | at most 32,766 parameters per statement | `:copyfrom` into a table and join | - -Everything else, with the reasons: [Limitations](docs/Limitations.md). +Misspelled options are errors with a "did you mean" hint. Everything about them, including sharing options between +`sql:` blocks: [Options](docs/Options.md). ## Documentation @@ -204,36 +180,10 @@ Everything else, with the reasons: [Limitations](docs/Limitations.md). | [Transactions and connections](docs/Transactions.md) | transactions, pools, forks, Async, shared connections | | [Types and names](docs/Types.md) | type mapping, time zones, enums, naming rules | | [Options](docs/Options.md) | every option and `overrides` | -| [Limitations](docs/Limitations.md) | the complete list, per database | +| [Troubleshooting](docs/Troubleshooting.md) | "I got this error / this looks odd": causes and fixes, per database | | [Examples](docs/Examples.md) | per-driver configs and generated output | ## Contributing -Contributions are welcome. You need Ruby 3.3 (`bundle install`), [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html) -(version in [`scripts/tool-versions.env`](scripts/tool-versions.env)), `curl` + `tar` to build the WASM, and -PostgreSQL / MySQL servers (or Docker) for the end-to-end tests. - -``` -bundle exec rake test # unit + golden tests (no database needed) -bundle exec rake rubocop steep # the generated code must stay RuboCop- and Steep-clean -rake sqlc_generate_process # regenerate examples/ with exe/plugin.rb -rake sqlc_generate_wasm # build dist/plugin.wasm, then regenerate examples/ with it -rake end2end_sqlite3 # end-to-end, no server needed -rake end2end_pg end2end_mysql2 # end-to-end, needs servers (see docs/Testing.md) -rake run_end2end_tests # all of it in Docker (docker-compose) -``` - -`examples/` is the plugin's committed output and CI fails when it drifts, so regenerate it with every change to the -generated code. Pull requests need both CI workflows (`build.yml`, `tests.yml`) green. - -Releases are started by a maintainer (**Actions → Release → Run workflow**) and are automated from there: the full test -gate, the tag, the GitHub release with the tested `sqlc-gen-ruby.wasm` and its sha256, and a check of the published file. - -| Doc | What | -|---|---| -| [docs/Architecture.md](docs/Architecture.md) | how the generator works, where to change what, adding a feature / driver / option | -| [docs/Testing.md](docs/Testing.md) | the test layers, CI workflows, what to run before pushing | -| [docs/Releasing.md](docs/Releasing.md) | the release process and versioning rules | -| [docs/Design.md](docs/Design.md) | the WASM packaging and design decisions | -| [docs/Coverage.md](docs/Coverage.md) | every documented behavior mapped to its end-to-end test | -| [CLAUDE.md](CLAUDE.md) | the same orientation condensed for AI coding assistants | +Contributions are welcome: see [Contributing](docs/Contributing.md) for the setup, the test commands and the release +process, and [Architecture](docs/Architecture.md) for how the generator works. diff --git a/docs/Contributing.md b/docs/Contributing.md new file mode 100644 index 0000000..97c8b99 --- /dev/null +++ b/docs/Contributing.md @@ -0,0 +1,30 @@ +# Contributing + +You need Ruby 3.3 (`bundle install`), [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html) (version in +[`scripts/tool-versions.env`](../scripts/tool-versions.env)), `curl` + `tar` to build the WASM, and PostgreSQL / MySQL +servers (or Docker) for the end-to-end tests. + +``` +bundle exec rake test # unit + golden tests (no database needed) +bundle exec rake rubocop steep # the generated code must stay RuboCop- and Steep-clean +rake sqlc_generate_process # regenerate examples/ with exe/plugin.rb +rake sqlc_generate_wasm # build dist/plugin.wasm, then regenerate examples/ with it +rake end2end_sqlite3 # end-to-end, no server needed +rake end2end_pg end2end_mysql2 # end-to-end, needs servers (see Testing.md) +rake run_end2end_tests # all of it in Docker (docker-compose) +``` + +`examples/` is the plugin's committed output and CI fails when it drifts, so regenerate it with every change to the +generated code. Pull requests need both CI workflows (`build.yml`, `tests.yml`) green. + +Releases are started by a maintainer (**Actions → Release → Run workflow**) and are automated from there: the full test +gate, the tag, the GitHub release with the tested `sqlc-gen-ruby.wasm` and its sha256, and a check of the published file. + +| Doc | What | +|---|---| +| [Architecture.md](Architecture.md) | how the generator works, where to change what, adding a feature / driver / option | +| [Testing.md](Testing.md) | the test layers, CI workflows, what to run before pushing | +| [Releasing.md](Releasing.md) | the release process and versioning rules | +| [Design.md](Design.md) | the WASM packaging and design decisions | +| [Coverage.md](Coverage.md) | every documented behavior mapped to its end-to-end test | +| [../CLAUDE.md](../CLAUDE.md) | the same orientation condensed for AI coding assistants | diff --git a/docs/Design.md b/docs/Design.md index f66f4a0..381a102 100644 --- a/docs/Design.md +++ b/docs/Design.md @@ -45,7 +45,7 @@ transactions, nullability, `overrides`) and Ruby conventions (bang finders, logg now idiomatic Ruby rather than a copy of the C# one. ## Known limitations -See [Limitations](Limitations.md). Design choices that look like limitations: +See [Limitations](Troubleshooting.md). Design choices that look like limitations: * Overrides only change the RBS types; values are never converted, so the generated code stays a thin layer over the driver. * SQLite queries do not overlap: a `SQLite3::Database` is one connection, so the generated code takes one lock per database (held for a whole transaction) to keep threads and fibers from mixing their transactions. diff --git a/docs/Generated-Code.md b/docs/Generated-Code.md index 4a2af5d..6ce5841 100644 --- a/docs/Generated-Code.md +++ b/docs/Generated-Code.md @@ -37,7 +37,7 @@ each query and transaction. Parameter values are never logged. * Comments above a query become its doc comment; table / column / enum comments of the schema become doc comments of the models. * The code is RuboCop-clean. Where a name from your schema breaks a RuboCop naming rule (`"UserId"`, `naïve`), that - class or method gets a `rubocop:disable` (on SQLite keep the query files ASCII, see [Limitations](Limitations.md)). + class or method gets a `rubocop:disable` (on SQLite keep the query files ASCII, see [Troubleshooting](Troubleshooting.md)). Query names such as `HasAuthors` / `IsActive` keep their name (`has_authors`) with `Naming/PredicatePrefix` disabled. ## More diff --git a/docs/Limitations.md b/docs/Limitations.md deleted file mode 100644 index a5f25a5..0000000 --- a/docs/Limitations.md +++ /dev/null @@ -1,57 +0,0 @@ -# Limitations - -What to know before you adopt the plugin. Most of these come from sqlc or the databases, not from the plugin. - -## All databases - -* Two result columns with the same name (e.g. `a.id, b.id`) are an error: alias one (`b.id AS book_id`) or use - `sqlc.embed`. -* `:batchexec` / `:batchone` / `:batchmany` are not atomic on their own: wrap them in `transaction` for all-or-nothing. -* Query names that would replace a generated method (`Transaction`, `Initialize`, ...) are errors. -* Large `sqlc.slice` arrays hit database limits (see SQLite and MySQL below). Load larger sets into a table - (`:copyfrom`) and join it instead. -* Timestamps without a time zone store the local wall-clock time (see [Types](Types.md#time-zones)). - -## PostgreSQL - -* `sqlc.slice` is not offered by sqlc for PostgreSQL: use `= ANY($1::bigint[])` with an array. -* `:copyfrom` cannot express `ON CONFLICT` / `RETURNING`: those are generation errors, use `:batchexec`. -* `:execresult`'s `last_insert_id` is `nil`: use `RETURNING`. -* Behind pgbouncer in transaction mode set `preparedStatements: false` ([Options](Options.md#preparedstatements-pg)). - -## MySQL - -* `transaction` on a client inside a transaction that was started with raw SQL or by another library commits that - transaction first (MySQL's `BEGIN` does), so a later `ROLLBACK` no longer undoes it. Run the outer work in - `transaction` too (inner calls become savepoints), or finish the other transaction first. -* sqlc lowercases the table name that `sqlc.embed` expands to (`sqlc.embed(MixedItems)` becomes - `SELECT mixeditems.itemid, ...`). MySQL table names are case-sensitive on Linux, so a mixed-case table fails with - `Unknown column 'mixeditems.itemid'`. This is a sqlc limitation the plugin cannot undo (the request only carries the - lowercased name); give the table an alias and embed that: `` SELECT sqlc.embed(m) FROM `MixedItems` AS m ``. -* A statement is limited by the server's `max_allowed_packet` (large `sqlc.slice` arrays). -* `:copyfrom` needs `local_infile` on client and server and commits an open raw-SQL transaction - (see [Queries](Queries.md#copyfrom)). -* No server-side prepared statements: parameters are quoted client-side. - -## SQLite - -* A statement takes at most 32,766 parameters (`SQLITE_MAX_VARIABLE_NUMBER`), so a `sqlc.slice` with more elements - raises `SQLite3::SQLException: too many SQL variables`. -* `NUMERIC` affinity can return an `Integer` where `Float` is declared. -* sqlc does not see parameters in `ON CONFLICT ... DO UPDATE SET` (it leaves `sqlc.arg(...)` in the SQL or reports - fewer parameters than there are `?`), so such queries are generation errors. Use `excluded.` - (`DO UPDATE SET n = counters.n + excluded.n`) or move the value into `VALUES`. -* sqlc (1.27 – 1.31 at least) misplaces the SQL of a query file that contains a non-ASCII character (in a name, - literal or comment): that query and the following ones lose characters at the end (`FROM user` instead of - `FROM users`). This is a sqlc bug, not a plugin one; keep SQLite query files ASCII and pass such values as - parameters. -* `busy_timeout=` makes the sqlite3 gem wait for a locked database while holding Ruby's GVL, so other threads and - fibers stop too; `busy_handler_timeout=` waits without blocking them. -* Transactions belong to the thread that opened them; fibers sharing the same database take turns, and a transaction - holds it until it ends. Only the yielded `tx` runs inside the transaction from other fibers / threads. -* sqlc before 1.30 reports quoted table names with their quotes (see [Names](Types.md#names)). - -## Not applicable to Ruby - -They exist in the C# plugin: `withAsyncSuffix` / `withCancellationToken` (use the Fiber scheduler), `useDapper`, -target framework / csproj options (`moduleName` replaces the namespace). diff --git a/docs/Transactions.md b/docs/Transactions.md index f83fddf..a849e11 100644 --- a/docs/Transactions.md +++ b/docs/Transactions.md @@ -24,7 +24,7 @@ end injected `#with` objects must do the same. * A `transaction` inside a transaction that another library opened on the same connection (`BEGIN`) is a savepoint on PostgreSQL and SQLite. MySQL has no nested transactions: its `BEGIN` commits the open transaction first (see - [Limitations](Limitations.md)). + [Limitations](Troubleshooting.md)). ## When something goes wrong diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md new file mode 100644 index 0000000..6122467 --- /dev/null +++ b/docs/Troubleshooting.md @@ -0,0 +1,54 @@ +# Troubleshooting + +Something failed or looks odd? Find it here. Most of these come from sqlc or the databases, not from the plugin. + +## Any database + +| You see | Why | Do this | +|---|---|---| +| Generation error: two result columns with the same name (`a.id, b.id`) | a row has one member per name | alias one (`b.id AS book_id`) or use `sqlc.embed` | +| Generation error: a query named `Transaction`, `Initialize`, ... | it would replace a generated method | rename the query | +| Half the rows of a batch were written before an error | `:batchexec` / `:batchone` / `:batchmany` are not atomic on their own | wrap them in `transaction` | +| A timestamp comes back one hour off around a daylight-saving change | `timestamp` / `DATETIME` store the local wall-clock time, so the repeated hour is ambiguous | use `timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), see [Types](Types.md#time-zones) | +| A huge `sqlc.slice` fails | database limits (SQLite and MySQL below) | `:copyfrom` the values into a table and join it | + +## PostgreSQL + +| You see | Why | Do this | +|---|---|---| +| `sqlc.slice` is not available | sqlc does not offer it for PostgreSQL | `WHERE id = ANY($1::bigint[])` with an array | +| Prepared statement errors behind pgbouncer in transaction mode | statements are prepared per connection | `preparedStatements: false` ([Options](Options.md#preparedstatements-pg)) | +| Generation error in a `:copyfrom` query with `ON CONFLICT` / `RETURNING` | `COPY` cannot express them | use `:batchexec` | +| `last_insert_id` of `:execresult` is `nil` | PostgreSQL has no last-insert id | use `RETURNING` (`:one` or `:execlastid`) | + +## MySQL + +| You see | Why | Do this | +|---|---|---| +| A rollback did not undo work done before `transaction` | `transaction` on a client inside a transaction started with raw SQL or by another library commits it first (MySQL's `BEGIN` does) | run the outer work in `transaction` too (inner calls become savepoints), or finish the other transaction first | +| `Unknown column 'mixeditems.itemid'` with `sqlc.embed(MixedItems)` | sqlc lowercases the table name it expands to, and MySQL table names are case-sensitive on Linux; the plugin only receives the lowercased name | give the table an alias and embed that: `` SELECT sqlc.embed(m) FROM `MixedItems` AS m `` | +| `:copyfrom` is refused / fails | `LOAD DATA LOCAL INFILE` needs `local_infile` | `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the server, see [Queries](Queries.md#copyfrom) | +| A `sqlc.slice` fails with a packet error | the statement exceeds the server's `max_allowed_packet` | `:copyfrom` into a table and join it | +| `:execrows` counts changed rows, not matched rows | MySQL's default | connect with `flags: Mysql2::Client::FOUND_ROWS` | +| `:copyfrom` committed my raw-SQL transaction | `LOAD DATA` commits an open `BEGIN` transaction | call `:copyfrom` inside `transaction` | + +MySQL does not use server-side prepared statements: parameters are quoted client-side with `Mysql2::Client#escape` +and sent with `Client#query`. + +## SQLite + +| You see | Why | Do this | +|---|---|---| +| Queries lose their last characters (`FROM user` instead of `FROM users`) | sqlc (1.27 – 1.31 at least) misplaces the SQL of a query file that contains a non-ASCII character in a name, literal or comment. A sqlc bug, not a plugin one | keep SQLite query files ASCII and pass such values as parameters | +| Generation error with `sqlc.arg(...)` in `ON CONFLICT ... DO UPDATE SET` | sqlc does not see parameters there (it leaves `sqlc.arg(...)` in the SQL or reports fewer parameters than there are `?`) | use `excluded.` (`DO UPDATE SET n = counters.n + excluded.n`) or move the value into `VALUES` | +| `SQLite3::SQLException: too many SQL variables` | a statement takes at most 32,766 parameters (`SQLITE_MAX_VARIABLE_NUMBER`) | `:copyfrom` into a table and join it | +| An `Integer` where `Float` is declared | `NUMERIC` affinity | cast in the query or use `overrides` | +| Other threads freeze while the database is locked | `busy_timeout=` waits while holding Ruby's GVL | use `busy_handler_timeout=`, which waits without blocking them | +| A fiber or thread waits for a transaction | SQLite transactions belong to the thread that opened them; fibers sharing a database take turns, and a transaction holds it until it ends | use the yielded `tx` inside the transaction | +| The model is named `Items` instead of `Item` | sqlc before 1.30 (including the pinned 1.27) reports a quoted table name with its quotes | `rename` or `exactTableNames`, see [Names](Types.md#names) | +| Steep complains about `Monitor` / `Timeout` | the generated code uses them | add `library "monitor"` (and `library "timeout"` for `rubyVersion` below 3.3) to your `Steepfile` | + +## Options from the C# plugin + +`withAsyncSuffix` / `withCancellationToken` (use the Fiber scheduler), `useDapper` and the target framework / csproj +options do not apply to Ruby; `moduleName` replaces the namespace. From abc3ca8ed5cc0801eeff9e05cc34c2e379888503 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 7 Nov 2026 19:48:34 +0000 Subject: [PATCH 3/5] Fix link labels Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01FskpZBzDwZ1DHnoWBrmEt8 --- docs/Design.md | 2 +- docs/Transactions.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Design.md b/docs/Design.md index 381a102..4767ed4 100644 --- a/docs/Design.md +++ b/docs/Design.md @@ -45,7 +45,7 @@ transactions, nullability, `overrides`) and Ruby conventions (bang finders, logg now idiomatic Ruby rather than a copy of the C# one. ## Known limitations -See [Limitations](Troubleshooting.md). Design choices that look like limitations: +See [Troubleshooting](Troubleshooting.md). Design choices that look like limitations: * Overrides only change the RBS types; values are never converted, so the generated code stays a thin layer over the driver. * SQLite queries do not overlap: a `SQLite3::Database` is one connection, so the generated code takes one lock per database (held for a whole transaction) to keep threads and fibers from mixing their transactions. diff --git a/docs/Transactions.md b/docs/Transactions.md index a849e11..7c72e04 100644 --- a/docs/Transactions.md +++ b/docs/Transactions.md @@ -24,7 +24,7 @@ end injected `#with` objects must do the same. * A `transaction` inside a transaction that another library opened on the same connection (`BEGIN`) is a savepoint on PostgreSQL and SQLite. MySQL has no nested transactions: its `BEGIN` commits the open transaction first (see - [Limitations](Troubleshooting.md)). + [Troubleshooting](Troubleshooting.md)). ## When something goes wrong From f853ae24a1dfca66af217a6599fc3d12efd4c593 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 7 Nov 2026 20:14:34 +0000 Subject: [PATCH 4/5] Docs: fix Coverage table split, refresh stale Architecture snippet, wording after the README split Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01FskpZBzDwZ1DHnoWBrmEt8 --- CLAUDE.md | 2 +- docs/Architecture.md | 12 ++++++++---- docs/Coverage.md | 7 +++---- docs/Design.md | 4 ++-- docs/Testing.md | 2 +- 5 files changed, 15 insertions(+), 12 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 891fa12..fd06c03 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,7 +33,7 @@ UPDATE_GOLDEN=1 bundle exec ruby -Itest test/extended_golden_test.rb # rewrite the C# sibling plugin's style. Parameter *values* are never logged. * PostgreSQL prepared statements are per connection (`ConnectionState`) because the query object is shared by pooled connections. MySQL uses client-side quoting and `Mysql2::Client#query` instead of server-side prepared statements. -* Every README claim or supported feature needs a real-database E2E test for each applicable engine, listed in +* Every documented behavior (README or `docs/`) or supported feature needs a real-database E2E test for each applicable engine, listed in `docs/Coverage.md`. A bug fix starts with an E2E test that fails before the fix; use unit tests only where E2E cannot reach the behavior (for example option validation or generation errors). * E2E coverage runs on every supported Ruby and includes full-suite option variants. Update `docs/Coverage.md` and diff --git a/docs/Architecture.md b/docs/Architecture.md index 5b1bc4a..a491ade 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -42,16 +42,20 @@ module PgCodegen GET_AUTHOR_SQL = <<~'SQL' # raw heredoc: the SQL is never interpolated SELECT id, name, bio FROM authors WHERE id = $1 LIMIT 1 SQL - GetAuthorRow = Data.define(:id, :name, :bio) # Struct.new(...) below Ruby 3.2 + class GetAuthorRow < ::Data.define(:id, :name, :bio) # a keyword Struct below Ruby 3.2 + end def get_author(id:) # a :one query returns the row or nil ... with_client('get_author') do |client| # one pooled connection + logging - ...prepare once per connection, exec_prepared... + query_params = [id] + result = run_prepared(client, 'PgCodegen::QuerySql#get_author', GET_AUTHOR_SQL, query_params) + row = result.first + row && GetAuthorRow.new(row['id'], row['name'], row['bio']) end end - def get_author!(id:) # ... and gets a bang variant - get_author(id: id) || raise(NotFound, 'GetAuthor returned no row') + def get_author!(...) # ... and gets a bang variant + get_author(...) || raise(NotFound, 'GetAuthor returned no row') end def transaction(isolation: nil) { |tx| ... } # commit / rollback, nested = SAVEPOINT diff --git a/docs/Coverage.md b/docs/Coverage.md index 397ebb7..e59793c 100644 --- a/docs/Coverage.md +++ b/docs/Coverage.md @@ -1,9 +1,10 @@ # Supported feature coverage -Every README behavior is linked to the real-database test that exercises it for each applicable engine. `unit:` is used +Every behavior documented in the README and `docs/` is linked to the real-database test that exercises it for each applicable engine. `unit:` is used only when the behavior is rejected or decided during generation and generated code cannot reach it. +CI runs `test/coverage_map_test.rb`, which checks every linked test method exists in its referenced `tests/` or `test/` file. -| README claim / feature | PostgreSQL | MySQL | SQLite | Unit-only checks | +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | |---|---|---|---|---| | Options: driver, Ruby target, types, Gemfile, module name, renames, exact names, overrides and global/local merge; CamelCase query file classes (`admin_users.sql` → `AdminUsersSql`); distinct prepared statements for modules sharing a pool (also names over 63 bytes) | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | unit: MySQL-specific `preparedStatements` is rejected | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | [`test_unknown_options_are_rejected_with_a_hint`](../test/options_test.rb#test_unknown_options_are_rejected_with_a_hint), [`test_override_forms`](../test/options_test.rb#test_override_forms), [`test_immutable_data_support`](../test/options_test.rb#test_immutable_data_support), [`test_omitted_parts_mean_the_oldest_covered_release`](../test/options_test.rb#test_omitted_parts_mean_the_oldest_covered_release), [`test_version_decisions_agree_for_every_form`](../test/options_test.rb#test_version_decisions_agree_for_every_form), [`test_major_only_version_orders_as_its_oldest_release`](../test/options_test.rb#test_major_only_version_orders_as_its_oldest_release), [`test_legacy_timeout_guard_is_emitted_only_for_pre_33_targets`](../test/features_test.rb#test_legacy_timeout_guard_is_emitted_only_for_pre_33_targets), [`test_query_file_names_become_camel_case_classes`](../test/features_test.rb#test_query_file_names_become_camel_case_classes), [`test_statement_names_of_different_classes_never_collide`](../test/features_test.rb#test_statement_names_of_different_classes_never_collide) | | `preparedStatements` disabled and configured `moduleName` | [`test_flow`](../tests/end2end_pg.rb#test_flow) (Ruby 4.0 variant) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow) (Ruby 4.0 variant) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow) (Ruby 4.0 variant) | [`test_pg_can_skip_prepared_statements`](../test/features_test.rb#test_pg_can_skip_prepared_statements) | @@ -44,8 +45,6 @@ only when the behavior is rejected or decided during generation and generated co | Primary + read-replica query objects; a write to the read-only replica raises and leaves it usable | [`test_primary_and_read_replica_query_objects`](../tests/end2end_pg.rb#test_primary_and_read_replica_query_objects) | [`test_primary_and_read_replica_query_objects`](../tests/end2end_mysql2.rb#test_primary_and_read_replica_query_objects) | [`test_primary_and_read_replica_query_objects`](../tests/end2end_sqlite3.rb#test_primary_and_read_replica_query_objects) | | | Memoized frozen query objects, subclasses and prepended wrappers forwarding `...` / `**`, `SimpleDelegator`, shared across threads | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_pg.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_mysql2.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_sqlite3.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | | | Rows through `Marshal` (Rails.cache) and JSON (`to_h.to_json`) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_pg.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_mysql2.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_sqlite3.rb#test_rows_round_trip_through_marshal_and_json) | | - -CI runs `test/coverage_map_test.rb`, which checks every linked test method exists in its referenced `tests/` or `test/` file. | Models named like Ruby / driver constants (`Exception`, `Data`, `Thread`, `Set`, `Monitor`, ...) do not shadow them | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_pg.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_mysql2.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_sqlite3.rb#test_models_named_like_core_constants_do_not_shadow_them) | | | Transaction `tx` used from an `Enumerator`, `Fiber`, `Thread`, `Async` task and nested savepoint; `TransactionEnded` after the transaction | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_pg.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_mysql2.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_sqlite3.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | | | PostgreSQL arrays of enums created after the connection was opened | [`test_enum_arrays_of_a_type_created_after_connecting_are_decoded`](../tests/end2end_pg.rb#test_enum_arrays_of_a_type_created_after_connecting_are_decoded) | not MySQL | not SQLite | | diff --git a/docs/Design.md b/docs/Design.md index 4767ed4..9b0d36e 100644 --- a/docs/Design.md +++ b/docs/Design.md @@ -44,8 +44,8 @@ that plugin (`:execrows`, `:execlastid` for every driver, `:copyfrom`, `sqlc.sli transactions, nullability, `overrides`) and Ruby conventions (bang finders, logger, pooling), so the output is now idiomatic Ruby rather than a copy of the C# one. -## Known limitations -See [Troubleshooting](Troubleshooting.md). Design choices that look like limitations: +## Deliberate trade-offs +Things that look like limitations but are design choices (symptoms and fixes: [Troubleshooting](Troubleshooting.md)): * Overrides only change the RBS types; values are never converted, so the generated code stays a thin layer over the driver. * SQLite queries do not overlap: a `SQLite3::Database` is one connection, so the generated code takes one lock per database (held for a whole transaction) to keep threads and fibers from mixing their transactions. diff --git a/docs/Testing.md b/docs/Testing.md index 8ce6c1f..ef6ae8d 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -1,6 +1,6 @@ # Testing and CI -Every README claim and supported feature needs a real-database E2E test for each applicable engine, listed in +Every documented behavior (README and `docs/`) and supported feature needs a real-database E2E test for each applicable engine, listed in [`Coverage.md`](Coverage.md). A bug fix starts with a real-database E2E regression that fails before the fix. Use unit tests only where E2E cannot reach the behavior, such as option validation or generation errors. CI runs the full E2E suites on every supported Ruby version plus full-suite option variants; generator-options E2E runs on Ruby 3.1 and 4.0. From 175f35534901dd12a481896f94674e840e45b4c5 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 16:54:20 +0000 Subject: [PATCH 5/5] Restructure the docs: accurate per-database types, split Connections, merge Design into Architecture - README: short overview; the type table is now per database (SQLite dates and booleans, MySQL booleans and JSON were shown with PostgreSQL's types) - Generated-Code: files, query class, rows, naming rules (moved from Types), logging, RuboCop / Steep notes - Connections: split out of Transactions (pools, shared connections, forks, fibers, lost connections, prepared statements) - Transactions: basics, isolation, tx, failures; the generated Gemfile already pins timeout >= 0.4 for rubyVersion below 3.3 - Architecture absorbs Design (WASM packaging, design decisions); the snippet matches examples/pg (rows are module-level classes) - Testing: tables instead of run-on paragraphs - Coverage: rows grouped by topic; fixed a row with missing columns and mislabelled 'not applicable' cells - Examples: intro and clearer links (generator script updated) Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01FskpZEyJE4jgCmz45fQCXg --- CLAUDE.md | 2 +- README.md | 158 ++++++++++++++++------------- docs/Architecture.md | 147 ++++++++++++++++++--------- docs/Connections.md | 89 +++++++++++++++++ docs/Contributing.md | 46 ++++++--- docs/Coverage.md | 119 +++++++++++++++------- docs/Design.md | 51 ---------- docs/Examples.md | 26 +++-- docs/Generated-Code.md | 115 ++++++++++++++++----- docs/Options.md | 25 +++-- docs/Queries.md | 100 +++++++++++++------ docs/Testing.md | 138 ++++++++++++++++--------- docs/Transactions.md | 161 +++++++++++++++--------------- docs/Troubleshooting.md | 27 +++-- docs/Types.md | 151 +++++++++++++++------------- scripts/docs/generate_examples.rb | 23 ++++- 16 files changed, 869 insertions(+), 509 deletions(-) create mode 100644 docs/Connections.md delete mode 100644 docs/Design.md diff --git a/CLAUDE.md b/CLAUDE.md index fd06c03..d8480ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ sqlc-gen-ruby: a [sqlc](https://sqlc.dev) plugin, written in pure Ruby, that generates Ruby data-access code (+ RBS) for `pg`, `mysql2` and `sqlite3`. It ships as one WASM module (the ruby.wasm interpreter with the plugin embedded). -Docs: [Architecture](docs/Architecture.md) · [Testing](docs/Testing.md) · [Releasing](docs/Releasing.md) · [Design](docs/Design.md). +Docs: [Architecture](docs/Architecture.md) · [Testing](docs/Testing.md) · [Coverage](docs/Coverage.md) · [Releasing](docs/Releasing.md). ## Commands ``` diff --git a/README.md b/README.md index 7078343..901eb9d 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,10 @@ --- +[sqlc](https://sqlc.dev) reads your schema and your queries and checks them against each other. This plugin turns +the result into plain Ruby files: one method per query, one class per result row, and `.rbs` signatures for all of it. +There is no ORM and no runtime library: the generated code only needs your database gem and `connection_pool`. + ## How it works **1. You write SQL** @@ -33,7 +37,7 @@ SELECT * FROM authors WHERE id = $1 LIMIT 1; INSERT INTO authors (name, bio) VALUES ($1, $2) RETURNING *; ``` -**2. `sqlc generate` writes Ruby, and RBS to go with it** +**2. `sqlc generate` writes Ruby, and RBS to go with it** (abridged) ```ruby # Returns the author with the given id, or nil when there is none. @@ -42,21 +46,24 @@ def get_author(id:) end ``` ```rbs -def get_author: (id: ::Integer) -> GetAuthorRow? -def get_author!: (id: ::Integer) -> GetAuthorRow class GetAuthorRow attr_reader id: ::Integer attr_reader name: ::String attr_reader bio: ::String? end + +def get_author: (id: ::Integer) -> GetAuthorRow? +def get_author!: (id: ::Integer) -> GetAuthorRow ``` **3. You call it** ```ruby +require_relative 'app/db/query_sql' + queries = PgCodegen::QuerySql.connect({ dbname: 'app', host: 'localhost' }, size: 5) # a connection pool -author = queries.create_author(name: 'Seuss', bio: nil) # => # +author = queries.create_author(name: 'Seuss', bio: nil) # => # queries.get_author(id: author.id) # the row, or nil queries.get_author!(id: 42) # raises QuerySql::NotFound @@ -65,15 +72,13 @@ queries.transaction do |tx| # COMMIT at the end, end ``` -No ORM and no runtime dependency on the plugin: you get plain Ruby files that are committed and reviewable, in the -style you would have written by hand (keyword arguments, `Data` rows, pattern matching, RuboCop- and Steep-clean). -See the [full generated output](docs/Examples.md) for every driver. +The complete, committed output for every database is in [`examples/`](docs/Examples.md). ## Install -The plugin is one WebAssembly file ([ruby.wasm](https://github.com/ruby/ruby.wasm) with the plugin inside): the only -thing to install is [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html). Add it to `sqlc.yaml` (the URL and -sha256 of every version are in its [release notes](https://github.com/DaredevilOSS/sqlc-gen-ruby/releases)): +The plugin is one WebAssembly file, so the only thing to install is +[sqlc](https://docs.sqlc.dev/en/latest/overview/install.html). Add the plugin to `sqlc.yaml`; the URL and sha256 of +every version are in its [release notes](https://github.com/DaredevilOSS/sqlc-gen-ruby/releases): ```yaml version: "2" @@ -93,97 +98,108 @@ sql: driver: pg # or mysql2 / sqlite3 ``` -Run `sqlc generate`, add `connection_pool` and your driver gem (`pg`, `mysql2` or `sqlite3`) to your Gemfile (or set -`generateGemfile: true` and the plugin writes one), and `require_relative 'app/db/query_sql'`. +Then: -## What's supported +1. run `sqlc generate`; +2. add `connection_pool` and your driver gem (`pg`, `mysql2` or `sqlite3`) to your Gemfile, or set + `generateGemfile: true` and the plugin writes one; +3. `require_relative 'app/db/query_sql'`. -Ruby 3.1 – 4.0 · PostgreSQL (`pg`) · MySQL (`mysql2`) · SQLite (`sqlite3`) +## What's supported -### Query annotations +Ruby 3.1 – 4.0, with PostgreSQL (`pg`), MySQL (`mysql2`) and SQLite (`sqlite3`). -| Annotation | Returns | PostgreSQL | MySQL | SQLite | -|---|---|:---:|:---:|:---:| -| `:one` | the row or `nil`, plus a bang finder (`get_author!`) | ✅ | ✅ | ✅ | -| `:many` | an `Array` of rows | ✅ | ✅ | ✅ | -| `:exec` | `nil` | ✅ | ✅ | ✅ | -| `:execrows` | number of affected rows | ✅ | ✅ | ✅ | -| `:execlastid` | the new id | ✅ | ✅ | ✅ | -| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | ✅ | ✅ | ✅ | -| `:batchexec` `:batchone` `:batchmany` | run the query for every element of an `Array`, on one connection | ✅ | ✅ | ✅ | -| `:copyfrom` | bulk insert of an `Array` or any `Enumerable` | ✅ `COPY` | ✅ `LOAD DATA` | ✅ multi-row `INSERT` | +### Queries -### sqlc macros +Every sqlc annotation works on all three databases. Details: [Queries](docs/Queries.md). -| Macro | PostgreSQL | MySQL | SQLite | -|---|:---:|:---:|:---:| -| `sqlc.arg(name)`: name a parameter | ✅ | ✅ | ✅ | -| `sqlc.narg(name)`: nullable, defaults to `nil` | ✅ | ✅ | ✅ | -| `sqlc.slice(ids)`: `IN (...)` with an `Array` | `= ANY($1::bigint[])` | ✅ | ✅ | -| `sqlc.embed(table)`: the row holds the table's model | ✅ | ✅ | ✅ | +| Annotation | The method returns | +|---|---| +| `:one` | the row or `nil`; a second method, `get_author!`, raises instead of returning `nil` | +| `:many` | an `Array` of rows | +| `:exec` | `nil` | +| `:execrows` | the number of affected rows | +| `:execlastid` | the new id | +| `:execresult` | an `ExecResult` (`rows_affected`, `last_insert_id`) | +| `:batchexec` `:batchone` `:batchmany` | runs the query once per element of an `Array`, on one connection | +| `:copyfrom` | bulk insert, all rows or none: `COPY` (PostgreSQL), `LOAD DATA` (MySQL), multi-row `INSERT` (SQLite) | + +| Macro | What it does | +|---|---| +| `sqlc.arg(name)` | names a parameter | +| `sqlc.narg(name)` | names a nullable parameter, which defaults to `nil` | +| `sqlc.slice(ids)` | `IN (...)` with an `Array` (MySQL, SQLite; on PostgreSQL use `= ANY($1::bigint[])`) | +| `sqlc.embed(table)` | puts the table's model into the row | ### Types -| SQL | Ruby | -|---|---| -| integers, floats | `Integer`, `Float` | -| `numeric`, `decimal` | `BigDecimal` | -| text, `varchar` | `String` | -| enums | `String`, plus constants (`ExtendedBioType::MEMOIR`, `ExtendedBioType::ALL`) | -| `date`, `timestamp`, `timestamptz`, `datetime` | `Date`, `Time` | -| `inet`, `cidr` | `IPAddr` | -| `json`, `jsonb` | parsed Ruby objects on PostgreSQL, JSON text on MySQL | -| arrays (PostgreSQL) | `Array[T]` | -| nullable columns and parameters | `T?` | +Values are what the driver gem returns, so the Ruby type of a column depends on the database. +Details and the full list: [Types](docs/Types.md). -Types follow what each driver returns; override any of them with [`overrides`](docs/Options.md#overrides). -Everything else (ranges, geometric types, composite types, SQLite affinity, time zones): [Types and names](docs/Types.md). +| SQL | PostgreSQL | MySQL | SQLite | +|---|---|---|---| +| integers | `Integer` | `Integer` | `Integer` | +| `real`, `double` | `Float` | `Float` | `Float` | +| `numeric`, `decimal` | `BigDecimal` | `BigDecimal` | `Float` | +| text, binary | `String` | `String` | `String` | +| `boolean` | `true` / `false` | `Integer` (0 / 1) | `Integer` (0 / 1) | +| `date` | `Date` | `Date` | `String` | +| `timestamp`, `datetime` | `Time` | `Time` | `String` | +| JSON | parsed Ruby objects | `String` (JSON text) | `String` (JSON text) | +| enums | `String` + constants | `String` + constants | | +| arrays | `Array[T]` | | | +| `inet`, `cidr` | `IPAddr` | | | -### Built for production +Nullable columns and parameters are `T?`. The declared type of any column can be changed with +[`overrides`](docs/Options.md#overrides). + +### At runtime | | | |---|---| -| **Transactions** | `transaction` with commit/rollback, nested savepoints and isolation levels (`isolation:` / `mode:`) | -| **Pooling** | `connection_pool` built for you, or bring your own pool, `SQLite3::Database` or a shared ActiveRecord / Sequel connection | -| **Resilience** | `Timeout`, `Thread#kill`, stopped `Async` tasks, lost connections, deadlocks and forked workers (Puma, Sidekiq) are handled | -| **Concurrency** | plain Ruby that works with the Fiber scheduler (`async`) | -| **Logging** | `connect(params, logger: Rails.logger)` logs `QuerySql#get_author (0.4ms)`; parameter values are never logged | -| **Docs and types** | SQL comments become doc comments; `.rbs` for every model, row and method | - -Details: [Transactions and connections](docs/Transactions.md) · [Generated code](docs/Generated-Code.md) · [Queries](docs/Queries.md). +| **Transactions** | `transaction` commits or rolls back, nests with savepoints and takes an isolation level. [More](docs/Transactions.md) | +| **Connections** | a `connection_pool` built for you, your own pool, or a connection shared with ActiveRecord / Sequel. [More](docs/Connections.md) | +| **Concurrency** | threads, forked workers (Puma, Sidekiq) and the Fiber scheduler (`async`). [More](docs/Connections.md#forked-processes) | +| **Failures** | timeouts, killed threads, lost connections and deadlocks never leave a connection inside a transaction. [More](docs/Transactions.md#when-something-goes-wrong) | +| **Logging** | pass `logger:` to log each query's name and duration; parameter values are never logged. [More](docs/Generated-Code.md#logging) | ## Options -Set them under `options:` of the `codegen` entry. +Set them under `options:` of the `codegen` entry. Details: [Options](docs/Options.md). | Option | Default | What it does | |---|---|---| | `driver` | required | `pg`, `mysql2` or `sqlite3`; must match the sqlc `engine:` | -| `rubyVersion` | `3.3` | oldest Ruby the code must run on (`3.1` – `4.0`); `Data` rows from 3.2, `Struct` below | +| `rubyVersion` | `3.3` | the oldest Ruby the code must run on (`3.1` – `4.0`) | | `generateTypes` | `true` | also generate `.rbs` signatures | | `generateGemfile` | `false` | also generate a `Gemfile` | | `moduleName` | `Codegen` | the module the code lives in, e.g. `MyApp::Db` | -| `overrides` | none | change the Ruby type of columns / parameters | +| `overrides` | none | change the declared Ruby type of columns and parameters | | `rename` | none | rename tables and columns, e.g. `{authors: Writer, bio: biography}` | -| `exactTableNames` | `false` | keep `authors` as `Authors` instead of singularizing to `Author` | -| `preparedStatements` | `true` | `pg` only; `false` for pgbouncer in transaction mode | - -Misspelled options are errors with a "did you mean" hint. Everything about them, including sharing options between -`sql:` blocks: [Options](docs/Options.md). +| `exactTableNames` | `false` | name the model `Authors` instead of the singular `Author` | +| `preparedStatements` | `true` | `pg` only; set `false` behind pgbouncer in transaction mode | ## Documentation +**Using the plugin** + | | | |---|---| -| [Generated code](docs/Generated-Code.md) | files, rows, finders, logging, style | +| [Generated code](docs/Generated-Code.md) | the files, classes and methods you get, and how they are named | | [Queries](docs/Queries.md) | annotations, `:copyfrom`, macros, arguments | -| [Transactions and connections](docs/Transactions.md) | transactions, pools, forks, Async, shared connections | -| [Types and names](docs/Types.md) | type mapping, time zones, enums, naming rules | -| [Options](docs/Options.md) | every option and `overrides` | -| [Troubleshooting](docs/Troubleshooting.md) | "I got this error / this looks odd": causes and fixes, per database | -| [Examples](docs/Examples.md) | per-driver configs and generated output | +| [Types](docs/Types.md) | the Ruby type of every SQL type, time zones, enums | +| [Connections](docs/Connections.md) | pools, shared connections, forked workers, fibers | +| [Transactions](docs/Transactions.md) | commit and rollback, savepoints, isolation, what happens on errors | +| [Options](docs/Options.md) | every option, `overrides` | +| [Troubleshooting](docs/Troubleshooting.md) | known errors and surprises per database, with their fixes | +| [Examples](docs/Examples.md) | schemas, queries and generated output for each database | -## Contributing +**Working on the plugin** -Contributions are welcome: see [Contributing](docs/Contributing.md) for the setup, the test commands and the release -process, and [Architecture](docs/Architecture.md) for how the generator works. +| | | +|---|---| +| [Contributing](docs/Contributing.md) | setup, commands, pull requests | +| [Architecture](docs/Architecture.md) | how the generator works and how it is packed into WebAssembly | +| [Testing](docs/Testing.md) | test layers, CI, what to run before pushing | +| [Coverage](docs/Coverage.md) | every documented behavior and the test that proves it | +| [Releasing](docs/Releasing.md) | the release workflow and versioning | diff --git a/docs/Architecture.md b/docs/Architecture.md index a491ade..73e01bd 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -1,8 +1,18 @@ -# Architecture: how the generator works +# Architecture -A sqlc plugin is a program that gets a protobuf `GenerateRequest` on stdin (the parsed schema, the queries, the -plugin's `options:`) and writes a `GenerateResponse` (a list of files) to stdout. This plugin does exactly that in -pure Ruby; [Design.md](Design.md) explains how it is packed into a WASM module. +How the generator works, why it is built this way, and where to change what. + +## What a sqlc plugin is + +sqlc parses the schema and the queries, then hands the plugin a protobuf `GenerateRequest` on stdin (tables, enums, +queries with their typed parameters and columns, and the plugin's `options:`). The plugin answers on stdout with a +`GenerateResponse`: a list of files. sqlc can run a plugin as a process or as a WASI module. + +This plugin is pure Ruby. It runs as a process during development (`exe/plugin.rb`) and is released as one WASI module: +the [ruby.wasm](https://github.com/ruby/ruby.wasm) interpreter with the plugin inside +([Packaging](#packaging-as-webassembly)). + +## The pipeline ``` stdin ─► Proto (decode) ─► Generator ─┬─► Options / RubyVersion what the user asked for @@ -14,37 +24,41 @@ stdin ─► Proto (decode) ─► Generator ─┬─► Options / RubyVersion stdout ◄─ Proto (encode) ◄─ GenerateResponse (query_sql.rb, query_sql.rbs, models.rb, connection.rb, Gemfile, ...) ``` -## The pieces +Everything is in `lib/sqlc_gen_ruby/`: -| Piece | Responsibility | +| File | Responsibility | |---|---| | `exe/plugin.rb` | entry point. Errors go to stderr with exit code 1 (sqlc shows them); `SQLC_GEN_RUBY_DEBUG=1` adds a backtrace | -| `proto.rb`, `plugin_pb.rb` | a ~150 line protobuf wire codec and the subset of sqlc's `codegen.proto` we read. Field numbers must match upstream. `google-protobuf` is a C extension and cannot run inside ruby.wasm | -| `options.rb`, `ruby_version.rb` | parse + validate the `options:` JSON (`driver`, `rubyVersion`, `overrides`, `rename`, ...; unknown keys are errors) | -| `generator.rb` | the orchestrator: one file per query file (`query_sql.rb`), plus `models.rb`, `connection.rb` (per-connection state and helpers the query classes share), `Gemfile`; builds the module / class / `Data` skeleton | -| `method_gen.rb` | the code of one query: signature (keyword arguments), SQL constant, command (`:one`, `:many`, `:exec`, `:execrows`, `:execlastid`, `:execresult`, `:batchexec` / `:batchone` / `:batchmany`, `:copyfrom`), `sqlc.slice`, bang finder, and the RBS signature | +| `proto.rb`, `plugin_pb.rb` | a small protobuf wire codec and the subset of sqlc's `codegen.proto` the plugin reads. Field numbers must match upstream | +| `options.rb`, `ruby_version.rb` | parse and validate the `options:` JSON; unknown keys are errors | +| `generator.rb` | the orchestrator: one file per query file, plus `models.rb`, `connection.rb` and `Gemfile`; builds the module / class / row skeleton | +| `method_gen.rb` | the code of one query: signature, SQL constant, the annotation's behavior, `sqlc.slice`, the bang method, and the RBS signature | | `row_layout.rb` | turns result columns into row attributes; `sqlc.embed` collapses a table's columns into one model attribute | -| `type_resolver.rb` | RBS type of a column / parameter: driver type table, arrays, nullability (`T?`), `overrides:` (a `db_type` override types array elements too) | +| `type_resolver.rb` | the RBS type of a column or parameter: driver type table, arrays, nullability, `overrides` | | `catalog.rb` | the schema's tables and enums, and their Ruby names | -| `drivers/base.rb` | the driver interface and what all drivers share: `instrument` logging, `NotFound`, transaction RBS | -| `drivers/{pg,mysql2,sqlite3}.rb` | the per-library parts: type mapping, connecting / pooling, executing statements, transactions, `:copyfrom` | -| `code.rb` | text helpers: indentation, `snake_case`, `camelize`, Ruby-keyword-safe parameter names, `rubocop:disable` pairs for schema names that break a naming cop | +| `drivers/base.rb` | the driver interface and what all drivers share: logging, `NotFound`, transaction RBS | +| `drivers/{pg,mysql2,sqlite3}.rb` | the per-library parts: type mapping, connecting and pooling, executing statements, transactions, `:copyfrom` | +| `code.rb` | text helpers: indentation, `snake_case`, `camelize`, keyword-safe names, `rubocop:disable` pairs | + +**The one rule:** `MethodGen` and `Generator` decide *what* is generated and are the same for every database. A driver +only supplies *snippets* (how to prepare a statement, how to read the affected rows, ...) through the hooks of +`Drivers::Base`. A `case driver` in `MethodGen` belongs in a driver hook instead. -**Rule of thumb:** `MethodGen` / `Generator` decide *what* is generated and are the same for every database; a driver only -supplies *snippets* (how to prepare a statement, how to read the affected rows, ...). If you find a `case driver` in -`MethodGen`, it belongs in a driver hook instead. +## The generated code -## What the generated code looks like +Abridged from [`examples/pg/query_sql.rb`](../examples/pg/query_sql.rb), the plugin's real output: ```ruby module PgCodegen - class QuerySql - GET_AUTHOR_SQL = <<~'SQL' # raw heredoc: the SQL is never interpolated - SELECT id, name, bio FROM authors WHERE id = $1 LIMIT 1 - SQL - class GetAuthorRow < ::Data.define(:id, :name, :bio) # a keyword Struct below Ruby 3.2 - end + GET_AUTHOR_SQL = <<~'SQL' # raw heredoc: the SQL is never interpolated + SELECT id, name, bio FROM authors + WHERE id = $1 LIMIT 1 + SQL + + class GetAuthorRow < ::Data.define(:id, :name, :bio) # a keyword Struct below Ruby 3.2 + end + class QuerySql def get_author(id:) # a :one query returns the row or nil ... with_client('get_author') do |client| # one pooled connection + logging query_params = [id] @@ -57,39 +71,76 @@ module PgCodegen def get_author!(...) # ... and gets a bang variant get_author(...) || raise(NotFound, 'GetAuthor returned no row') end - - def transaction(isolation: nil) { |tx| ... } # commit / rollback, nested = SAVEPOINT end end ``` -Look at [`examples/pg/query_sql.rb`](../examples/pg/query_sql.rb) for a complete file; it is the plugin's real output. -Design decisions worth knowing (the *why* is in the comments of `drivers/base.rb` and `drivers/pg.rb`): +## Design decisions -* **PostgreSQL prepared statements are per connection.** A pooled query object is shared by many connections, and a +The *why* behind most of these is in the comments of `drivers/base.rb` and `drivers/pg.rb`. + +* **Idiomatic Ruby, not a port.** The plugin started as a port of + [sqlc-gen-csharp](https://github.com/DaredevilOSS/sqlc-gen-csharp). Its output now follows Ruby conventions instead: + keyword arguments rather than parameter objects, `Data` rows, implicit returns, bang methods, 2-space indentation. +* **Nothing from the plugin leaks into the application.** The generated code only requires the driver gem and + `connection_pool`. +* **A thin layer over the driver.** Values are never converted: `overrides` only change the RBS types. +* **Pool injection.** `QuerySql.new(pool)` accepts anything that responds to `#with`, so an application can share its + pool and tests can pass a fake. `connect(...)` is only a convenience that builds a `connection_pool`. +* **PostgreSQL prepared statements are per connection.** A query object is shared by many pooled connections, and a prepared statement exists only on the connection that prepared it, so each connection remembers its own - (`ConnectionState`). MySQL quotes values client-side and uses `Mysql2::Client#query` so queries cooperate with the - Fiber scheduler. -* **Pool injection.** `QuerySql.new(pool)` accepts anything that responds to `#with`, so an app can share its pool - and tests can pass a fake. `connect(...)` is only a convenience that builds a `connection_pool`. -* **SQLite statements are closed.** They are closed in `ensure` because an open statement blocks `COMMIT`. -* **Parameters are encoded by the driver.** `Driver#encode_param` (e.g. `bytea`, `json` and arrays on `pg`) is used by - every command, batches included; add a type's encoding there, not in `MethodGen`. -* **`:copyfrom` is all or nothing** on every driver (a savepoint / transaction around the load), and MySQL's + (`ConnectionState`, a module mixed into the connection). Statements are named per module and query class, so several + query objects can share a pool. +* **MySQL quotes parameters client-side** and uses `Mysql2::Client#query`, not server-side prepared statements, so + queries cooperate with the Fiber scheduler. +* **SQLite queries do not overlap.** A `SQLite3::Database` is one connection, so the generated code takes one lock per + database (held for a whole transaction) to keep threads and fibers from mixing their transactions. Statements are + closed in `ensure`, because an open statement blocks `COMMIT`. +* **Parameters are encoded by the driver.** `Driver#encode_param` (`bytea`, `json` and arrays on `pg`) is used by every + annotation, batches included. Add a type's encoding there, not in `MethodGen`. +* **`:copyfrom` is all or nothing** on every driver (a savepoint or transaction around the load), and MySQL's `LOAD DATA` warnings (skipped duplicates, coerced values) are raised as errors. -* **Nothing from the plugin leaks into the app.** The generated code only requires the driver gem and `connection_pool`. + +## Packaging as WebAssembly + +sqlc gives a WASI plugin no filesystem, no network and no way to install gems. That shapes the plugin: + +* `lib/` and `exe/` use only Ruby core and `json`. +* `google-protobuf` is a C extension and cannot be loaded inside ruby.wasm, so the plugin has its own codec + (`proto.rb`). + +`scripts/wasm/build.sh` builds `dist/plugin.wasm`: + +1. download the *prebuilt* `ruby-3.3-wasm32-unknown-wasip1-full` release of ruby.wasm (no need to compile CRuby); +2. strip the ~18 MB of DWARF debug info and names from the interpreter (`strip_wasm.rb`); +3. stage a slim stdlib, `lib/` and `exe/`, plus an empty `syntax_suggest/core_ext.rb` (the interpreter loads it to + format errors and would otherwise print a warning before the plugin's own error message); +4. embed that filesystem into the interpreter with `rbwasm pack` (wasi-vfs). + +**The argv trick:** sqlc runs the module as `plugin.wasm /plugin.CodegenService/Generate`, and the interpreter takes +that path as its script. The build embeds a file at exactly that path containing `load '/app/exe/plugin.rb'`. + +Tool versions (sqlc, ruby.wasm) are pinned in [`scripts/tool-versions.env`](../scripts/tool-versions.env). ## Adding things -**A new query feature** (an annotation or macro): -1. read what sqlc sends in a real request: `scripts/capture_request.rb` saves one to `test/fixtures/` (see [Testing](Testing.md)); -2. handle it in `MethodGen` (add a driver hook in `Base` if the code differs per database); -3. add a query using it to `examples/authors/*/{schema,query}.sql` for every driver that can run it; -4. `rake sqlc_generate_process`, review the diff of `examples/`, update the golden files, add unit and end-to-end tests. +**A query feature** (an annotation or macro): + +1. look at what sqlc sends: `scripts/capture_request.rb` saves a real request to `test/fixtures/`; +2. handle it in `MethodGen`, with a hook in `Drivers::Base` if the code differs per database; +3. add a query that uses it to `examples/authors/*/{schema,query}.sql` for every database that can run it; +4. run `rake sqlc_generate_process`, read the diff of `examples/`, update the golden files; +5. add end-to-end tests and list them in [Coverage](Coverage.md) ([Testing](Testing.md)); document the feature. + +**An option:** + +1. parse and default it in `options.rb`, and use it where needed; +2. document it in [Options](Options.md) and the README option table; +3. add a case to `test/options_test.rb` and, if it changes the output, to the `codegen-options` matrix in `build.yml`. -**A new driver** (e.g. `trilogy`): subclass `Drivers::Base`, implement the abstract hooks (the abstract ones raise -`NotImplementedError`), register it in `Generator#instantiate_driver` and `Options::DRIVERS`, add `examples/authors/` + an end-to-end test file `tests/end2end_.rb`, and a service -container job in `.github/workflows/tests.yml`. +**A driver** (for example `trilogy`): -**A new option:** `options.rb` (parse + default), use it where needed, document it in the README option table and docs/Options.md, and -add a case to `test/options_test.rb` and (if it changes output) to the `codegen-options` matrix in `build.yml`. +1. subclass `Drivers::Base` and implement its hooks (the required ones raise `NotImplementedError`); +2. register it in `Generator#instantiate_driver` and `Options::DRIVERS`; +3. add `examples/authors//`, an end-to-end suite `tests/end2end_.rb` and its job in + `.github/workflows/tests.yml`. diff --git a/docs/Connections.md b/docs/Connections.md new file mode 100644 index 0000000..6ec5f3d --- /dev/null +++ b/docs/Connections.md @@ -0,0 +1,89 @@ +# Connections + +A query object runs its queries on connections it takes from a pool. It is safe to create once and share between +threads. + +## Getting a query object + +**Let the plugin build the pool** (`pg`, `mysql2`). `connect` takes the driver's connection parameters and passes the +other options to [`connection_pool`](https://github.com/mperham/connection_pool): + +```ruby +queries = PgCodegen::QuerySql.connect({ dbname: 'app', host: 'localhost' }, size: 5, timeout: 5) +queries = PgCodegen::QuerySql.connect(ENV.fetch('DATABASE_URL'), size: 5) # pg also takes a URL +queries = Mysql2Codegen::QuerySql.connect({ database: 'app', host: 'localhost' }, size: 5) +``` + +**Bring your own pool.** Anything that responds to `#with` and yields a connection works, including a fake in tests: + +```ruby +queries = PgCodegen::QuerySql.new(pool) +``` + +**SQLite** has no `connect`: pass a `SQLite3::Database`, opened with any options (`results_as_hash: true` included). + +```ruby +queries = Sqlite3Codegen::QuerySql.new(SQLite3::Database.new('app.db')) +``` + +A `SQLite3::Database` is one connection, so threads and fibers sharing it take turns. + +Both `connect` and `new` take `logger:`, see [Logging](Generated-Code.md#logging). + +## Sharing a connection with another library + +A connection that ActiveRecord (`raw_connection`), Sequel or your own code also uses keeps its settings. The generated +queries pass what they need with every call instead of changing the connection: + +| Driver | Left untouched | What the generated queries use | +|---|---|---| +| `pg` | `type_map_for_results`, `type_map_for_queries`, `field_name_type` | their own type maps | +| `mysql2` | the client's query options and `Mysql2::Client.default_query_options` | `as: :array, cast: true, cast_booleans: false` | +| `sqlite3` | `results_as_hash` and the other settings of the `SQLite3::Database` | | + +For a transaction the other library opened on the same connection, see +[Transactions](Transactions.md#inside-another-librarys-transaction). + +## Forked processes + +Puma, Sidekiq and Resque workers, and anything else that calls `Process.fork`, can inherit a query object: + +* **Pooled connections:** connections opened before the fork stay the parent's. `connection_pool` replaces them in the + child, and the child lets go of every inherited connection without closing the parent's session, also ones another + thread had checked out at fork time and ones the child never uses. A child that exits does not end them either. +* **A `pg` connection used directly** (not from a `ConnectionPool`) reconnects in the child. +* **A `mysql2` client used directly** raises `Mysql2::Error::ConnectionError` in the child: open clients in the + process that uses them. +* **SQLite:** the sqlite3 gem closes writable databases a child inherits, so open the database again in the child. +* **macOS:** connecting from a forked child can crash in libpq's GSSAPI probe. Set `gssencmode: 'disable'` (or + `PGGSSENCMODE=disable`), as usual for forking `pg` applications there. + +## Fibers and Async + +The generated code is plain synchronous Ruby and works with the Fiber scheduler +([`async`](https://github.com/socketry/async)). Inside `Sync { ... }`: + +| Driver | | +|---|---| +| `pg` | pooled queries run concurrently | +| `mysql2` | pooled queries run concurrently (on Windows `mysql2` blocks the thread) | +| `sqlite3` | queries run one at a time | + +## Lost connections + +A connection the server closed (restart, failover, killed backend) is reconnected (`pg`) or replaced in the pool +(`mysql2`) the next time it is checked out. The call that hit the dead connection still raises. + +A `mysql2` client that mysql2 closed because a running query was interrupted (a stopped `Async` task, `Thread#raise`) +is replaced the same way, without failing the next call. + +Inside a transaction a lost connection is never replaced silently: see +[Transactions](Transactions.md#lost-connections). + +## Prepared statements + +* **PostgreSQL** prepares each query once per connection. Behind pgbouncer in transaction mode, turn that off with + [`preparedStatements: false`](Options.md#preparedstatements-pg). +* **MySQL** does not use server-side prepared statements: parameters are quoted client-side with + `Mysql2::Client#escape` and sent with `Client#query`. +* **SQLite** prepares and closes a statement for each call. diff --git a/docs/Contributing.md b/docs/Contributing.md index 97c8b99..5f310ee 100644 --- a/docs/Contributing.md +++ b/docs/Contributing.md @@ -1,8 +1,26 @@ # Contributing -You need Ruby 3.3 (`bundle install`), [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html) (version in -[`scripts/tool-versions.env`](../scripts/tool-versions.env)), `curl` + `tar` to build the WASM, and PostgreSQL / MySQL -servers (or Docker) for the end-to-end tests. +Contributions are welcome. This page gets you set up; the other contributor docs go deeper. + +| Doc | What | +|---|---| +| [Architecture](Architecture.md) | how the generator works, the WASM packaging, adding a feature / option / driver | +| [Testing](Testing.md) | the test layers, CI workflows, what to run before pushing | +| [Coverage](Coverage.md) | every documented behavior mapped to its end-to-end test | +| [Releasing](Releasing.md) | the release workflow and versioning rules | +| [CLAUDE.md](../CLAUDE.md) | the same orientation, condensed for AI coding assistants | + +## Setup + +You need: + +* Ruby 3.3 and `bundle install`; +* [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html), in the version pinned in + [`scripts/tool-versions.env`](../scripts/tool-versions.env); +* `curl` and `tar` to build the WASM; +* PostgreSQL and MySQL servers, or Docker, for the end-to-end tests. + +## Commands ``` bundle exec rake test # unit + golden tests (no database needed) @@ -11,20 +29,16 @@ rake sqlc_generate_process # regenerate examples/ with exe/plugin.rb rake sqlc_generate_wasm # build dist/plugin.wasm, then regenerate examples/ with it rake end2end_sqlite3 # end-to-end, no server needed rake end2end_pg end2end_mysql2 # end-to-end, needs servers (see Testing.md) -rake run_end2end_tests # all of it in Docker (docker-compose) +rake run_end2end_tests # end-to-end in Docker (docker-compose) ``` -`examples/` is the plugin's committed output and CI fails when it drifts, so regenerate it with every change to the -generated code. Pull requests need both CI workflows (`build.yml`, `tests.yml`) green. +## Making a change -Releases are started by a maintainer (**Actions → Release → Run workflow**) and are automated from there: the full test -gate, the tag, the GitHub release with the tested `sqlc-gen-ruby.wasm` and its sha256, and a check of the published file. +* **Generated output:** `examples/` is the plugin's committed output and CI fails when it drifts. When a change alters + the generated code, regenerate `examples/` and the golden files in the same commit, and read their diff. +* **Tests:** new behavior and bug fixes come with an end-to-end test and a row in [Coverage](Coverage.md). +* **Docs:** user-facing behavior is documented in `docs/`; the README stays a short overview that links there. +* **Pull requests** target `main`, need both CI workflows (`build.yml`, `tests.yml`) green, and get a label: `bug`, + `enhancement` or `documentation` (`breaking` for incompatible changes). -| Doc | What | -|---|---| -| [Architecture.md](Architecture.md) | how the generator works, where to change what, adding a feature / driver / option | -| [Testing.md](Testing.md) | the test layers, CI workflows, what to run before pushing | -| [Releasing.md](Releasing.md) | the release process and versioning rules | -| [Design.md](Design.md) | the WASM packaging and design decisions | -| [Coverage.md](Coverage.md) | every documented behavior mapped to its end-to-end test | -| [../CLAUDE.md](../CLAUDE.md) | the same orientation condensed for AI coding assistants | +Releases are started by a maintainer and automated from there: see [Releasing](Releasing.md). diff --git a/docs/Coverage.md b/docs/Coverage.md index e59793c..6927748 100644 --- a/docs/Coverage.md +++ b/docs/Coverage.md @@ -1,59 +1,110 @@ -# Supported feature coverage +# Coverage -Every behavior documented in the README and `docs/` is linked to the real-database test that exercises it for each applicable engine. `unit:` is used -only when the behavior is rejected or decided during generation and generated code cannot reach it. -CI runs `test/coverage_map_test.rb`, which checks every linked test method exists in its referenced `tests/` or `test/` file. +Every behavior documented in the README and `docs/` is listed here with the test that proves it, for each database it +applies to. See [Testing](Testing.md) for the rules. + +* A cell links to the end-to-end test in `tests/` that runs against the real database. +* *Unit-only checks* are used only when the behavior is decided or rejected during generation, where the generated + code cannot reach it. +* `test/coverage_map_test.rb` checks that every linked test exists. + +## Options and generated files | Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | |---|---|---|---|---| | Options: driver, Ruby target, types, Gemfile, module name, renames, exact names, overrides and global/local merge; CamelCase query file classes (`admin_users.sql` → `AdminUsersSql`); distinct prepared statements for modules sharing a pool (also names over 63 bytes) | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | unit: MySQL-specific `preparedStatements` is rejected | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | [`test_unknown_options_are_rejected_with_a_hint`](../test/options_test.rb#test_unknown_options_are_rejected_with_a_hint), [`test_override_forms`](../test/options_test.rb#test_override_forms), [`test_immutable_data_support`](../test/options_test.rb#test_immutable_data_support), [`test_omitted_parts_mean_the_oldest_covered_release`](../test/options_test.rb#test_omitted_parts_mean_the_oldest_covered_release), [`test_version_decisions_agree_for_every_form`](../test/options_test.rb#test_version_decisions_agree_for_every_form), [`test_major_only_version_orders_as_its_oldest_release`](../test/options_test.rb#test_major_only_version_orders_as_its_oldest_release), [`test_legacy_timeout_guard_is_emitted_only_for_pre_33_targets`](../test/features_test.rb#test_legacy_timeout_guard_is_emitted_only_for_pre_33_targets), [`test_query_file_names_become_camel_case_classes`](../test/features_test.rb#test_query_file_names_become_camel_case_classes), [`test_statement_names_of_different_classes_never_collide`](../test/features_test.rb#test_statement_names_of_different_classes_never_collide) | | `preparedStatements` disabled and configured `moduleName` | [`test_flow`](../tests/end2end_pg.rb#test_flow) (Ruby 4.0 variant) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow) (Ruby 4.0 variant) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow) (Ruby 4.0 variant) | [`test_pg_can_skip_prepared_statements`](../test/features_test.rb#test_pg_can_skip_prepared_statements) | | Generated query/model/connection files and generated `.rbs`; runtime return/attribute checks | [`test_models_are_generated_for_every_table`](../tests/end2end_pg.rb#test_models_are_generated_for_every_table), [`test_declared_types_match_runtime_values`](../tests/end2end_pg.rb#test_declared_types_match_runtime_values) | [`test_models_are_generated_for_every_table`](../tests/end2end_mysql2.rb#test_models_are_generated_for_every_table), [`test_declared_types_match_runtime_values`](../tests/end2end_mysql2.rb#test_declared_types_match_runtime_values), [`test_declared_types_match_runtime_values_for_nulls`](../tests/end2end_mysql2.rb#test_declared_types_match_runtime_values_for_nulls) | [`test_models_are_generated_for_every_table`](../tests/end2end_sqlite3.rb#test_models_are_generated_for_every_table), [`test_declared_types_match_runtime_values`](../tests/end2end_sqlite3.rb#test_declared_types_match_runtime_values) | [`test_unknown_types_in_queries_fall_back_to_untyped`](../test/features_test.rb#test_unknown_types_in_queries_fall_back_to_untyped), [`test_models_fall_back_to_untyped_for_unknown_types`](../test/features_test.rb#test_models_fall_back_to_untyped_for_unknown_types) | +| RuboCop style, query/schema comments and source SQL constants | [`test_flow`](../tests/end2end_pg.rb#test_flow) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow) | [`test_generated_ruby_is_formatted_like_ruby`](../test/features_test.rb#test_generated_ruby_is_formatted_like_ruby), [`test_ruby31_struct_rows_of_unnamed_columns_are_rubocop_clean`](../test/features_test.rb#test_ruby31_struct_rows_of_unnamed_columns_are_rubocop_clean), [`test_query_comments_become_doc_comments`](../test/features_test.rb#test_query_comments_become_doc_comments), [`test_table_column_and_enum_comments_become_doc_comments`](../test/features_test.rb#test_table_column_and_enum_comments_become_doc_comments), [`test_sql_is_a_frozen_raw_heredoc`](../test/features_test.rb#test_sql_is_a_frozen_raw_heredoc) | +| Query names like `HasAuthors` / `IsActive` keep their name and stay RuboCop-clean | [`test_predicate_like_query_names_are_kept`](../tests/end2end_pg.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_mysql2.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_sqlite3.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_rubocop_clean`](../test/features_test.rb#test_predicate_like_query_names_are_rubocop_clean) | + +## Queries + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| | `:one`, `:many`, `:exec`, nil for no row, bang finder methods and `NotFound` error | [`test_flow`](../tests/end2end_pg.rb#test_flow), [`test_one_on_empty_table_returns_nil`](../tests/end2end_pg.rb#test_one_on_empty_table_returns_nil), [`test_every_one_query_has_a_bang_method`](../tests/end2end_pg.rb#test_every_one_query_has_a_bang_method), [`test_bang_finder_raises_not_found`](../tests/end2end_pg.rb#test_bang_finder_raises_not_found) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow), [`test_one_on_empty_table_returns_nil`](../tests/end2end_mysql2.rb#test_one_on_empty_table_returns_nil), [`test_every_one_query_has_a_bang_method`](../tests/end2end_mysql2.rb#test_every_one_query_has_a_bang_method), [`test_bang_finder_raises_not_found`](../tests/end2end_mysql2.rb#test_bang_finder_raises_not_found) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow), [`test_every_one_query_has_a_bang_method`](../tests/end2end_sqlite3.rb#test_every_one_query_has_a_bang_method), [`test_bang_finder_raises_not_found`](../tests/end2end_sqlite3.rb#test_bang_finder_raises_not_found) | [`test_command_signatures`](../test/features_test.rb#test_command_signatures) | | `:execrows`, `:execlastid`, `:execresult` semantics | [`test_execrows_returns_affected_row_count`](../tests/end2end_pg.rb#test_execrows_returns_affected_row_count), [`test_execlastid_returns_the_returning_column`](../tests/end2end_pg.rb#test_execlastid_returns_the_returning_column), [`test_execresult_returns_rows_affected`](../tests/end2end_pg.rb#test_execresult_returns_rows_affected) | [`test_execrows_returns_affected_row_count`](../tests/end2end_mysql2.rb#test_execrows_returns_affected_row_count), [`test_execrows_counts_changed_rows_unless_found_rows_is_set`](../tests/end2end_mysql2.rb#test_execrows_counts_changed_rows_unless_found_rows_is_set), [`test_execlastid_returns_the_new_id`](../tests/end2end_mysql2.rb#test_execlastid_returns_the_new_id), [`test_execresult_returns_rows_affected_and_last_id`](../tests/end2end_mysql2.rb#test_execresult_returns_rows_affected_and_last_id) | [`test_execrows_returns_affected_row_count`](../tests/end2end_sqlite3.rb#test_execrows_returns_affected_row_count), [`test_execlastid_returns_the_returning_column`](../tests/end2end_sqlite3.rb#test_execlastid_returns_the_returning_column), [`test_execresult_returns_rows_affected_and_last_id`](../tests/end2end_sqlite3.rb#test_execresult_returns_rows_affected_and_last_id) | [`test_pg_and_sqlite_execlastid_need_returning`](../test/features_test.rb#test_pg_and_sqlite_execlastid_need_returning) | +| `:execresult` with `RETURNING` runs the statement | covered by `:execresult` semantics | covered by `:execresult` semantics | [`test_execresult_with_returning_runs_the_statement`](../tests/end2end_sqlite3.rb#test_execresult_with_returning_runs_the_statement) | | | `:batchexec`, `:batchone`, `:batchmany`; batches use one connection and are not atomic unless wrapped in a transaction | [`test_batch_commands_run_once_per_args`](../tests/end2end_pg.rb#test_batch_commands_run_once_per_args), [`test_batches_are_not_atomic_without_a_transaction`](../tests/end2end_pg.rb#test_batches_are_not_atomic_without_a_transaction) | [`test_batch_commands_run_once_per_args`](../tests/end2end_mysql2.rb#test_batch_commands_run_once_per_args), [`test_batches_are_not_atomic_without_a_transaction`](../tests/end2end_mysql2.rb#test_batches_are_not_atomic_without_a_transaction) | [`test_batch_commands_run_once_per_args`](../tests/end2end_sqlite3.rb#test_batch_commands_run_once_per_args), [`test_batches_are_not_atomic_without_a_transaction`](../tests/end2end_sqlite3.rb#test_batches_are_not_atomic_without_a_transaction) | [`test_batch_commands_take_an_array_of_args`](../test/features_test.rb#test_batch_commands_take_an_array_of_args) | | `:copyfrom` round trips, all-or-none behavior and conflict clauses | [`test_copyfrom_many_rows_and_none`](../tests/end2end_pg.rb#test_copyfrom_many_rows_and_none), [`test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic`](../tests/end2end_pg.rb#test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic) | [`test_copyfrom_loads_binary_values`](../tests/end2end_mysql2.rb#test_copyfrom_loads_binary_values), [`test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic`](../tests/end2end_mysql2.rb#test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic) | [`test_copyfrom_is_all_or_nothing`](../tests/end2end_sqlite3.rb#test_copyfrom_is_all_or_nothing), [`test_copyfrom_keeps_conflict_clauses`](../tests/end2end_sqlite3.rb#test_copyfrom_keeps_conflict_clauses) | [`test_copyfrom_keeps_or_rejects_conflict_clauses`](../test/features_test.rb#test_copyfrom_keeps_or_rejects_conflict_clauses) | +| `:copyfrom` keeps the case of quoted / mixed-case table and column names (MySQL `LOAD DATA`) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_pg.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_mysql2.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_sqlite3.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | | +| `:copyfrom` takes any `Enumerable` (empty, lazy, `Enumerator`) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_pg.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_mysql2.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_sqlite3.rb#test_copyfrom_takes_any_enumerable) | | | `sqlc.arg`, `sqlc.narg`, positional/named keyword args, optional nils and argument errors | [`test_named_nullable_arguments_are_optional`](../tests/end2end_pg.rb#test_named_nullable_arguments_are_optional), [`test_required_argument_after_optional_named_argument`](../tests/end2end_pg.rb#test_required_argument_after_optional_named_argument), [`test_missing_keyword_arguments_are_errors`](../tests/end2end_pg.rb#test_missing_keyword_arguments_are_errors) | [`test_nullable_named_arg`](../tests/end2end_mysql2.rb#test_nullable_named_arg), [`test_required_argument_after_optional_named_argument`](../tests/end2end_mysql2.rb#test_required_argument_after_optional_named_argument), [`test_missing_keyword_arguments_are_errors`](../tests/end2end_mysql2.rb#test_missing_keyword_arguments_are_errors) | [`test_nullable_named_arg`](../tests/end2end_sqlite3.rb#test_nullable_named_arg), [`test_required_argument_after_optional_named_argument`](../tests/end2end_sqlite3.rb#test_required_argument_after_optional_named_argument), [`test_missing_keyword_arguments_are_errors`](../tests/end2end_sqlite3.rb#test_missing_keyword_arguments_are_errors) | [`test_named_nullable_parameters_are_optional_keywords`](../test/features_test.rb#test_named_nullable_parameters_are_optional_keywords) | | `sqlc.slice`, placeholder order and repeated parameters | unit: PostgreSQL uses arrays instead of slices | [`test_slice`](../tests/end2end_mysql2.rb#test_slice) | [`test_slice`](../tests/end2end_sqlite3.rb#test_slice), [`test_numbered_placeholders_keep_their_bindings_after_slice_expansion`](../tests/end2end_sqlite3.rb#test_numbered_placeholders_keep_their_bindings_after_slice_expansion), [`test_non_slice_numbered_placeholders_keep_sql_order`](../tests/end2end_sqlite3.rb#test_non_slice_numbered_placeholders_keep_sql_order), [`test_batch_numbered_placeholders_keep_sql_order`](../tests/end2end_sqlite3.rb#test_batch_numbered_placeholders_keep_sql_order) | [`test_slices_are_rejected_for_postgres`](../test/features_test.rb#test_slices_are_rejected_for_postgres) | +| Empty `sqlc.slice`: `IN` matches nothing, `NOT IN` everything | unit: PostgreSQL uses arrays instead of slices | [`test_empty_slice_matches_nothing`](../tests/end2end_mysql2.rb#test_empty_slice_matches_nothing), [`test_not_in_an_empty_slice_matches_everything`](../tests/end2end_mysql2.rb#test_not_in_an_empty_slice_matches_everything) | [`test_empty_slice_matches_nothing`](../tests/end2end_sqlite3.rb#test_empty_slice_matches_nothing), [`test_not_in_an_empty_slice_matches_everything`](../tests/end2end_sqlite3.rb#test_not_in_an_empty_slice_matches_everything) | | | `sqlc.embed`, outer-join nil, same table embedded twice | [`test_embed`](../tests/end2end_pg.rb#test_embed), [`test_embed_on_the_outer_side_of_a_join_is_nil_without_a_match`](../tests/end2end_pg.rb#test_embed_on_the_outer_side_of_a_join_is_nil_without_a_match), [`test_same_table_embedded_twice`](../tests/end2end_pg.rb#test_same_table_embedded_twice) | [`test_embed`](../tests/end2end_mysql2.rb#test_embed), [`test_outer_join_embed_is_nil_without_a_match`](../tests/end2end_mysql2.rb#test_outer_join_embed_is_nil_without_a_match), [`test_same_table_embedded_twice`](../tests/end2end_mysql2.rb#test_same_table_embedded_twice) | [`test_embed`](../tests/end2end_sqlite3.rb#test_embed), [`test_outer_join_embed_is_nil_without_a_match`](../tests/end2end_sqlite3.rb#test_outer_join_embed_is_nil_without_a_match), [`test_same_table_embedded_twice`](../tests/end2end_sqlite3.rb#test_same_table_embedded_twice) | [`test_embedded_tables_become_model_attributes`](../test/features_test.rb#test_embedded_tables_become_model_attributes) | +| Unnamed result columns (`column_1`, ...) and aliases with quotes | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_pg.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_pg.rb#test_alias_with_a_quote_is_read) | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_mysql2.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_mysql2.rb#test_alias_with_a_quote_is_read) | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_sqlite3.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_sqlite3.rb#test_alias_with_a_quote_is_read) | | + +## Rows, models and names + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| | Frozen Ruby 3.2+ `Data`, mutable older `Struct`, `with`, `to_h`, pattern matching | [`test_rows_and_embedded_models_have_supported_layout_and_pattern_matching`](../tests/end2end_pg.rb#test_rows_and_embedded_models_have_supported_layout_and_pattern_matching) | [`test_rows_and_embedded_models_have_supported_layout_and_pattern_matching`](../tests/end2end_mysql2.rb#test_rows_and_embedded_models_have_supported_layout_and_pattern_matching) | [`test_rows_and_embedded_models_have_supported_layout_and_pattern_matching`](../tests/end2end_sqlite3.rb#test_rows_and_embedded_models_have_supported_layout_and_pattern_matching) | [`test_struct_classes_are_valid_and_usable`](../test/features_test.rb#test_struct_classes_are_valid_and_usable), [`test_data_classes_are_usable`](../test/features_test.rb#test_data_classes_are_usable) | +| Rows through `Marshal` (Rails.cache) and JSON (`to_h.to_json`) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_pg.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_mysql2.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_sqlite3.rb#test_rows_round_trip_through_marshal_and_json) | | +| Names: keywords, method collisions (also private row hooks such as `initialize`), acronyms, quoted mixed case, repeated `$n`, plural models | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_pg.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names), [`test_repeated_column_parameters_get_distinct_names`](../tests/end2end_pg.rb#test_repeated_column_parameters_get_distinct_names) | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_mysql2.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names), [`test_repeated_positional_parameters_get_distinct_names`](../tests/end2end_mysql2.rb#test_repeated_positional_parameters_get_distinct_names) | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_sqlite3.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names) | [`test_snake_case_keeps_acronyms_together`](../test/features_test.rb#test_snake_case_keeps_acronyms_together), [`test_query_names_cannot_replace_generated_methods`](../test/features_test.rb#test_query_names_cannot_replace_generated_methods), [`test_tables_with_the_same_singular_get_distinct_models`](../test/features_test.rb#test_tables_with_the_same_singular_get_distinct_models), [`test_columns_named_like_row_methods_still_build_and_read_rows`](../test/features_test.rb#test_columns_named_like_row_methods_still_build_and_read_rows) | +| Models named like Ruby / driver constants (`Exception`, `Data`, `Thread`, `Set`, `Monitor`, ...) do not shadow them | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_pg.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_mysql2.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_sqlite3.rb#test_models_named_like_core_constants_do_not_shadow_them) | | + +## Types + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| | Enums, constants (also for keyword / invalid names), database labels and MySQL `SET` | [`test_enum_column`](../tests/end2end_pg.rb#test_enum_column), [`test_enum_values_named_like_ruby_keywords_get_valid_constants`](../tests/end2end_pg.rb#test_enum_values_named_like_ruby_keywords_get_valid_constants) | [`test_enum_and_set_columns`](../tests/end2end_mysql2.rb#test_enum_and_set_columns), [`test_enum_values_named_like_ruby_keywords_get_valid_constants`](../tests/end2end_mysql2.rb#test_enum_values_named_like_ruby_keywords_get_valid_constants) | not supported by the SQLite schema | [`test_models_cover_tables_and_enums`](../test/features_test.rb#test_models_cover_tables_and_enums) | +| Top-level Float JSON parameters, Symbol enum values | [`test_top_level_float_json_parameters_round_trip_exactly`](../tests/end2end_pg.rb#test_top_level_float_json_parameters_round_trip_exactly) | [`test_symbol_enum_value_is_bound_as_its_name`](../tests/end2end_mysql2.rb#test_symbol_enum_value_is_bound_as_its_name) | not SQLite (no enums / JSON encoding) | | | PostgreSQL arrays, JSON/JSONB, type round trips and RBS; a `Time` keeps its instant in a `timestamp` column whatever its offset; a `DateTime` keeps its instant in `timestamp` / `timestamptz` | [`test_array_parameters`](../tests/end2end_pg.rb#test_array_parameters), [`test_numeric_types_round_trip`](../tests/end2end_pg.rb#test_numeric_types_round_trip), [`test_string_types_round_trip`](../tests/end2end_pg.rb#test_string_types_round_trip), [`test_datetime_types_round_trip`](../tests/end2end_pg.rb#test_datetime_types_round_trip), [`test_array_types_round_trip`](../tests/end2end_pg.rb#test_array_types_round_trip), [`test_special_types_round_trip`](../tests/end2end_pg.rb#test_special_types_round_trip), [`test_type_families_round_trip_nil`](../tests/end2end_pg.rb#test_type_families_round_trip_nil), [`test_declared_types_match_runtime_values`](../tests/end2end_pg.rb#test_declared_types_match_runtime_values), [`test_declared_types_match_runtime_values_for_nulls`](../tests/end2end_pg.rb#test_declared_types_match_runtime_values_for_nulls), [`test_timestamp_parameters_keep_their_instant_whatever_their_offset`](../tests/end2end_pg.rb#test_timestamp_parameters_keep_their_instant_whatever_their_offset), [`test_datetime_parameters_keep_their_instant`](../tests/end2end_pg.rb#test_datetime_parameters_keep_their_instant), [`test_json_and_jsonb_parameters_round_trip_all_json_value_types`](../tests/end2end_pg.rb#test_json_and_jsonb_parameters_round_trip_all_json_value_types) | not PostgreSQL types | not PostgreSQL types | [`test_multi_dimensional_arrays`](../test/features_test.rb#test_multi_dimensional_arrays), [`test_more_postgres_types_composites_and_domains`](../test/features_test.rb#test_more_postgres_types_composites_and_domains) | -| MySQL unsigned integers, BIT, YEAR, DATETIME/TIME (a `Time` keeps its instant in any zone, also in `:copyfrom`), JSON, enums/SET and client quoting | not MySQL | [`test_client_side_quoting_round_trips_supported_values`](../tests/end2end_mysql2.rb#test_client_side_quoting_round_trips_supported_values), [`test_time_values_keep_their_instant_whatever_their_offset`](../tests/end2end_mysql2.rb#test_time_values_keep_their_instant_whatever_their_offset), [`test_time_values_follow_a_utc_database_timezone`](../tests/end2end_mysql2.rb#test_time_values_follow_a_utc_database_timezone), [`test_declared_types_match_runtime_values`](../tests/end2end_mysql2.rb#test_declared_types_match_runtime_values) | not MySQL | [`test_mysql_uses_client_side_binding_and_sqlite_closes_prepared_statements`](../test/features_test.rb#test_mysql_uses_client_side_binding_and_sqlite_closes_prepared_statements) | -| SQLite affinity, JSON/UUID, and NUMERIC integer/float results | not SQLite | not SQLite | [`test_declared_types_match_runtime_values`](../tests/end2end_sqlite3.rb#test_declared_types_match_runtime_values), [`test_declared_types_with_parameters`](../tests/end2end_sqlite3.rb#test_declared_types_with_parameters) | [`test_sqlite_declared_types_follow_type_affinity`](../test/features_test.rb#test_sqlite_declared_types_follow_type_affinity), [`test_sqlite_json_and_uuid_declarations_are_strings`](../test/features_test.rb#test_sqlite_json_and_uuid_declarations_are_strings) | -| SQLite `busy_handler_timeout=` waits for another connection without blocking other threads | not SQLite | not SQLite | [`test_busy_handler_timeout_waits_for_another_connection_without_blocking_threads`](../tests/end2end_sqlite3.rb#test_busy_handler_timeout_waits_for_another_connection_without_blocking_threads) | | -| Names: keywords, method collisions (also private row hooks such as `initialize`), acronyms, quoted mixed case, repeated `$n`, plural models | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_pg.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names), [`test_repeated_column_parameters_get_distinct_names`](../tests/end2end_pg.rb#test_repeated_column_parameters_get_distinct_names) | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_mysql2.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names), [`test_repeated_positional_parameters_get_distinct_names`](../tests/end2end_mysql2.rb#test_repeated_positional_parameters_get_distinct_names) | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_sqlite3.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names) | [`test_snake_case_keeps_acronyms_together`](../test/features_test.rb#test_snake_case_keeps_acronyms_together), [`test_query_names_cannot_replace_generated_methods`](../test/features_test.rb#test_query_names_cannot_replace_generated_methods), [`test_tables_with_the_same_singular_get_distinct_models`](../test/features_test.rb#test_tables_with_the_same_singular_get_distinct_models), [`test_columns_named_like_row_methods_still_build_and_read_rows`](../test/features_test.rb#test_columns_named_like_row_methods_still_build_and_read_rows) | +| PostgreSQL arrays of enums created after the connection was opened | [`test_enum_arrays_of_a_type_created_after_connecting_are_decoded`](../tests/end2end_pg.rb#test_enum_arrays_of_a_type_created_after_connecting_are_decoded) | not MySQL | not SQLite | | +| MySQL unsigned integers, BIT, YEAR, DATETIME/TIME (a `Time` keeps its instant in any zone, also in `:copyfrom`), JSON, enums/SET and client quoting | not PostgreSQL | [`test_client_side_quoting_round_trips_supported_values`](../tests/end2end_mysql2.rb#test_client_side_quoting_round_trips_supported_values), [`test_time_values_keep_their_instant_whatever_their_offset`](../tests/end2end_mysql2.rb#test_time_values_keep_their_instant_whatever_their_offset), [`test_time_values_follow_a_utc_database_timezone`](../tests/end2end_mysql2.rb#test_time_values_follow_a_utc_database_timezone), [`test_declared_types_match_runtime_values`](../tests/end2end_mysql2.rb#test_declared_types_match_runtime_values) | not MySQL | [`test_mysql_uses_client_side_binding_and_sqlite_closes_prepared_statements`](../test/features_test.rb#test_mysql_uses_client_side_binding_and_sqlite_closes_prepared_statements) | +| SQLite affinity, JSON/UUID, and NUMERIC integer/float results | not PostgreSQL | not MySQL | [`test_declared_types_match_runtime_values`](../tests/end2end_sqlite3.rb#test_declared_types_match_runtime_values), [`test_declared_types_with_parameters`](../tests/end2end_sqlite3.rb#test_declared_types_with_parameters) | [`test_sqlite_declared_types_follow_type_affinity`](../test/features_test.rb#test_sqlite_declared_types_follow_type_affinity), [`test_sqlite_json_and_uuid_declarations_are_strings`](../test/features_test.rb#test_sqlite_json_and_uuid_declarations_are_strings) | +| SQLite `JSON` / `UUID` columns are `String`s, also numeric-looking text | not PostgreSQL | not MySQL | [`test_json_and_uuid_columns_return_strings_that_look_numeric`](../tests/end2end_sqlite3.rb#test_json_and_uuid_columns_return_strings_that_look_numeric) | | + +## Connections + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| | Connections, injected pools, reconnect behavior and SQLite serialization | [`test_queries_can_share_an_injected_pool`](../tests/end2end_pg.rb#test_queries_can_share_an_injected_pool), [`test_a_connection_closed_by_the_server_is_reconnected`](../tests/end2end_pg.rb#test_a_connection_closed_by_the_server_is_reconnected), [`test_connection_lost_inside_transaction_is_not_reconnected_until_the_transaction_ends`](../tests/end2end_pg.rb#test_connection_lost_inside_transaction_is_not_reconnected_until_the_transaction_ends) | [`test_queries_can_share_an_injected_pool`](../tests/end2end_mysql2.rb#test_queries_can_share_an_injected_pool), [`test_a_connection_closed_by_the_server_is_replaced`](../tests/end2end_mysql2.rb#test_a_connection_closed_by_the_server_is_replaced), [`test_a_stopped_async_task_does_not_leave_a_closed_connection_in_the_pool`](../tests/end2end_mysql2.rb#test_a_stopped_async_task_does_not_leave_a_closed_connection_in_the_pool) | [`test_fibers_sharing_a_database_take_turns`](../tests/end2end_sqlite3.rb#test_fibers_sharing_a_database_take_turns), [`test_threads_sharing_a_database_take_turns_in_transactions`](../tests/end2end_sqlite3.rb#test_threads_sharing_a_database_take_turns_in_transactions), [`test_a_database_returning_hash_rows_still_builds_rows`](../tests/end2end_sqlite3.rb#test_a_database_returning_hash_rows_still_builds_rows) | [`test_engine_must_match_the_driver`](../test/features_test.rb#test_engine_must_match_the_driver) | -| Transaction result, commit/rollback, nested savepoint, exit semantics, rescued PostgreSQL error and rescued MySQL deadlock / lock wait timeout | [`test_nested_savepoint_rolls_back_while_outer_transaction_commits`](../tests/end2end_pg.rb#test_nested_savepoint_rolls_back_while_outer_transaction_commits), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_pg.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version), [`test_transaction_whose_failed_statement_was_rescued_raises_instead_of_committing`](../tests/end2end_pg.rb#test_transaction_whose_failed_statement_was_rescued_raises_instead_of_committing) | [`test_nested_savepoint_rolls_back_while_outer_transaction_commits`](../tests/end2end_mysql2.rb#test_nested_savepoint_rolls_back_while_outer_transaction_commits), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_mysql2.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version), [`test_rescued_deadlock_in_a_transaction_raises_instead_of_committing`](../tests/end2end_mysql2.rb#test_rescued_deadlock_in_a_transaction_raises_instead_of_committing), [`test_rescued_deadlock_in_a_nested_transaction_raises_instead_of_committing`](../tests/end2end_mysql2.rb#test_rescued_deadlock_in_a_nested_transaction_raises_instead_of_committing), [`test_rescued_lock_wait_timeout_rolls_back_only_its_statement`](../tests/end2end_mysql2.rb#test_rescued_lock_wait_timeout_rolls_back_only_its_statement) | [`test_nested_transaction_rolls_back_to_a_savepoint`](../tests/end2end_sqlite3.rb#test_nested_transaction_rolls_back_to_a_savepoint), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_sqlite3.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version) | [`test_nested_transactions_reject_isolation_and_mode`](../test/features_test.rb#test_nested_transactions_reject_isolation_and_mode) | -| Isolation applied; invalid isolation/mode rejected with `ArgumentError` before begin; SQLite locks | [`test_transaction_isolation_level`](../tests/end2end_pg.rb#test_transaction_isolation_level) | [`test_transaction_isolation_level_is_applied`](../tests/end2end_mysql2.rb#test_transaction_isolation_level_is_applied) | [`test_transaction_mode`](../tests/end2end_sqlite3.rb#test_transaction_mode), [`test_immediate_and_exclusive_transactions_block_a_second_connection`](../tests/end2end_sqlite3.rb#test_immediate_and_exclusive_transactions_block_a_second_connection), [`test_unknown_transaction_mode_is_rejected_before_beginning`](../tests/end2end_sqlite3.rb#test_unknown_transaction_mode_is_rejected_before_beginning) | [`test_nested_transactions_reject_isolation_and_mode`](../test/features_test.rb#test_nested_transactions_reject_isolation_and_mode) | -| Logger debug format, transaction logs, secret-free parameters | [`test_logger_receives_query_names_and_durations`](../tests/end2end_pg.rb#test_logger_receives_query_names_and_durations) | [`test_logger_receives_query_names_and_durations`](../tests/end2end_mysql2.rb#test_logger_receives_query_names_and_durations) | [`test_logger_receives_query_names_and_durations`](../tests/end2end_sqlite3.rb#test_logger_receives_query_names_and_durations) | [`test_logger_support`](../test/features_test.rb#test_logger_support) | -| Fiber scheduler concurrency / MySQL no prepared-statement leaks | [`test_queries_run_concurrently_under_the_fiber_scheduler`](../tests/end2end_pg.rb#test_queries_run_concurrently_under_the_fiber_scheduler) | [`test_generated_queries_overlap_under_sync_including_transactions`](../tests/end2end_mysql2.rb#test_generated_queries_overlap_under_sync_including_transactions), [`test_generated_queries_do_not_create_server_side_prepared_statements`](../tests/end2end_mysql2.rb#test_generated_queries_do_not_create_server_side_prepared_statements) | [`test_fibers_sharing_a_database_take_turns`](../tests/end2end_sqlite3.rb#test_fibers_sharing_a_database_take_turns) | | -| Duplicate result columns, unsupported options, invalid `:copyfrom` clauses and generation errors | E2E cannot reach generation-time rejection | E2E cannot reach generation-time rejection | E2E cannot reach generation-time rejection | [`test_duplicate_result_columns_are_rejected`](../test/features_test.rb#test_duplicate_result_columns_are_rejected), [`test_copyfrom_keeps_or_rejects_conflict_clauses`](../test/features_test.rb#test_copyfrom_keeps_or_rejects_conflict_clauses), [`test_unknown_options_are_rejected_with_a_hint`](../test/options_test.rb#test_unknown_options_are_rejected_with_a_hint), [`test_query_names_cannot_replace_generated_methods`](../test/features_test.rb#test_query_names_cannot_replace_generated_methods) | -| RuboCop style, query/schema comments and source SQL constants | [`test_flow`](../tests/end2end_pg.rb#test_flow) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow) | [`test_generated_ruby_is_formatted_like_ruby`](../test/features_test.rb#test_generated_ruby_is_formatted_like_ruby), [`test_ruby31_struct_rows_of_unnamed_columns_are_rubocop_clean`](../test/features_test.rb#test_ruby31_struct_rows_of_unnamed_columns_are_rubocop_clean), [`test_query_comments_become_doc_comments`](../test/features_test.rb#test_query_comments_become_doc_comments), [`test_table_column_and_enum_comments_become_doc_comments`](../test/features_test.rb#test_table_column_and_enum_comments_become_doc_comments), [`test_sql_is_a_frozen_raw_heredoc`](../test/features_test.rb#test_sql_is_a_frozen_raw_heredoc) | -| Unnamed result columns (`column_1`, ...) and aliases with quotes | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_pg.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_pg.rb#test_alias_with_a_quote_is_read) | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_mysql2.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_mysql2.rb#test_alias_with_a_quote_is_read) | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_sqlite3.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_sqlite3.rb#test_alias_with_a_quote_is_read) | | -| Empty `sqlc.slice`: `IN` matches nothing, `NOT IN` everything | unit: PostgreSQL uses arrays instead of slices | [`test_empty_slice_matches_nothing`](../tests/end2end_mysql2.rb#test_empty_slice_matches_nothing), [`test_not_in_an_empty_slice_matches_everything`](../tests/end2end_mysql2.rb#test_not_in_an_empty_slice_matches_everything) | [`test_empty_slice_matches_nothing`](../tests/end2end_sqlite3.rb#test_empty_slice_matches_nothing), [`test_not_in_an_empty_slice_matches_everything`](../tests/end2end_sqlite3.rb#test_not_in_an_empty_slice_matches_everything) | | -| A transaction ended by the database keeps the original error; a query abandoned by Timeout is cancelled; a Timeout in a PostgreSQL transaction raises `Timeout::Error`, also with Ruby 3.1's bundled timeout 0.2.0 | [`test_query_abandoned_by_a_timeout_is_cancelled`](../tests/end2end_pg.rb#test_query_abandoned_by_a_timeout_is_cancelled), [`test_transaction_timeout_rolls_back_and_keeps_connection_usable`](../tests/end2end_pg.rb#test_transaction_timeout_rolls_back_and_keeps_connection_usable) | [`test_deadlock_in_a_nested_transaction_raises_the_deadlock_error`](../tests/end2end_mysql2.rb#test_deadlock_in_a_nested_transaction_raises_the_deadlock_error) | [`test_conflict_that_ends_the_transaction_raises_the_constraint_error`](../tests/end2end_sqlite3.rb#test_conflict_that_ends_the_transaction_raises_the_constraint_error) | | -| Top-level Float JSON parameters, Symbol enum values | [`test_top_level_float_json_parameters_round_trip_exactly`](../tests/end2end_pg.rb#test_top_level_float_json_parameters_round_trip_exactly) | [`test_symbol_enum_value_is_bound_as_its_name`](../tests/end2end_mysql2.rb#test_symbol_enum_value_is_bound_as_its_name) | not SQLite (no enums / JSON encoding) | | -| Modules sharing a connection keep separate prepared statements | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | not MySQL (no server-side prepared statements) | not SQLite (statements are per call) | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) | -| SQLite parameters sqlc does not report (`ON CONFLICT ... DO UPDATE SET`) are generation errors | not PostgreSQL | not MySQL | unit: rejected at generation | [`test_sqlite_parameters_sqlc_does_not_report_are_generation_errors`](../test/features_test.rb#test_sqlite_parameters_sqlc_does_not_report_are_generation_errors), [`test_sqlite_placeholders_in_literals_and_comments_are_not_parameters`](../test/features_test.rb#test_sqlite_placeholders_in_literals_and_comments_are_not_parameters) | -| Forked processes (Puma / Sidekiq workers): pooled connections are replaced in the child without ending the parent's sessions; `pg` direct connections reconnect, `mysql2` direct clients raise, SQLite children open their own database | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_pg.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_mysql2.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable`](../tests/end2end_sqlite3.rb#test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable) | | -| A forked child's plain `exit` leaves the parent's sessions open: a pooled connection another thread has checked out at fork time, a direct connection the child never uses | [pg](../tests/end2end_pg.rb) `test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open` · [mysql2](../tests/end2end_mysql2.rb) `test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open` | -| A connection shared with another library keeps its settings (pg type maps / `field_name_type`, mysql2 query options and `default_query_options`, SQLite `results_as_hash`) | [`test_a_connection_shared_with_another_library_keeps_its_settings`](../tests/end2end_pg.rb#test_a_connection_shared_with_another_library_keeps_its_settings) | [`test_a_client_with_other_query_options_still_returns_typed_rows`](../tests/end2end_mysql2.rb#test_a_client_with_other_query_options_still_returns_typed_rows) | [`test_a_database_shared_with_another_library_keeps_its_settings`](../tests/end2end_sqlite3.rb#test_a_database_shared_with_another_library_keeps_its_settings) | | | `pg`: `QuerySql.connect` takes a connection URL (`DATABASE_URL`) | [`test_connect_accepts_a_connection_url`](../tests/end2end_pg.rb#test_connect_accepts_a_connection_url) | not MySQL (mysql2 takes no URL) | not SQLite (no `connect`) | | -| `transaction` inside a transaction opened by another library: a savepoint (PostgreSQL, SQLite); MySQL limitation: commits it | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_pg.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | [`test_transaction_inside_a_transaction_opened_by_another_library_commits_it`](../tests/end2end_mysql2.rb#test_transaction_inside_a_transaction_opened_by_another_library_commits_it) | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_sqlite3.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | | +| A connection shared with another library keeps its settings (pg type maps / `field_name_type`, mysql2 query options and `default_query_options`, SQLite `results_as_hash`) | [`test_a_connection_shared_with_another_library_keeps_its_settings`](../tests/end2end_pg.rb#test_a_connection_shared_with_another_library_keeps_its_settings) | [`test_a_client_with_other_query_options_still_returns_typed_rows`](../tests/end2end_mysql2.rb#test_a_client_with_other_query_options_still_returns_typed_rows) | [`test_a_database_shared_with_another_library_keeps_its_settings`](../tests/end2end_sqlite3.rb#test_a_database_shared_with_another_library_keeps_its_settings) | | +| Forked processes (Puma / Sidekiq workers): pooled connections are replaced in the child without ending the parent's sessions; `pg` direct connections reconnect, `mysql2` direct clients raise, SQLite children open their own database | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_pg.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_mysql2.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable`](../tests/end2end_sqlite3.rb#test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable) | | +| A forked child's plain `exit` leaves the parent's sessions open: a pooled connection another thread has checked out at fork time, a direct connection the child never uses | [`test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open`](../tests/end2end_pg.rb#test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open) | [`test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open`](../tests/end2end_mysql2.rb#test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open) | not SQLite (a child opens its own database) | | | Schema-per-tenant `search_path` switching with prepared statements | [`test_queries_follow_the_search_path_of_each_tenant`](../tests/end2end_pg.rb#test_queries_follow_the_search_path_of_each_tenant) | not MySQL (databases, no search path) | not SQLite (no schemas) | | | Primary + read-replica query objects; a write to the read-only replica raises and leaves it usable | [`test_primary_and_read_replica_query_objects`](../tests/end2end_pg.rb#test_primary_and_read_replica_query_objects) | [`test_primary_and_read_replica_query_objects`](../tests/end2end_mysql2.rb#test_primary_and_read_replica_query_objects) | [`test_primary_and_read_replica_query_objects`](../tests/end2end_sqlite3.rb#test_primary_and_read_replica_query_objects) | | | Memoized frozen query objects, subclasses and prepended wrappers forwarding `...` / `**`, `SimpleDelegator`, shared across threads | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_pg.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_mysql2.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_sqlite3.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | | -| Rows through `Marshal` (Rails.cache) and JSON (`to_h.to_json`) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_pg.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_mysql2.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_sqlite3.rb#test_rows_round_trip_through_marshal_and_json) | | -| Models named like Ruby / driver constants (`Exception`, `Data`, `Thread`, `Set`, `Monitor`, ...) do not shadow them | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_pg.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_mysql2.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_sqlite3.rb#test_models_named_like_core_constants_do_not_shadow_them) | | +| Fiber scheduler concurrency / MySQL no prepared-statement leaks | [`test_queries_run_concurrently_under_the_fiber_scheduler`](../tests/end2end_pg.rb#test_queries_run_concurrently_under_the_fiber_scheduler) | [`test_generated_queries_overlap_under_sync_including_transactions`](../tests/end2end_mysql2.rb#test_generated_queries_overlap_under_sync_including_transactions), [`test_generated_queries_do_not_create_server_side_prepared_statements`](../tests/end2end_mysql2.rb#test_generated_queries_do_not_create_server_side_prepared_statements) | [`test_fibers_sharing_a_database_take_turns`](../tests/end2end_sqlite3.rb#test_fibers_sharing_a_database_take_turns) | | +| SQLite `busy_handler_timeout=` waits for another connection without blocking other threads | not PostgreSQL | not MySQL | [`test_busy_handler_timeout_waits_for_another_connection_without_blocking_threads`](../tests/end2end_sqlite3.rb#test_busy_handler_timeout_waits_for_another_connection_without_blocking_threads) | | + +## PostgreSQL prepared statements + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| Modules sharing a connection keep separate prepared statements | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | not MySQL (no server-side prepared statements) | not SQLite (statements are per call) | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) | +| PostgreSQL statements dropped by `DEALLOCATE ALL` / `DISCARD ALL`, or refused after a column type change (cached plan must not change result type), are prepared again | [`test_statements_are_prepared_again_after_deallocate_all`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_deallocate_all), [`test_statements_are_prepared_again_after_a_column_type_change`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_a_column_type_change) | not MySQL (no server-side prepared statements) | not SQLite | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) | +| PostgreSQL `PREPARE` interrupted (Timeout / `Thread#raise`, an exception raised by it) after the server ran it leaves the statement usable | [`test_an_interrupt_during_prepare_leaves_the_statement_usable`](../tests/end2end_pg.rb#test_an_interrupt_during_prepare_leaves_the_statement_usable), [`test_a_statement_left_prepared_by_a_failed_prepare_is_used`](../tests/end2end_pg.rb#test_a_statement_left_prepared_by_a_failed_prepare_is_used) | not MySQL (no server-side prepared statements) | not SQLite | | + +## Transactions + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| Transaction result, commit/rollback, nested savepoint, exit semantics, rescued PostgreSQL error and rescued MySQL deadlock / lock wait timeout | [`test_nested_savepoint_rolls_back_while_outer_transaction_commits`](../tests/end2end_pg.rb#test_nested_savepoint_rolls_back_while_outer_transaction_commits), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_pg.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version), [`test_transaction_whose_failed_statement_was_rescued_raises_instead_of_committing`](../tests/end2end_pg.rb#test_transaction_whose_failed_statement_was_rescued_raises_instead_of_committing) | [`test_nested_savepoint_rolls_back_while_outer_transaction_commits`](../tests/end2end_mysql2.rb#test_nested_savepoint_rolls_back_while_outer_transaction_commits), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_mysql2.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version), [`test_rescued_deadlock_in_a_transaction_raises_instead_of_committing`](../tests/end2end_mysql2.rb#test_rescued_deadlock_in_a_transaction_raises_instead_of_committing), [`test_rescued_deadlock_in_a_nested_transaction_raises_instead_of_committing`](../tests/end2end_mysql2.rb#test_rescued_deadlock_in_a_nested_transaction_raises_instead_of_committing), [`test_rescued_lock_wait_timeout_rolls_back_only_its_statement`](../tests/end2end_mysql2.rb#test_rescued_lock_wait_timeout_rolls_back_only_its_statement) | [`test_nested_transaction_rolls_back_to_a_savepoint`](../tests/end2end_sqlite3.rb#test_nested_transaction_rolls_back_to_a_savepoint), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_sqlite3.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version) | [`test_nested_transactions_reject_isolation_and_mode`](../test/features_test.rb#test_nested_transactions_reject_isolation_and_mode) | +| Isolation applied; invalid isolation/mode rejected with `ArgumentError` before begin; SQLite locks | [`test_transaction_isolation_level`](../tests/end2end_pg.rb#test_transaction_isolation_level) | [`test_transaction_isolation_level_is_applied`](../tests/end2end_mysql2.rb#test_transaction_isolation_level_is_applied) | [`test_transaction_mode`](../tests/end2end_sqlite3.rb#test_transaction_mode), [`test_immediate_and_exclusive_transactions_block_a_second_connection`](../tests/end2end_sqlite3.rb#test_immediate_and_exclusive_transactions_block_a_second_connection), [`test_unknown_transaction_mode_is_rejected_before_beginning`](../tests/end2end_sqlite3.rb#test_unknown_transaction_mode_is_rejected_before_beginning) | [`test_nested_transactions_reject_isolation_and_mode`](../test/features_test.rb#test_nested_transactions_reject_isolation_and_mode) | +| `transaction` inside a transaction opened by another library: a savepoint (PostgreSQL, SQLite); MySQL limitation: commits it | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_pg.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | [`test_transaction_inside_a_transaction_opened_by_another_library_commits_it`](../tests/end2end_mysql2.rb#test_transaction_inside_a_transaction_opened_by_another_library_commits_it) | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_sqlite3.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | | | Transaction `tx` used from an `Enumerator`, `Fiber`, `Thread`, `Async` task and nested savepoint; `TransactionEnded` after the transaction | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_pg.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_mysql2.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_sqlite3.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | | -| PostgreSQL arrays of enums created after the connection was opened | [`test_enum_arrays_of_a_type_created_after_connecting_are_decoded`](../tests/end2end_pg.rb#test_enum_arrays_of_a_type_created_after_connecting_are_decoded) | not MySQL | not SQLite | | + +## Errors and interrupts + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| A transaction ended by the database keeps the original error; a query abandoned by Timeout is cancelled; a Timeout in a PostgreSQL transaction raises `Timeout::Error`, also with Ruby 3.1's bundled timeout 0.2.0 | [`test_query_abandoned_by_a_timeout_is_cancelled`](../tests/end2end_pg.rb#test_query_abandoned_by_a_timeout_is_cancelled), [`test_transaction_timeout_rolls_back_and_keeps_connection_usable`](../tests/end2end_pg.rb#test_transaction_timeout_rolls_back_and_keeps_connection_usable) | [`test_deadlock_in_a_nested_transaction_raises_the_deadlock_error`](../tests/end2end_mysql2.rb#test_deadlock_in_a_nested_transaction_raises_the_deadlock_error) | [`test_conflict_that_ends_the_transaction_raises_the_constraint_error`](../tests/end2end_sqlite3.rb#test_conflict_that_ends_the_transaction_raises_the_constraint_error) | | | Interrupts at `BEGIN` / `COMMIT` (`Thread#raise`, raised by the statement) roll back and leave no open transaction | [`test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction`](../tests/end2end_pg.rb#test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction) | [`test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction`](../tests/end2end_mysql2.rb#test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction) | [`test_interrupts_around_begin_and_commit_never_leave_the_database_in_a_transaction`](../tests/end2end_sqlite3.rb#test_interrupts_around_begin_and_commit_never_leave_the_database_in_a_transaction) | | | `Thread#kill` inside a transaction rolls back; on PostgreSQL a kill during a query (inside or outside a transaction) cancels it at once and leaves the connection idle; on MySQL a kill during a query replaces the client and ends its transaction | [`test_thread_kill_rolls_back_and_leaves_connection_idle`](../tests/end2end_pg.rb#test_thread_kill_rolls_back_and_leaves_connection_idle), [`test_thread_kill_during_a_query_cancels_it_and_leaves_the_connection_idle`](../tests/end2end_pg.rb#test_thread_kill_during_a_query_cancels_it_and_leaves_the_connection_idle) | [`test_thread_kill_rolls_back_and_leaves_connection_outside_transaction`](../tests/end2end_mysql2.rb#test_thread_kill_rolls_back_and_leaves_connection_outside_transaction), [`test_thread_kill_during_a_query_replaces_the_connection_and_ends_its_transaction`](../tests/end2end_mysql2.rb#test_thread_kill_during_a_query_replaces_the_connection_and_ends_its_transaction) | [`test_thread_kill_rolls_back_and_leaves_connection_inactive`](../tests/end2end_sqlite3.rb#test_thread_kill_rolls_back_and_leaves_connection_inactive) | | -| PostgreSQL statements dropped by `DEALLOCATE ALL` / `DISCARD ALL`, or refused after a column type change (cached plan must not change result type), are prepared again | [`test_statements_are_prepared_again_after_deallocate_all`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_deallocate_all), [`test_statements_are_prepared_again_after_a_column_type_change`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_a_column_type_change) | not MySQL (no server-side prepared statements) | not SQLite | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) | -| `:execresult` with `RETURNING` runs the statement | covered by `:execresult` semantics | covered by `:execresult` semantics | [`test_execresult_with_returning_runs_the_statement`](../tests/end2end_sqlite3.rb#test_execresult_with_returning_runs_the_statement) | | -| `:copyfrom` keeps the case of quoted / mixed-case table and column names (MySQL `LOAD DATA`) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_pg.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_mysql2.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_sqlite3.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | | -| `:copyfrom` takes any `Enumerable` (empty, lazy, `Enumerator`) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_pg.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_mysql2.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_sqlite3.rb#test_copyfrom_takes_any_enumerable) | | -| Query names like `HasAuthors` / `IsActive` keep their name and stay RuboCop-clean | [`test_predicate_like_query_names_are_kept`](../tests/end2end_pg.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_mysql2.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_sqlite3.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_rubocop_clean`](../test/features_test.rb#test_predicate_like_query_names_are_rubocop_clean) | -| SQLite `JSON` / `UUID` columns are `String`s, also numeric-looking text | not PostgreSQL | not MySQL | [`test_json_and_uuid_columns_return_strings_that_look_numeric`](../tests/end2end_sqlite3.rb#test_json_and_uuid_columns_return_strings_that_look_numeric) | | -| PostgreSQL `PREPARE` interrupted (Timeout / `Thread#raise`, an exception raised by it) after the server ran it leaves the statement usable | [`test_an_interrupt_during_prepare_leaves_the_statement_usable`](../tests/end2end_pg.rb#test_an_interrupt_during_prepare_leaves_the_statement_usable), [`test_a_statement_left_prepared_by_a_failed_prepare_is_used`](../tests/end2end_pg.rb#test_a_statement_left_prepared_by_a_failed_prepare_is_used) | not MySQL (no server-side prepared statements) | not SQLite | | + +## Logging + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| Logger debug format, transaction logs, secret-free parameters | [`test_logger_receives_query_names_and_durations`](../tests/end2end_pg.rb#test_logger_receives_query_names_and_durations) | [`test_logger_receives_query_names_and_durations`](../tests/end2end_mysql2.rb#test_logger_receives_query_names_and_durations) | [`test_logger_receives_query_names_and_durations`](../tests/end2end_sqlite3.rb#test_logger_receives_query_names_and_durations) | [`test_logger_support`](../test/features_test.rb#test_logger_support) | + +## Generation errors + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| Duplicate result columns, unsupported options, invalid `:copyfrom` clauses and generation errors | E2E cannot reach generation-time rejection | E2E cannot reach generation-time rejection | E2E cannot reach generation-time rejection | [`test_duplicate_result_columns_are_rejected`](../test/features_test.rb#test_duplicate_result_columns_are_rejected), [`test_copyfrom_keeps_or_rejects_conflict_clauses`](../test/features_test.rb#test_copyfrom_keeps_or_rejects_conflict_clauses), [`test_unknown_options_are_rejected_with_a_hint`](../test/options_test.rb#test_unknown_options_are_rejected_with_a_hint), [`test_query_names_cannot_replace_generated_methods`](../test/features_test.rb#test_query_names_cannot_replace_generated_methods) | +| SQLite parameters sqlc does not report (`ON CONFLICT ... DO UPDATE SET`) are generation errors | not PostgreSQL | not MySQL | unit: rejected at generation | [`test_sqlite_parameters_sqlc_does_not_report_are_generation_errors`](../test/features_test.rb#test_sqlite_parameters_sqlc_does_not_report_are_generation_errors), [`test_sqlite_placeholders_in_literals_and_comments_are_not_parameters`](../test/features_test.rb#test_sqlite_placeholders_in_literals_and_comments_are_not_parameters) | diff --git a/docs/Design.md b/docs/Design.md deleted file mode 100644 index 9b0d36e..0000000 --- a/docs/Design.md +++ /dev/null @@ -1,51 +0,0 @@ -# Design: a sqlc plugin written in Ruby, shipped as WASM - -sqlc plugins receive a protobuf `GenerateRequest` on stdin and answer with a `GenerateResponse` on stdout, either as a -process or as a WASI module. This plugin is pure Ruby and is distributed as a WASI module (`plugin.wasm`). - -## Source layout -What each file in `lib/sqlc_gen_ruby/` does is in [Architecture.md](Architecture.md). Two constraints shape it: -* `google-protobuf` is a C extension and cannot be loaded inside ruby.wasm, so the plugin brings its own ~150 line - protobuf codec (`proto.rb`) and the subset of sqlc's `codegen.proto` it reads (`plugin_pb.rb`, field numbers must - match upstream). -* Driver differences are data, not copy/paste: `Drivers::Base` defines hooks (how to prepare and run a statement, how - rows are read - `row_data_type` `:hash` for `row['col']` vs `:array` for `row[0]` -, how `:execlastid` / - `:execresult` / `:copyfrom` / transactions work) and a column mapping table; `Pg`, `Mysql2` and `Sqlite3` fill them in. - -## The WASM build (`scripts/wasm/build.sh`) - -1. downloads the *prebuilt* `ruby-3.3-wasm32-unknown-wasip1-full` release from ruby/ruby.wasm (this avoids compiling - CRuby with a wasi-sdk); -2. strips the ~18 MB of DWARF debug info / name section from the interpreter (`strip_wasm.rb`); -3. stages a slim stdlib (the plugin only uses core + `json`) + `lib/` + `exe/`, plus an empty - `syntax_suggest/core_ext.rb` (the interpreter loads it to format errors and would otherwise print a warning before - the plugin's error message); -4. embeds that filesystem into the interpreter with `rbwasm pack` (wasi-vfs), because sqlc gives plugins no filesystem. - -**The argv trick:** sqlc runs the module as `plugin.wasm /plugin.CodegenService/Generate`, so the interpreter treats that -path as its script. The build embeds a file at exactly that path containing `load '/app/exe/plugin.rb'`. - -## Verification -How the plugin is tested (golden files from real sqlc requests, `sqlc diff` over `examples/`, end-to-end tests -against real databases) is described in [Testing.md](Testing.md); how it works inside in [Architecture.md](Architecture.md). - -## What the generated code looks like (and why) -Generated code follows what Ruby code usually looks like rather than mirroring the C# plugin: keyword arguments instead of -parameter objects, `Data` rows, implicit returns, `result.map { ... }`, 2-space indentation, a frozen-string-literal -header, SQL in raw heredocs, `connection_pool` for pooling with an injectable pool, nested transactions via savepoints. -PostgreSQL prepared statements are named per query class and remembered per *connection* (a module mixed into each -connection), so several query objects can share a pool. MySQL quotes parameters client-side and sends queries through -`Mysql2::Client#query` rather than server-side prepared statements. - -## History -The plugin started as a port of the C# `sqlc-gen-csharp` plugin and reproduced its output byte for byte (verified -against the C# binary; see the git history before the commit that removed the C# code). It has since gained the sqlc features of -that plugin (`:execrows`, `:execlastid` for every driver, `:copyfrom`, `sqlc.slice`, `sqlc.embed`, enums, models, -transactions, nullability, `overrides`) and Ruby conventions (bang finders, logger, pooling), so the output is -now idiomatic Ruby rather than a copy of the C# one. - -## Deliberate trade-offs -Things that look like limitations but are design choices (symptoms and fixes: [Troubleshooting](Troubleshooting.md)): -* Overrides only change the RBS types; values are never converted, so the generated code stays a thin layer over the driver. -* SQLite queries do not overlap: a `SQLite3::Database` is one connection, so the generated code takes one lock per - database (held for a whole transaction) to keep threads and fibers from mixing their transactions. diff --git a/docs/Examples.md b/docs/Examples.md index 024acc1..97eb154 100644 --- a/docs/Examples.md +++ b/docs/Examples.md @@ -1,9 +1,17 @@ # Examples -## Engine `postgresql`: [examples/pg](../examples/pg) -### [Schema](../examples/authors/postgresql/schema.sql) | [Queries](../examples/authors/postgresql/query.sql) | [End2End Test](../tests/end2end_pg.rb) +The same example application for each database: a schema, queries that use every supported feature, and the code +the plugin generates from them. The generated code is committed, and the end-to-end tests run against it. -### Config +In each generated directory, `query_sql.rb` has the query methods, `models.rb` the table and enum models, +`connection.rb` the shared helpers, and the `.rbs` files their signatures +(see [The generated code](Generated-Code.md)). + +## PostgreSQL (`pg`) + +[Schema](../examples/authors/postgresql/schema.sql) · [Queries](../examples/authors/postgresql/query.sql) · [Generated code](../examples/pg) · [End-to-end tests](../tests/end2end_pg.rb) + +Options: ```yaml driver: pg rubyVersion: "3.3" @@ -11,11 +19,11 @@ generateTypes: true generateGemfile: false ``` -## Engine `mysql`: [examples/mysql2](../examples/mysql2) +## MySQL (`mysql2`) -### [Schema](../examples/authors/mysql/schema.sql) | [Queries](../examples/authors/mysql/query.sql) | [End2End Test](../tests/end2end_mysql2.rb) +[Schema](../examples/authors/mysql/schema.sql) · [Queries](../examples/authors/mysql/query.sql) · [Generated code](../examples/mysql2) · [End-to-end tests](../tests/end2end_mysql2.rb) -### Config +Options: ```yaml driver: mysql2 rubyVersion: "3.3" @@ -23,11 +31,11 @@ generateTypes: true generateGemfile: false ``` -## Engine `sqlite`: [examples/sqlite3](../examples/sqlite3) +## SQLite (`sqlite3`) -### [Schema](../examples/authors/sqlite/schema.sql) | [Queries](../examples/authors/sqlite/query.sql) | [End2End Test](../tests/end2end_sqlite3.rb) +[Schema](../examples/authors/sqlite/schema.sql) · [Queries](../examples/authors/sqlite/query.sql) · [Generated code](../examples/sqlite3) · [End-to-end tests](../tests/end2end_sqlite3.rb) -### Config +Options: ```yaml driver: sqlite3 rubyVersion: "3.3" diff --git a/docs/Generated-Code.md b/docs/Generated-Code.md index 6ce5841..8a4fdfc 100644 --- a/docs/Generated-Code.md +++ b/docs/Generated-Code.md @@ -1,46 +1,107 @@ # The generated code -For each query file you get `query_sql.rb` with a `QuerySql` class (one method per query; the file name in CamelCase: -`admin_users.sql` → `AdminUsersSql` in `admin_users_sql.rb`), plus: +What `sqlc generate` writes, and how the classes, methods and attributes are named. +The real output for every database is in [Examples](Examples.md). -* `models.rb`: one class per table, one module of constants per enum, -* `connection.rb`: shared helpers, -* the matching `.rbs` files (unless `generateTypes: false`), and a `Gemfile` with `generateGemfile: true`. +## Files -See [Examples](Examples.md) for the real output of every driver. +| File | Contents | +|---|---| +| `query_sql.rb` | one per query file: the `QuerySql` class (one method per query) and the row classes of its queries. The names follow the file: `admin_users.sql` → `AdminUsersSql` in `admin_users_sql.rb` | +| `models.rb` | one class per table, one module of constants per enum | +| `connection.rb` | helpers the query classes share | +| `*.rbs` | signatures for each of the above (unless `generateTypes: false`) | +| `Gemfile` | the gems the code needs (only with `generateGemfile: true`) | -## Rows +Everything lives in one module: `PgCodegen`, `Mysql2Codegen` or `Sqlite3Codegen`, or the +[`moduleName`](Options.md) you set. -Frozen `Data` objects from Ruby 3.2 (with `with` and `to_h`) or mutable keyword `Struct`s below Ruby 3.2; both support -pattern matching: +## The query class ```ruby -case row -in { id:, name: } then ... +queries = PgCodegen::QuerySql.connect({ dbname: 'app' }, size: 5) # see Connections + +queries.get_author(id: 1) # => # or nil +queries.get_author!(id: 1) # raises PgCodegen::QuerySql::NotFound instead of returning nil +queries.list_authors # => [#, ...] +``` + +* One method per query, named after it in snake_case. What it returns depends on the query's annotation: see + [Queries](Queries.md). +* Arguments are keywords: see [Arguments](Queries.md#arguments). +* Every `:one` query also gets a bang method (`get_author!`) that raises `QuerySql::NotFound` when there is no row. +* `transaction`, `connect` and `new` are described in [Transactions](Transactions.md) and [Connections](Connections.md). + +## Rows and models + +A query returns `Row` objects (`GetAuthorRow`); `sqlc.embed` and `models.rb` use one class per table (`Author`). + +| `rubyVersion` | Rows are | | +|---|---|---| +| 3.2 and later | `Data` | frozen; `with` and `to_h` | +| below 3.2 | keyword `Struct` | mutable | + +Both support pattern matching: + +```ruby +case queries.get_author(id: 1) +in { id:, name: } then puts "#{id}: #{name}" +in nil then puts 'no such author' end ``` -## Arguments and finders +## Names -* Arguments are keywords named after the parameters: see [Queries](Queries.md#arguments). -* **Bang finders:** every `:one` query also gets a `get_author!` that raises `QuerySql::NotFound` instead of returning - `nil`. +**Methods and arguments** + +* Query names and parameters are snake_cased, with acronyms kept together (`MIN_ID` → `min_id`, `HTTPServer` → + `http_server`). +* A Ruby keyword gets a trailing `_` as an argument (`end_:`). +* A parameter used twice is one keyword when it is named with `sqlc.arg`; repeated `?` / `$n` of one column become + `id:`, `id_2:`. +* A query whose name would replace a generated method (`Transaction`, `Initialize`, ...) is a generation error. + +**Row attributes** + +* An attribute has the column's name as sqlc reports it: PostgreSQL and SQLite keep `"camelCase"` as `camelCase`, MySQL + reports `camelcase`. +* A result column without a usable name (`SELECT 1, now()`) is named by its position: `column_1`, `column_2`, ... +* A name that would replace a method of the row gets a trailing `_` (`class_`, `hash_`, `initialize_`). `Struct` rows + have more methods than `Data` rows, so a column `count` is `count` from `rubyVersion` 3.2 and `count_` below: check + the generated `.rbs` when you change `rubyVersion`. +* Keywords are fine as attributes: a column `end` is the attribute `end`, but the argument `end_:`. + +**Models** + +* A table's model is its singular name (`authors` → `Author`). A table `authors` next to a table `author` keeps the + plural (`Authors`). Change this with [`rename` or `exactTableNames`](Options.md). +* A model may have the name of a Ruby or driver constant (tables `exceptions`, `data`, `threads`, `sets`, ...). The + generated code refers to the originals as `::Exception`, `::Data`, `::Thread`, ..., so the models never shadow them. +* SQLite: sqlc before 1.30 reports a quoted table name with its quotes, so `CREATE TABLE "Items"` gives the model + `Items` there and `Item` from sqlc 1.30 on. Use `rename` or `exactTableNames` to keep the name when you upgrade sqlc. + +Enum constants are described in [Types](Types.md#enums). ## Logging -`connect(params, logger: Rails.logger)` / `new(pool, logger:)` logs `QuerySql#get_author (0.4ms)` at debug level for -each query and transaction. Parameter values are never logged. +Pass a logger (anything with `#debug`) and every query and transaction is logged at debug level with its name and +duration, for example `PgCodegen::QuerySql#get_author (0.4ms)`: -## Style +```ruby +PgCodegen::QuerySql.connect(params, logger: Rails.logger) +PgCodegen::QuerySql.new(pool, logger: Rails.logger) +``` -* SQL lives in frozen heredoc constants (`GET_AUTHOR_SQL`). -* Comments above a query become its doc comment; table / column / enum comments of the schema become doc comments of - the models. -* The code is RuboCop-clean. Where a name from your schema breaks a RuboCop naming rule (`"UserId"`, `naïve`), that - class or method gets a `rubocop:disable` (on SQLite keep the query files ASCII, see [Troubleshooting](Troubleshooting.md)). - Query names such as `HasAuthors` / `IsActive` keep their name (`has_authors`) with `Naming/PredicatePrefix` disabled. +Parameter values are never logged. -## More +## Style and static checks -* [Transactions and connections](Transactions.md): `transaction`, pools, forks, Async. -* [Types and names](Types.md). +* The SQL of each query is a frozen heredoc constant (`GET_AUTHOR_SQL`). +* A comment above a query becomes the doc comment of its method. Table, column and enum comments of the schema become + doc comments of the models. +* **RuboCop:** the code is RuboCop-clean. Where a name from your schema breaks a naming rule (`"UserId"`, `naïve`), + that class or method gets a `rubocop:disable`. Query names such as `HasAuthors` / `IsActive` keep their name + (`has_authors`), with `Naming/PredicatePrefix` disabled. +* **Steep:** the `.rbs` files type-check with Steep. Add the standard libraries the code uses to your `Steepfile`: + `library "monitor"` for SQLite, and `library "timeout"` for a `rubyVersion` below 3.3. This repository's + [`Steepfile`](../Steepfile) shows the full list for all three drivers. diff --git a/docs/Options.md b/docs/Options.md index 75db913..1449ef8 100644 --- a/docs/Options.md +++ b/docs/Options.md @@ -12,22 +12,32 @@ codegen: moduleName: MyApp::Db ``` -Unknown or misspelled options are errors, with a hint for near misses (`generate_types` → `generateTypes`). -Options shared by several `sql:` blocks can go in the top-level `options:` of `sqlc.yaml` under the plugin's name -(`options: {ruby: {moduleName: MyApp::Db}}`); a codegen entry's own options win, and the `overrides` of both apply. - | Option | Default | What it does | |---|---|---| | `driver` | required | `pg`, `mysql2` or `sqlite3`. Must match the sqlc `engine:`. | -| `rubyVersion` | `3.3` | Oldest Ruby the code must run on, `3.1` – `4.0`. Rows are `Data` classes from 3.2 and `Struct`s below; `3` means any 3.x, so it gets what Ruby 3.1 needs (`Struct`s, the Timeout handling below 3.3). | +| `rubyVersion` | `3.3` | Oldest Ruby the code must run on, `3.1` – `4.0`. Rows are `Data` classes from 3.2 and `Struct`s below; `3` means any 3.x, so it gets what Ruby 3.1 needs (`Struct`s, the [Timeout handling](Transactions.md#ruby-below-33-with-timeout--04) below 3.3). | | `generateTypes` | `true` | Also generate `.rbs` type signatures. | | `generateGemfile` | `false` | Also generate a `Gemfile` with the gems the code needs. | | `moduleName` | `Codegen` | The module the code lives in, e.g. `MyApp::Db`. | | `overrides` | none | Change the type of columns / parameters, see [Overrides](#overrides). | -| `rename` | none | Rename tables (model class) and columns (attribute), e.g. `{authors: Writer, bio: biography}`; `schema.table` works too. | +| `rename` | none | Rename tables (model class) and columns (attribute), e.g. `{authors: Writer, bio: biography}`; `schema.table` works too. See [Names](Generated-Code.md#names). | | `exactTableNames` | `false` | Keep table names as model names (`authors` → `Authors`) instead of singularizing them (`Author`). | | `preparedStatements` | `true` | `pg` only (an error with other drivers). See [below](#preparedstatements-pg). | +Unknown or misspelled options are errors, with a hint for near misses (`generate_types` → `generateTypes`). + +## Sharing options between `sql:` blocks + +Options used by several `sql:` blocks can go in the top-level `options:` of `sqlc.yaml`, under the plugin's name: + +```yaml +options: + ruby: + moduleName: MyApp::Db +``` + +A codegen entry's own options win, and the `overrides` of both apply. + ## `preparedStatements` (pg) `false` sends the SQL with every call instead of preparing it (needed behind pgbouncer in transaction mode). @@ -56,4 +66,5 @@ overrides: ``` The first match wins in this order: `:`, `

.`, `*:`, `db_type`. -Overrides only change the RBS types; the values are whatever the driver returns. Unknown keys are errors. +Overrides only change the declared (RBS) types: values are never converted and stay what the driver returns (see +[Types](Types.md)). Unknown keys are errors. diff --git a/docs/Queries.md b/docs/Queries.md index 7a9d5e0..2b292ef 100644 --- a/docs/Queries.md +++ b/docs/Queries.md @@ -1,48 +1,88 @@ # Queries +Each query in your `.sql` files becomes one method. Its annotation (`-- name: GetAuthor :one`) decides what the +method returns. + ## Annotations | Annotation | Returns | Notes | |---|---|---| -| `:one` | the row or `nil` | plus a bang finder (`get_author!`) that raises `QuerySql::NotFound` instead of returning `nil` | +| `:one` | the row or `nil` | also generates `get_author!`, which raises `QuerySql::NotFound` instead of returning `nil` | | `:many` | `Array` of rows | | | `:exec` | `nil` | | -| `:execrows` | number of affected rows | MySQL: rows changed, not rows matched; connect with `flags: Mysql2::Client::FOUND_ROWS` to count matched rows | -| `:execlastid` | the new id | PostgreSQL / SQLite: needs `RETURNING id`; MySQL: the auto-increment id | -| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | `last_insert_id` is `nil` on PostgreSQL (use `RETURNING`) | -| `:batchexec` / `:batchone` / `:batchmany` | `nil` / `Array` of rows or `nil` / `Array` of `Array`s of rows | take an `Array` of `Args` and run the query once per element, on one connection. Not atomic on their own: wrap them in `transaction` for all-or-nothing | -| `:copyfrom` | `nil` | bulk insert of an `Array` (or any `Enumerable`, also lazy) of `Args`; all rows or none | - -### `:copyfrom` +| `:execrows` | number of affected rows | MySQL counts rows changed, not rows matched; connect with `flags: Mysql2::Client::FOUND_ROWS` to count matched rows | +| `:execlastid` | the new id | PostgreSQL / SQLite: the query needs `RETURNING id`. MySQL: the auto-increment id | +| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | `last_insert_id` is `nil` on PostgreSQL; use `RETURNING` there | +| `:batchexec` | `nil` | see [Batches](#batches) | +| `:batchone` | `Array` of rows or `nil` | | +| `:batchmany` | `Array` of `Array`s of rows | | +| `:copyfrom` | `nil` | see [`:copyfrom`](#copyfrom) | -| | Implementation | -|---|---| -| PostgreSQL | `COPY ... FROM STDIN` | -| MySQL | `LOAD DATA LOCAL INFILE`: needs `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the server; duplicate keys and invalid values raise `Mysql2::Error` instead of being skipped | -| SQLite | batched multi-row `INSERT`s | +## Arguments -Conflict clauses are kept: `INSERT OR REPLACE` / `OR IGNORE` / `ON CONFLICT ... DO ...` on SQLite, `INSERT IGNORE` -(duplicates skipped) and `REPLACE` on MySQL. Clauses `COPY` / `LOAD DATA` cannot express (`ON CONFLICT` / `RETURNING` -on PostgreSQL, `ON DUPLICATE KEY UPDATE` on MySQL) are generation errors; use `:batchexec` for those. +```ruby +queries.get_author(id: 1) +queries.create_author(name: 'Seuss', bio: nil) +``` -* MySQL: `LOAD DATA` names the table and columns as the `INSERT` writes them (`` `MixedCase` ``): sqlc reports MySQL - names in lowercase, but table names are case-sensitive on Linux. -* MySQL: `LOAD DATA` commits an open transaction that was started with raw SQL (`BEGIN`); call `:copyfrom` inside - `transaction` instead. +* Arguments are keywords named after the parameters. Name one yourself with `sqlc.arg(name)`; a parameter sqlc cannot + name (`ANY($1::BIGINT[])`) becomes `param1`, `param2`, ... +* Missing or unknown keywords raise `ArgumentError`. +* A nullable parameter named with `sqlc.arg` / `sqlc.narg` is optional and defaults to `nil`. Other nullable + parameters (`VALUES ($1, $2)`) must be passed, even as `nil`. ## Macros -| Macro | PostgreSQL | MySQL | SQLite | Notes | +| Macro | PostgreSQL | MySQL | SQLite | What it does | |---|:---:|:---:|:---:|---| -| `sqlc.arg` | ✅ | ✅ | ✅ | names a parameter (unnamed ones such as `ANY($1::BIGINT[])` become `param1`, ...) | -| `sqlc.narg` | ✅ | ✅ | ✅ | a nullable parameter (`T?`, defaults to `nil`) | -| `sqlc.slice` | 🚫 | ✅ | ✅ | `IN (sqlc.slice(ids))` takes an `Array` (an empty one: `IN` matches nothing, `NOT IN` everything). PostgreSQL: use `= ANY($1::bigint[])` with an array | -| `sqlc.embed` | ✅ | ✅ | ✅ | the row gets an attribute holding the table's model (`author`, `author2` for the same table twice); `nil` when an outer join found no row | +| `sqlc.arg(name)` | ✅ | ✅ | ✅ | names a parameter | +| `sqlc.narg(name)` | ✅ | ✅ | ✅ | names a nullable parameter (`T?`, defaults to `nil`) | +| `sqlc.slice(ids)` | 🚫 | ✅ | ✅ | `IN (sqlc.slice(ids))` takes an `Array`. With an empty one, `IN` matches nothing and `NOT IN` everything | +| `sqlc.embed(table)` | ✅ | ✅ | ✅ | the row gets an attribute holding the table's model (`author`; `author2` when the table is embedded twice). It is `nil` when an outer join found no row | -🚫 = not offered by sqlc for that database. +🚫 sqlc does not offer `sqlc.slice` for PostgreSQL. Use an array parameter instead: `WHERE id = ANY($1::bigint[])`. -## Arguments +Two result columns with the same name (`a.id, b.id`) are a generation error: alias one (`b.id AS book_id`) or use +`sqlc.embed`. + +## Batches + +`:batchexec`, `:batchone` and `:batchmany` take an `Array` of `Args` and run the query once per element, on one +connection: + +```ruby +queries.create_authors_batch([ + PgCodegen::CreateAuthorsBatchArgs.new(name: 'a', bio: nil), + PgCodegen::CreateAuthorsBatchArgs.new(name: 'b', bio: nil) +]) +``` + +A batch is not atomic on its own: wrap it in [`transaction`](Transactions.md) for all-or-nothing. + +## `:copyfrom` + +A bulk insert of an `Array`, or any `Enumerable` (also a lazy one), of `Args`. Either all rows are inserted or +none. + +| Database | Implementation | Requirements | +|---|---|---| +| PostgreSQL | `COPY ... FROM STDIN` | | +| MySQL | `LOAD DATA LOCAL INFILE` | `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the server | +| SQLite | batched multi-row `INSERT`s | | + +**Conflict clauses** + +| Database | Kept | Generation error (use `:batchexec` instead) | +|---|---|---| +| PostgreSQL | | `ON CONFLICT`, `RETURNING` | +| MySQL | `INSERT IGNORE` (duplicates are skipped), `REPLACE` | `ON DUPLICATE KEY UPDATE` | +| SQLite | `INSERT OR REPLACE`, `INSERT OR IGNORE`, `ON CONFLICT ... DO ...` | | + +**MySQL notes** -Keywords named after the parameters (`get_author(id:)`); `sqlc.arg(name)` names them yourself. Missing or unknown ones -raise `ArgumentError`. Nullable parameters named with `sqlc.arg` / `sqlc.narg` default to `nil`; other nullable -parameters (`VALUES ($1, $2)`) must be passed, even as `nil`. +* Without `INSERT IGNORE`, duplicate keys and invalid values raise `Mysql2::Error` instead of being skipped or coerced + as `LOAD DATA` would do by itself. +* `LOAD DATA` commits a transaction that was opened with raw SQL (`BEGIN`). Call `:copyfrom` inside + [`transaction`](Transactions.md) instead. +* Table and column names are written as in your `INSERT` (`` `MixedCase` ``), so mixed-case tables work on Linux, where + MySQL table names are case-sensitive. diff --git a/docs/Testing.md b/docs/Testing.md index ef6ae8d..0459f7f 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -1,67 +1,109 @@ # Testing and CI -Every documented behavior (README and `docs/`) and supported feature needs a real-database E2E test for each applicable engine, listed in -[`Coverage.md`](Coverage.md). A bug fix starts with a real-database E2E regression that fails before the fix. Use unit -tests only where E2E cannot reach the behavior, such as option validation or generation errors. CI runs the full E2E -suites on every supported Ruby version plus full-suite option variants; generator-options E2E runs on Ruby 3.1 and 4.0. +## The rules -There are four layers. Each one answers a different question, and each runs in CI. +* Every documented behavior (README and `docs/`) needs an end-to-end test against a real database, for each database + it applies to, and a row in [Coverage](Coverage.md). +* A bug fix starts with an end-to-end test that fails before the fix. +* Unit tests are for what end-to-end tests cannot reach: option validation and generation errors. -| Layer | Command | Needs | Question it answers | +## The layers + +Each layer answers a different question, and each runs in CI. + +| Layer | Command | Needs | Question | |---|---|---|---| | Unit + golden | `bundle exec rake test` | nothing | does the plugin produce the expected files? | | Generated-code checks | `bundle exec rake rubocop steep` | nothing | is the generated code clean Ruby with correct types? | -| `sqlc diff` | `rake sqlc_generate_process` / `sqlc_generate_wasm` | sqlc | do the committed `examples/` match the plugin, through the real sqlc and the real WASM? | -| End-to-end | `rake end2end_sqlite3` / `end2end_pg` / `end2end_mysql2` | databases | does the *generated code* behave correctly against a real database? | - -## Unit and golden tests (`test/`) -* `golden_test.rb` runs the plugin on **real requests captured from sqlc** (`test/fixtures/*.bin`) and compares with - the checked-in `examples/`. `extended_golden_test.rb` does the same for option variants (Ruby 3.1/3.2, no types, - `generateGemfile`, ...) and synthetic requests (`synthetic_requests.rb`) against `test/golden/`. - After an intended output change: `UPDATE_GOLDEN=1 bundle exec ruby -Itest test/extended_golden_test.rb`, then **read the diff**. -* `features_test.rb` / `options_test.rb`: one focused test per feature or option. -* `codec_test.rb` cross-checks the hand-written protobuf codec against Google's `google-protobuf` (test-only gem). -* Refreshing a fixture after a new sqlc version or schema change: `scripts/capture_request.rb` (see its header). - -## `examples/` is both documentation and a test -`examples/authors//{schema,query}.sql` are the inputs; `examples//` is the plugin's output, committed. -CI runs `sqlc diff` against it, so a plugin change that is not accompanied by regenerated examples fails the build. +| `sqlc diff` | `sqlc -f sqlc.local.yaml diff` | sqlc | do the committed `examples/` match what the plugin generates now? | +| End-to-end | `rake end2end_sqlite3 end2end_pg end2end_mysql2` | databases | does the generated code behave correctly against a real database? | + +### Unit and golden tests (`test/`) + +| File | What it checks | +|---|---| +| `golden_test.rb` | runs the plugin on real requests captured from sqlc (`test/fixtures/*.bin`) and compares with the committed `examples/` | +| `extended_golden_test.rb` | the same for option variants (Ruby 3.1 / 3.2, no types, `generateGemfile`, ...) and synthetic requests (`synthetic_requests.rb`), against `test/golden/` | +| `features_test.rb`, `options_test.rb` | one focused test per feature or option | +| `codec_test.rb` | the hand-written protobuf codec against Google's `google-protobuf` (a test-only gem) | +| `coverage_map_test.rb` | every test linked from [Coverage](Coverage.md) exists | + +* After an intended output change, rewrite the golden files and **read the diff**: + `UPDATE_GOLDEN=1 bundle exec ruby -Itest test/extended_golden_test.rb` +* After a new sqlc version or a schema change, refresh the fixtures with `scripts/capture_request.rb` (see its header). + +### `examples/` and `sqlc diff` + +`examples/authors//{schema,query}.sql` are the inputs. `examples//` is the plugin's output, committed. +CI runs `sqlc diff` against it with both the process plugin and the WASM, so a plugin change without regenerated +examples fails the build. + Regenerate with `rake sqlc_generate_process` (pure Ruby, fast) or `rake sqlc_generate_wasm` (builds the WASM first). -## End-to-end tests (`tests/`) -They `require` the generated code from `examples/` and run it against a real database, with Minitest. -`rbs_types.rb` additionally checks that returned values match the generated `.rbs` types. -* SQLite needs nothing: `rake end2end_sqlite3`. -* PostgreSQL / MySQL: provide a server, load `examples/authors//schema.sql`, and set `DB_NAME DB_USER DB_PASS - POSTGRES_HOST MYSQL_HOST` (optionally `*_PORT`). MySQL needs `local_infile=1` for `:copyfrom`, and the MySQL tests - connect without a password (`DB_PASS` is only used for PostgreSQL), so use a user without one. - Or run everything in Docker: `rake run_end2end_tests` (docker-compose). -* They cover every query command (batches with encoded `bytea` / `json` values too), transactions (nested, - isolation, `break` / `throw`, a rescued failed statement on PostgreSQL, SQLite threads sharing a database, the - yielded `tx` used from other fibers / threads), models named like Ruby constants (`Exception`, `Data`, ...), pool - injection, logging, bang finders, row members named like row methods (`initialize`), MySQL changed vs. matched - rows, SQLite busy waiting, `:copyfrom` with awkward and binary values, lazy enumerators and its all-or-nothing behaviour, - PostgreSQL `PREPARE`s interrupted after the server ran them, SQLite JSON / UUID text that looks numeric, - Fiber-scheduler concurrency (`async` gem) and statement-leak checks. +### End-to-end tests (`tests/`) + +Minitest suites that `require` the generated code from `examples/` and run it against a real database. +`rbs_types.rb` also checks that returned values match the generated `.rbs` types. + +| Suite | Needs | +|---|---| +| `rake end2end_sqlite3` | nothing | +| `rake end2end_pg` | a PostgreSQL server with `examples/authors/postgresql/schema.sql` loaded | +| `rake end2end_mysql2` | a MySQL server with `examples/authors/mysql/schema.sql` loaded and `local_infile=1` (for `:copyfrom`) | +| `tests/end2end_options.rb` | sqlc and PostgreSQL; generates code with other options and runs it (PostgreSQL, SQLite) | +| `rake run_end2end_tests` | Docker: starts the servers with docker-compose and runs the suites | + +The servers are configured with `DB_NAME`, `DB_USER`, `DB_PASS`, `POSTGRES_HOST`, `MYSQL_HOST` (and optionally +`*_PORT`). The MySQL tests connect without a password (`DB_PASS` is only used for PostgreSQL), so use a MySQL user +that has none. + +What each test proves is listed in [Coverage](Coverage.md). ## CI workflows (`.github/workflows/`) -| Workflow | When | What | + +Every job has `timeout-minutes: 10`. Pull requests need `build.yml` and `tests.yml` green; both skip Markdown-only +changes. + +| Workflow | Runs on | What | |---|---|---| -| `build.yml` | push to `main`, pull requests (not for Markdown-only changes), release | build `plugin.wasm`; lint; `sqlc diff` with the WASM and the process plugin; Steep; generate with option combinations (Ruby 3.1/3.2/3.3/4.0 × types/Gemfile) and check the output with the matching Ruby | -| `tests.yml` | same | unit tests; plugin compatibility on Ruby 3.1 / 3.2 / 3.4 / 4.0; full E2E matrix on Ruby 3.1 / 3.2 / 3.3 / 3.4 / 4.0; Ruby 4.0 module-name variants for MySQL and SQLite and module-name + no-prepared-statements variant for PostgreSQL; a Ruby 3.1 PostgreSQL variant with Ruby's bundled timeout 0.2.0 (`E2E_TIMEOUT_GEM`, `Gemfile.e2e`); PostgreSQL 16.2, MySQL 8.3.0, and SQLite; generator-options E2E on Ruby 3.1 / 4.0 | -| `cross-platform.yml` | called by `build.yml` after `build-wasm` | macOS and Windows × Ruby 3.1 / 4.0 (oldest and newest supported), as two parallel jobs each. *Codegen*: unit + golden tests; `sqlc diff` with the Linux-built `plugin.wasm` and with the process plugin (`exe\plugin.cmd` on Windows); WASM codegen check. *End-to-End*: full E2E for SQLite, PostgreSQL (16 on macOS, 17 on Windows) and MySQL 8.4, installed natively (`ankane/setup-postgres`, `ankane/setup-mysql`, the image's MySQL on Windows), using the generated Gemfiles (installed gems cached per runner image); generator-options E2E | -| `docs.yml` | `examples/` change on `main`, manual | regenerates `docs/Examples.md` and opens a PR | -| `release.yml` | manual only | see [Releasing](Releasing.md) | +| `build.yml` | pull requests, pushes to `main`, releases | builds `plugin.wasm`; lint; `sqlc diff` with the WASM and with the process plugin; Steep; the option matrix; then calls `cross-platform.yml` | +| `tests.yml` | the same | unit tests; plugin compatibility; the end-to-end matrix | +| `cross-platform.yml` | called by `build.yml` | the Linux-built WASM, the process plugin and all end-to-end suites on macOS and Windows | +| `docs.yml` | changes of `examples/authors/` or `sqlc.ci.yaml` on `main`, manually | regenerates `docs/Examples.md` and opens a pull request | +| `release.yml` | manually | see [Releasing](Releasing.md) | + +**`build.yml` option matrix:** generates with combinations of `rubyVersion` (3.1, 3.2, 3.3, 4.0), `generateTypes` and +`generateGemfile`, and checks the output with the matching Ruby. + +**`tests.yml`:** -Tool versions (sqlc, ruby.wasm) live in `scripts/tool-versions.env`; dependabot updates actions and gems weekly. +| Job | Matrix | +|---|---| +| Unit tests | Ruby 3.3 | +| Plugin compatibility | the plugin itself on Ruby 3.1, 3.2, 3.4, 4.0 | +| End-to-end | Ruby 3.1, 3.2, 3.3, 3.4, 4.0 × PostgreSQL 16.2, MySQL 8.3.0, SQLite | +| End-to-end variants | Ruby 4.0 with a custom `moduleName` (all three), plus `preparedStatements: false` on PostgreSQL; Ruby 3.1 on PostgreSQL with Ruby's bundled timeout 0.2.0 (`E2E_TIMEOUT_GEM`, `Gemfile.e2e`) | +| Generator options end-to-end | Ruby 3.1 and 4.0 | + +**`cross-platform.yml`:** macOS and Windows × Ruby 3.1 and 4.0 (the oldest and newest supported), two jobs each. + +* *Codegen:* unit and golden tests; `sqlc diff` with the Linux-built `plugin.wasm` and with the process plugin + (`exe\plugin.cmd` on Windows); a WASM codegen check. +* *End-to-End:* all suites against natively installed databases (PostgreSQL 16 on macOS and 17 on Windows, MySQL 8.4, + SQLite), using the generated Gemfiles. + +Dependabot updates actions and gems weekly; tool versions (sqlc, ruby.wasm) are pinned in `scripts/tool-versions.env`. ## Before you push + ``` bundle exec rake test rubocop steep sqlc -f sqlc.local.yaml diff # committed examples must be current -scripts/tests/run_codegen.rb sqlc.local.yaml 3.1 true true -rake end2end_sqlite3 end2end_pg end2end_mysql2 # full Ruby 3.1–4.0 matrix, plus Ruby 4.0 variants -BUNDLE_GEMFILE=Gemfile.options.e2e bundle exec ruby tests/end2end_options.rb # Ruby 3.1 and 4.0 +scripts/tests/run_codegen.rb sqlc.local.yaml 3.1 true true # generate for Ruby 3.1 with types + Gemfile, check it +rake end2end_sqlite3 end2end_pg end2end_mysql2 +BUNDLE_GEMFILE=Gemfile.options.e2e bundle exec ruby tests/end2end_options.rb ``` -CI also runs the unit suite without Bundler on Ruby 3.1 and 4.0. For Ruby 3.1/3.2 E2E, generate scratch examples -with their matching `rubyVersion` and install the generated Gemfile before running the suite. + +To run the end-to-end suites for Ruby 3.1 / 3.2 locally, generate scratch examples with the matching `rubyVersion` +(`ruby scripts/tests/generate_examples.rb 3.1 sqlc.local.yaml --generate-gemfile`), point `EXAMPLES_DIR` at the +`.examples-ruby-3.1` directory it writes, and install the generated Gemfile first. diff --git a/docs/Transactions.md b/docs/Transactions.md index 7c72e04..5b827ce 100644 --- a/docs/Transactions.md +++ b/docs/Transactions.md @@ -1,4 +1,4 @@ -# Transactions and connections +# Transactions ```ruby queries.transaction(isolation: :serializable) do |tx| @@ -10,87 +10,90 @@ end ## The basics -* Everything in the block runs on one connection. It commits when the block ends (also on `break`, `return`, `throw`) - and rolls back when it raises or the thread is killed. A nested `transaction` is a savepoint and rolls back only its - own part. -* `isolation:` (`pg`, `mysql2`) takes `:read_uncommitted`, `:read_committed`, `:repeatable_read`, `:serializable`; - `mode:` (`sqlite3`) takes `:deferred`, `:immediate`, `:exclusive`. Other values raise `ArgumentError` before anything - runs, and so do `isolation:` / `mode:` on a nested `transaction` (a savepoint cannot change them). -* `tx` is a copy of the query object bound to the transaction: its queries stay in the transaction from any fiber or - thread (an `Enumerator`, an `Async` task, a `Thread`), one at a time, and raise `TransactionEnded` once the - transaction is over. Other query objects (the parent `queries`, another generated class sharing the pool) used from - another fiber or thread check out their own connection and run outside the transaction: pass `tx` instead. -* With `connection_pool` or SQLite, parent-object calls from the same thread reuse the transaction connection; - injected `#with` objects must do the same. -* A `transaction` inside a transaction that another library opened on the same connection (`BEGIN`) is a savepoint on - PostgreSQL and SQLite. MySQL has no nested transactions: its `BEGIN` commits the open transaction first (see - [Troubleshooting](Troubleshooting.md)). +* Everything in the block runs on one connection. +* **Commit:** when the block ends, also with `break`, `return` or `throw`. +* **Rollback:** when the block raises or its thread is killed. +* **Nesting:** a `transaction` inside a transaction is a savepoint and rolls back only its own part. +* **Result:** `transaction` returns the value of the block. + +### Isolation + +| Driver | Keyword | Values | +|---|---|---| +| `pg`, `mysql2` | `isolation:` | `:read_uncommitted`, `:read_committed`, `:repeatable_read`, `:serializable` | +| `sqlite3` | `mode:` | `:deferred`, `:immediate`, `:exclusive` | + +Any other value raises `ArgumentError` before anything runs. So does `isolation:` / `mode:` on a nested `transaction`, +because a savepoint cannot change them. + +### Use `tx` inside the block + +`tx` is a copy of the query object that is bound to the transaction: + +* Its queries stay in the transaction from any fiber or thread (an `Enumerator`, an `Async` task, a `Thread`), one at + a time. +* After the transaction is over it raises `TransactionEnded`. + +The original `queries` object also joins the transaction when it is called from the thread that opened it (with +`connection_pool` or SQLite; a pool you inject must hand out the same connection again inside `#with`). From another +fiber or thread, `queries` and other query classes sharing the pool check out their own connection and run **outside** +the transaction, so pass `tx` there. + +### Inside another library's transaction + +When another library already opened a transaction on the same connection (`BEGIN`): + +* **PostgreSQL, SQLite:** `transaction` is a savepoint in it. +* **MySQL** has no nested transactions, and its `BEGIN` commits the open transaction first. A later rollback no longer + undoes the earlier work. Run the outer work in `transaction` too (inner calls then become savepoints), or finish + the other transaction first. ## When something goes wrong -### Errors inside the block - -* **PostgreSQL:** if a statement fails and the block rescues the error, PostgreSQL still rolls the whole transaction - back, so `transaction` raises `PG::InFailedSqlTransaction` instead of pretending it committed. In a nested - `transaction` only the savepoint is rolled back: the outer block can rescue that error and go on. -* **MySQL:** a deadlock (or a lock wait timeout with `innodb_rollback_on_timeout`) rolls the whole transaction back on - the server, its savepoints too. If the block rescues that error, every later query in the transaction and - `transaction` itself raise it again (a `Mysql2::Error` with the same `error_number`, its message prefixed with - `transaction rolled back:`) instead of running in autocommit mode and reporting a commit; retry the whole - `transaction`. A lock wait timeout without that setting fails only its statement: the block can rescue it and go on. -* **SQLite:** threads and fibers sharing one `SQLite3::Database` take turns; a transaction holds the database until it - ends. - -### Interrupts (`Timeout`, `Thread#raise`, `Thread#kill`, a stopped `Async` task) - -* An interrupt that arrives while `BEGIN`, `COMMIT` or `ROLLBACK` runs waits for it to finish, and a `BEGIN` / - `COMMIT` that raises itself (a stopped `Async` task) is rolled back: the connection never goes back to the pool - inside a transaction. -* **PostgreSQL:** a query interrupted by any of them (inside or outside a transaction) is cancelled at once, so the - connection goes back to the pool idle and a killed transaction rolls back without waiting for its query. -* **MySQL:** a `Thread#kill` during a query closes that client (mysql2 cannot send `ROLLBACK` while it waits for the - query's result), so the server rolls the transaction back and the pool replaces the client. -* **Ruby < 3.3 with timeout < 0.4:** `break`, `return` and `throw` also roll back because Timeout cannot be told - apart; add `gem 'timeout', '>= 0.4'` to the generated Gemfile to keep those exits committing. An old `Timeout` that - interrupts a PostgreSQL query in a transaction still raises `Timeout::Error` (not the `PG::InFailedSqlTransaction` - of the cancelled query) after rolling back. +The rule behind everything below: a connection never goes back to the pool inside a transaction, and `transaction` +never reports a commit that did not happen. + +### A statement fails and the block rescues the error + +* **PostgreSQL** aborts the whole transaction when a statement fails, so `transaction` raises + `PG::InFailedSqlTransaction` at the end instead of pretending it committed. To carry on after an error, run the + statement in a nested `transaction`: only that savepoint is rolled back, and the outer block can rescue the error. +* **MySQL** rolls the whole transaction back on a deadlock (or a lock wait timeout with + `innodb_rollback_on_timeout`), savepoints included. Every later query in the block and `transaction` itself then + raise the error again: a `Mysql2::Error` with the same `error_number` and a message that starts with + `transaction rolled back:`. Retry the whole `transaction`. A lock wait timeout without that setting fails only its + statement, and the block can rescue it and go on. + +### Interrupts + +`Timeout`, `Thread#raise`, `Thread#kill` and a stopped `Async` task can arrive at any moment: + +* **During `BEGIN`, `COMMIT` or `ROLLBACK`:** the interrupt waits for the statement to finish. A `BEGIN` / `COMMIT` + that raises itself (a stopped `Async` task) is rolled back. +* **During a query on PostgreSQL** (inside or outside a transaction): the query is cancelled at once. The connection + goes back to the pool idle, and a killed transaction rolls back without waiting for its query. +* **`Thread#kill` during a query on MySQL:** the client is closed, because mysql2 cannot send `ROLLBACK` while it + waits for the query's result. The server rolls the transaction back and the pool replaces the client. + +### Ruby below 3.3 with timeout < 0.4 + +The `timeout` gem bundled with Ruby before 3.3 leaves a block with `throw`, which cannot be told apart from your own +`break`, `return` or `throw`. With that gem all of them **roll back** instead of committing. + +Use `gem 'timeout', '>= 0.4'` to keep those exits committing. A generated Gemfile (`generateGemfile: true`) for a +`rubyVersion` below 3.3 already has it. + +With the old gem, a `Timeout` that interrupts a PostgreSQL query in a transaction still raises `Timeout::Error` (not +the `PG::InFailedSqlTransaction` of the cancelled query) after rolling back. ### Lost connections -A connection the server closed (restart, failover, killed backend) is reconnected (`pg`) or replaced in the pool -(`mysql2`) the next time it is checked out; the call that hit the dead connection still raises. Inside `transaction` a -lost `pg` connection is not reconnected: every later call in the block and `transaction` itself raise -`PG::ConnectionBad`, so nothing after the loss runs outside the transaction (the server rolled it back). A `mysql2` -client that mysql2 closed because a running query was interrupted (an `Async` task stopped, `Thread#raise`) is -replaced the same way, without failing the next call. - -## Connections - -* `QuerySql.connect(params, **pool_options)` builds a [`connection_pool`](https://github.com/mperham/connection_pool). - To use a pool you already have (or a fake in tests) pass it to `QuerySql.new(pool)`; anything with `#with` works. -* `pg`: `QuerySql.connect` also takes a connection URL: `QuerySql.connect(ENV.fetch('DATABASE_URL'), size: 5)`. -* SQLite: pass a `SQLite3::Database` (opened with any options, `results_as_hash: true` included): `QuerySql.new(db)`. - SQLite transactions belong to the thread that opened them; fibers sharing the same database take turns, and a - transaction holds it until it ends. Only the yielded `tx` runs inside the transaction from other fibers / threads. -* **Shared with another library** (ActiveRecord's `raw_connection`, Sequel, your own code): the connection keeps its - settings. The generated queries pass their own type maps (`pg`; the connection's `type_map_for_results`, - `type_map_for_queries` and `field_name_type` are left as they are) or query options (`mysql2`: rows are read with - `as: :array, cast: true, cast_booleans: false`, whatever the client's or `Mysql2::Client.default_query_options`). - On SQLite `results_as_hash` and the other settings of the `SQLite3::Database` are kept too. -* **Forked processes** (Puma / Sidekiq / Resque workers, `Process.fork`): connections opened before the fork stay the - parent's. connection_pool's fork handling replaces the pooled ones in the child without closing the parent's - sessions, and the child lets go of every inherited connection the generated code has used (also ones another thread - had checked out at fork time, and ones the child never uses), so its exit does not end them either. - * A `pg` connection used directly (not from a `ConnectionPool`) reconnects in the child; a `mysql2` client used that - way raises `Mysql2::Error::ConnectionError` in the child (open clients in the process that uses them). - * SQLite: the sqlite3 gem closes writable databases a child inherits, so open the database again in the child. - * macOS: connecting from a forked child can crash in libpq's GSSAPI probe; set `gssencmode: 'disable'` (or - `PGGSSENCMODE=disable`), as usual for forking `pg` apps there. - -## Async / fibers - -The generated code is plain synchronous Ruby and works with the Fiber scheduler -([`async`](https://github.com/socketry/async)): inside `Sync { ... }`, pooled `pg` and `mysql2` queries run -concurrently (`mysql2` blocks the thread on Windows). MySQL parameters are quoted client-side with -`Mysql2::Client#escape` and sent with `Client#query`, not server-side prepared statements. SQLite queries run one at a -time. +A `pg` connection lost inside `transaction` is not reconnected. Every later call in the block and `transaction` itself +raise `PG::ConnectionBad`, so nothing after the loss runs outside the transaction; the server has rolled it back. + +Outside a transaction, lost connections are replaced: see [Connections](Connections.md#lost-connections). + +### SQLite + +A SQLite transaction belongs to the thread that opened it and holds the database until it ends. Other threads and +fibers sharing the same `SQLite3::Database` wait for it. diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md index 6122467..8f3e48e 100644 --- a/docs/Troubleshooting.md +++ b/docs/Troubleshooting.md @@ -1,6 +1,7 @@ # Troubleshooting -Something failed or looks odd? Find it here. Most of these come from sqlc or the databases, not from the plugin. +Known errors and surprises, with their cause and fix. Most of them come from sqlc or from the database itself, not +from the plugin. ## Any database @@ -9,8 +10,10 @@ Something failed or looks odd? Find it here. Most of these come from sqlc or the | Generation error: two result columns with the same name (`a.id, b.id`) | a row has one member per name | alias one (`b.id AS book_id`) or use `sqlc.embed` | | Generation error: a query named `Transaction`, `Initialize`, ... | it would replace a generated method | rename the query | | Half the rows of a batch were written before an error | `:batchexec` / `:batchone` / `:batchmany` are not atomic on their own | wrap them in `transaction` | -| A timestamp comes back one hour off around a daylight-saving change | `timestamp` / `DATETIME` store the local wall-clock time, so the repeated hour is ambiguous | use `timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), see [Types](Types.md#time-zones) | -| A huge `sqlc.slice` fails | database limits (SQLite and MySQL below) | `:copyfrom` the values into a table and join it | +| A timestamp comes back one hour off around a daylight-saving change | `timestamp` / `DATETIME` store the local wall-clock time, so the repeated hour is ambiguous | use `timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), see [Types](Types.md#dates-and-time-zones) | +| A huge `sqlc.slice` fails | database limits (see MySQL and SQLite below) | `:copyfrom` the values into a table and join it | +| `break` / `return` inside `transaction` rolls back | Ruby below 3.3 with its bundled `timeout` gem | `gem 'timeout', '>= 0.4'`, see [Transactions](Transactions.md#ruby-below-33-with-timeout--04) | +| Steep cannot find `Monitor` / `Timeout` | the generated code uses them (`Monitor` on SQLite, `Timeout` for a `rubyVersion` below 3.3) | add `library "monitor"` / `library "timeout"` to your `Steepfile` | ## PostgreSQL @@ -25,16 +28,13 @@ Something failed or looks odd? Find it here. Most of these come from sqlc or the | You see | Why | Do this | |---|---|---| -| A rollback did not undo work done before `transaction` | `transaction` on a client inside a transaction started with raw SQL or by another library commits it first (MySQL's `BEGIN` does) | run the outer work in `transaction` too (inner calls become savepoints), or finish the other transaction first | +| A rollback did not undo work done before `transaction` | `transaction` on a client inside a transaction started with raw SQL or by another library commits it first (MySQL's `BEGIN` does) | run the outer work in `transaction` too (inner calls become savepoints), or finish the other transaction first, see [Transactions](Transactions.md#inside-another-librarys-transaction) | | `Unknown column 'mixeditems.itemid'` with `sqlc.embed(MixedItems)` | sqlc lowercases the table name it expands to, and MySQL table names are case-sensitive on Linux; the plugin only receives the lowercased name | give the table an alias and embed that: `` SELECT sqlc.embed(m) FROM `MixedItems` AS m `` | | `:copyfrom` is refused / fails | `LOAD DATA LOCAL INFILE` needs `local_infile` | `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the server, see [Queries](Queries.md#copyfrom) | | A `sqlc.slice` fails with a packet error | the statement exceeds the server's `max_allowed_packet` | `:copyfrom` into a table and join it | | `:execrows` counts changed rows, not matched rows | MySQL's default | connect with `flags: Mysql2::Client::FOUND_ROWS` | | `:copyfrom` committed my raw-SQL transaction | `LOAD DATA` commits an open `BEGIN` transaction | call `:copyfrom` inside `transaction` | -MySQL does not use server-side prepared statements: parameters are quoted client-side with `Mysql2::Client#escape` -and sent with `Client#query`. - ## SQLite | You see | Why | Do this | @@ -42,13 +42,12 @@ and sent with `Client#query`. | Queries lose their last characters (`FROM user` instead of `FROM users`) | sqlc (1.27 – 1.31 at least) misplaces the SQL of a query file that contains a non-ASCII character in a name, literal or comment. A sqlc bug, not a plugin one | keep SQLite query files ASCII and pass such values as parameters | | Generation error with `sqlc.arg(...)` in `ON CONFLICT ... DO UPDATE SET` | sqlc does not see parameters there (it leaves `sqlc.arg(...)` in the SQL or reports fewer parameters than there are `?`) | use `excluded.` (`DO UPDATE SET n = counters.n + excluded.n`) or move the value into `VALUES` | | `SQLite3::SQLException: too many SQL variables` | a statement takes at most 32,766 parameters (`SQLITE_MAX_VARIABLE_NUMBER`) | `:copyfrom` into a table and join it | -| An `Integer` where `Float` is declared | `NUMERIC` affinity | cast in the query or use `overrides` | +| An `Integer` where `Float` is declared | `NUMERIC` affinity: SQLite stores `5.0` as `5` | call `to_f` on the value, or declare the column `Numeric` with [`overrides`](Options.md#overrides) | | Other threads freeze while the database is locked | `busy_timeout=` waits while holding Ruby's GVL | use `busy_handler_timeout=`, which waits without blocking them | -| A fiber or thread waits for a transaction | SQLite transactions belong to the thread that opened them; fibers sharing a database take turns, and a transaction holds it until it ends | use the yielded `tx` inside the transaction | -| The model is named `Items` instead of `Item` | sqlc before 1.30 (including the pinned 1.27) reports a quoted table name with its quotes | `rename` or `exactTableNames`, see [Names](Types.md#names) | -| Steep complains about `Monitor` / `Timeout` | the generated code uses them | add `library "monitor"` (and `library "timeout"` for `rubyVersion` below 3.3) to your `Steepfile` | +| A fiber or thread waits for a transaction | a SQLite transaction holds the database until it ends, and belongs to the thread that opened it | keep transactions short; inside one, use the yielded `tx` from other fibers / threads | +| The model is named `Items` instead of `Item` | sqlc before 1.30 (including the pinned 1.27) reports a quoted table name with its quotes | `rename` or `exactTableNames`, see [Names](Generated-Code.md#names) | -## Options from the C# plugin +## Coming from sqlc-gen-csharp -`withAsyncSuffix` / `withCancellationToken` (use the Fiber scheduler), `useDapper` and the target framework / csproj -options do not apply to Ruby; `moduleName` replaces the namespace. +These options of the C# plugin do not exist here: `withAsyncSuffix` / `withCancellationToken` (use the Fiber scheduler), `useDapper` and the target +framework / csproj options. `moduleName` replaces the namespace option. diff --git a/docs/Types.md b/docs/Types.md index fa1223e..e2af99f 100644 --- a/docs/Types.md +++ b/docs/Types.md @@ -1,82 +1,93 @@ -# Types and names +# Types -## Types +The generated code does not convert values: a column has the Ruby class its driver gem returns, and the `.rbs` +declares that class. Nullable columns and parameters are `T?`. -Types follow what the drivers return. +To declare a different type for a column, use [`overrides`](Options.md#overrides). -| SQL | Ruby | -|---|---| -| integers, floats | `Integer`, `Float` | -| `numeric` / `decimal` | `BigDecimal` | -| text | `String` | -| dates and times | `Date`, `Time` | -| network addresses | `IPAddr` | -| booleans | `true` / `false` | -| nullable columns and parameters | `T?` | +## Type mapping + +| SQL | PostgreSQL | MySQL | SQLite | +|---|---|---|---| +| integers | `Integer` | `Integer` (also `YEAR`) | `Integer` | +| `real`, `float`, `double` | `Float` | `Float` | `Float` | +| `numeric`, `decimal` | `BigDecimal` | `BigDecimal` | `Float` | +| `boolean` | `true` / `false` | `Integer` (0 / 1) | `Integer` (0 / 1) | +| text (`char`, `varchar`, `text`, ...) | `String` | `String` | `String` | +| binary (`bytea`, `blob`, `binary`, ...) | `String` | `String` | `String` | +| `date` | `Date` | `Date` | `String` | +| `timestamp`, `timestamptz`, `datetime` | `Time` | `Time` | `String` | +| `time` | `String` | `Time` | `String` | +| JSON | parsed (`untyped`) | `String` (JSON text) | `String` (JSON text) | +| `uuid` | `String` | | `String` | +| enums | `String` | `String` (`ENUM`, `SET`) | | +| arrays | `Array[T]` | | | +| `inet`, `cidr` | `IPAddr` | | | +| `bit` | `String` | `String` | | ### PostgreSQL -* `json` / `jsonb` are parsed (`untyped`). A `String` passed to a `json` / `jsonb` parameter is JSON text - (`'{"a":1}'`): pass `'"hi"'` or `'hi'.to_json` for a JSON string. Hashes, Arrays, numbers, `true` / `false` and `nil` - are encoded for you. -* Arrays are `Array[T]` (`Array[Array[T]]` for `T[][]`); arrays of types the pg gem does not decode (`uuid[]`, enum - arrays, ...) are `Array[String]`, also for types created after the connection was opened (a migration run while the - application is up). -* Geometric, range, full-text, `hstore`, `ltree`, `timetz`, `reg*` and composite types are `String`s; domains and - unknown types are `untyped`. -* A `Time` or `DateTime` parameter of a `timestamp` (without time zone) column (also in `:copyfrom` and batches) keeps - its instant whatever its UTC offset: it is written in the process's local zone, the zone the pg gem reads `timestamp` - values back in (PostgreSQL itself ignores the offset of a `timestamp` value). A `DateTime` of a `timestamptz` column - keeps its instant too (the pg gem alone would send its date only); the RBS still declares `::Time`, the type the - rows return. Elements of `timestamp[]` arrays are written as given. +* **JSON:** `json` / `jsonb` columns are parsed. For a parameter, Hashes, Arrays, numbers, `true` / `false` and `nil` + are encoded for you. A `String` is taken as JSON text (`'{"a":1}'`), so pass `'"hi"'` or `'hi'.to_json` for a JSON + string. +* **Arrays** are `Array[T]`, and `Array[Array[T]]` for `T[][]`. Arrays of types the pg gem does not decode (`uuid[]`, + enum arrays, ...) are `Array[String]`, also for types created after the connection was opened (a migration that + runs while the application is up). +* **`String`:** `money`, `interval`, `xml`, `macaddr`, geometric, range, full-text, `hstore`, `ltree`, `timetz`, + `reg*` and composite types. +* **`untyped`:** domains and types the plugin does not know. ### MySQL -* `JSON` columns and parameters are JSON text `String`s, as `mysql2` returns them. -* A `Time` / `DateTime` parameter (also in `:copyfrom`) keeps its instant whatever its UTC offset: it is written in - the zone `mysql2` reads `DATETIME` / `TIMESTAMP` values back in, the process's local zone, or UTC for a client - created with `database_timezone: :utc` (`connect(params.merge(database_timezone: :utc))`). The server also converts - `TIMESTAMP` columns with the session `time_zone`, so keep that equal to the client's zone. +* `BOOLEAN` is `TINYINT(1)`, so it is an `Integer`. +* `JSON` columns and parameters are JSON text, as `mysql2` returns them. ### SQLite -* Follows its [type affinity rules](https://www.sqlite.org/datatype3.html#determination_of_column_affinity) - (`VARCHAR(255)` → `String`, `DECIMAL(10,5)` → `Float`, `UNSIGNED BIG INT` → `Integer`). -* `JSON` and `UUID` columns are `String`s, also when SQLite stored numeric-looking text as a number (`'42'` → `"42"`, - in SQLite's spelling: `'1.50'` → `"1.5"`). -* The generated code uses `Monitor`: if you run Steep, add `library "monitor"` to your `Steepfile`. Code generated for - a `rubyVersion` below 3.3 also uses `Timeout` (see [Transactions](Transactions.md)): add `library "timeout"` too. - -### Time zones - -A column without a time zone (PostgreSQL `timestamp`, MySQL `DATETIME`) stores the local wall-clock time, so an -instant in the hour repeated when the local zone falls back from daylight saving time (e.g. 02:30 on the last Sunday of -October in Europe/Berlin happens twice) reads back as the first of the two, one hour off for the second. Use -`timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), to keep every instant. - -### Enums - -Enums are `String`s, with constants in `models.rb` (`ExtendedBioType::MEMOIR`, `ExtendedBioType::ALL`). Values that are -not valid constant names still get one: `'end'` → `END_`, `'all'` → `ALL_`, `'2x'` → `VALUE_2X`, `'a b'` → `A_B`; values -that end up with the same name get `_2`, `_3`, ... - -## Names - -* Query methods and keyword arguments are snake_cased, with acronyms kept together (`MIN_ID` → `min_id`, - `HTTPServer` → `http_server`); Ruby keywords get a trailing `_` (`end_:`). -* Row members use the column spelling reported by sqlc for the engine, except names that conflict with row methods, - including the private ones `Data` / `Struct` call while building and copying rows (`class_`, `hash_`, `initialize_`): - PostgreSQL and SQLite keep `"camelCase"` as `:camelCase`, while MySQL reports it as `:camelcase`. A column named - `end` is the row member `:end` but the argument `end_:`. `Struct` rows (`rubyVersion` below 3.2) have more methods - than `Data` rows, so more names get the `_`: a column `count` is `:count` from 3.2 but `:count_` below - (`Struct#count` would otherwise be `Enumerable#count`); check the generated `.rbs` when changing `rubyVersion`. -* A result column without a usable name (`SELECT 1, now()`) is named by its position: `column_1`, `column_2`, ... -* A parameter used twice is one keyword when named with `sqlc.arg`; repeated `?` / `$n` of one column become `id:`, - `id_2:`. -* Models may be named like Ruby or driver constants (tables `exceptions`, `data`, `threads`, `sets`, ...): the - generated code and `.rbs` refer to those as `::Exception`, `::Data`, `::Thread`, ... so the models cannot shadow them. -* A table `authors` next to a table `author` keeps its plural model name (`Authors`). Query names that would replace a - generated method (`Transaction`, `Initialize`, ...) are errors. -* SQLite: sqlc before 1.30 (including the pinned 1.27) reports a quoted table name with its quotes, so - `CREATE TABLE "Items"` gives the model `Items` there and `Item` with sqlc 1.30 and later (PostgreSQL / MySQL give - `Item` with any version). Use `rename` or `exactTableNames` to keep the name when upgrading sqlc. +SQLite has no fixed column types: the declared type only gives a column an +[affinity](https://www.sqlite.org/datatype3.html#determination_of_column_affinity), and the plugin follows the same +rules (`VARCHAR(255)` → `String`, `DECIMAL(10,5)` → `Float`, `UNSIGNED BIG INT` → `Integer`). + +* Dates, times and booleans are stored as text or numbers, so they are `String` or `Integer`. +* A `NUMERIC` / `DECIMAL` column is declared `Float`, but SQLite can return an `Integer` (it stores `5.0` as `5`). +* `JSON` and `UUID` columns are always `String`s, also when SQLite stored numeric-looking text as a number. The text + then comes back in SQLite's spelling: `'42'` → `"42"`, `'1.50'` → `"1.5"`. + +## Dates and time zones + +A `Time` or `DateTime` parameter keeps its instant whatever its UTC offset, also in `:copyfrom` and batches: + +* **PostgreSQL `timestamp`** (without time zone): the value is written in the process's local zone, the zone the pg + gem reads `timestamp` values back in. Elements of `timestamp[]` arrays are written as given. +* **PostgreSQL `timestamptz`:** a `DateTime` keeps its instant too. The RBS declares `::Time`, the class rows return. +* **MySQL `DATETIME` / `TIMESTAMP`:** the value is written in the zone `mysql2` reads values back in: the process's + local zone, or UTC for a client created with `database_timezone: :utc`. The server also converts `TIMESTAMP` columns + with the session `time_zone`, so keep that equal to the client's zone. + +**Daylight saving time.** A column without a time zone (PostgreSQL `timestamp`, MySQL `DATETIME`) stores the local +wall-clock time. When the local zone falls back from daylight saving time, one hour happens twice (02:30 on the last +Sunday of October in Europe/Berlin), and an instant in the second of the two reads back as the first, one hour off. +Use `timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), to keep every instant. + +## Enums + +An enum column is a `String`. `models.rb` has a module of constants for each enum: + +```ruby +PgCodegen::ExtendedBioType::MEMOIR # => "Memoir" +PgCodegen::ExtendedBioType::ALL # => ["Autobiography", "Biography", "Memoir"] +``` + +On PostgreSQL the module is named after the enum type. MySQL enums belong to a column, so the module is named after +the table and column: `Mysql2Codegen::ExtendedBiosBioType`. A MySQL `SET` column gets constants the same way. + +Values that are not valid constant names still get a constant: + +| Value | Constant | +|---|---| +| `'end'` | `END_` | +| `'all'` | `ALL_` | +| `'2x'` | `VALUE_2X` | +| `'a b'` | `A_B` | + +Values that end up with the same name get `_2`, `_3`, ... diff --git a/scripts/docs/generate_examples.rb b/scripts/docs/generate_examples.rb index 767838f..f27f9ab 100644 --- a/scripts/docs/generate_examples.rb +++ b/scripts/docs/generate_examples.rb @@ -8,6 +8,8 @@ doc_file = ARGV.fetch(0) config = YAML.safe_load_file(File.expand_path('../../sqlc.ci.yaml', __dir__)) +ENGINE_NAMES = { 'postgresql' => 'PostgreSQL', 'mysql' => 'MySQL', 'sqlite' => 'SQLite' }.freeze + def yaml_value(value) = value.is_a?(String) && value.match?(/\A[\d.]+\z/) ? %("#{value}") : value sections = config['sql'].map do |sql| @@ -15,11 +17,12 @@ def yaml_value(value) = value.is_a?(String) && value.match?(/\A[\d.]+\z/) ? %("# driver = options['driver'] yaml = options.map { |key, value| "#{key}: #{yaml_value(value)}" }.join("\n") [ - "## Engine `#{sql['engine']}`: [examples/#{driver}](../examples/#{driver})", + "## #{ENGINE_NAMES.fetch(sql['engine'])} (`#{driver}`)", '', - "### [Schema](../#{sql['schema']}) | [Queries](../#{sql['queries']}) | [End2End Test](../tests/end2end_#{driver}.rb)", + "[Schema](../#{sql['schema']}) · [Queries](../#{sql['queries']}) · [Generated code](../examples/#{driver}) · " \ + "[End-to-end tests](../tests/end2end_#{driver}.rb)", '', - '### Config', + 'Options:', '```yaml', yaml, '```', @@ -27,4 +30,16 @@ def yaml_value(value) = value.is_a?(String) && value.match?(/\A[\d.]+\z/) ? %("# ].join("\n") end -File.write(doc_file, "# Examples\n#{sections.join("\n")}") +intro = <<~MARKDOWN + # Examples + + The same example application for each database: a schema, queries that use every supported feature, and the code + the plugin generates from them. The generated code is committed, and the end-to-end tests run against it. + + In each generated directory, `query_sql.rb` has the query methods, `models.rb` the table and enum models, + `connection.rb` the shared helpers, and the `.rbs` files their signatures + (see [The generated code](Generated-Code.md)). + +MARKDOWN + +File.write(doc_file, intro + sections.join("\n"))