diff --git a/CLAUDE.md b/CLAUDE.md index 891fa12..d8480ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ sqlc-gen-ruby: a [sqlc](https://sqlc.dev) plugin, written in pure Ruby, that generates Ruby data-access code (+ RBS) for `pg`, `mysql2` and `sqlite3`. It ships as one WASM module (the ruby.wasm interpreter with the plugin embedded). -Docs: [Architecture](docs/Architecture.md) · [Testing](docs/Testing.md) · [Releasing](docs/Releasing.md) · [Design](docs/Design.md). +Docs: [Architecture](docs/Architecture.md) · [Testing](docs/Testing.md) · [Coverage](docs/Coverage.md) · [Releasing](docs/Releasing.md). ## Commands ``` @@ -33,7 +33,7 @@ UPDATE_GOLDEN=1 bundle exec ruby -Itest test/extended_golden_test.rb # rewrite the C# sibling plugin's style. Parameter *values* are never logged. * PostgreSQL prepared statements are per connection (`ConnectionState`) because the query object is shared by pooled connections. MySQL uses client-side quoting and `Mysql2::Client#query` instead of server-side prepared statements. -* Every README claim or supported feature needs a real-database E2E test for each applicable engine, listed in +* Every documented behavior (README or `docs/`) or supported feature needs a real-database E2E test for each applicable engine, listed in `docs/Coverage.md`. A bug fix starts with an E2E test that fails before the fix; use unit tests only where E2E cannot reach the behavior (for example option validation or generation errors). * E2E coverage runs on every supported Ruby and includes full-suite option variants. Update `docs/Coverage.md` and diff --git a/README.md b/README.md index d8273de..901eb9d 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,85 @@ -[![Build](https://github.com/DaredevilOSS/sqlc-gen-ruby/actions/workflows/build.yml/badge.svg?branch=main)](https://github.com/DaredevilOSS/sqlc-gen-ruby/actions/workflows/build.yml) -[![Tests](https://github.com/DaredevilOSS/sqlc-gen-ruby/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/DaredevilOSS/sqlc-gen-ruby/actions/workflows/tests.yml) +

sqlc-gen-ruby

-# sqlc-gen-ruby -A [sqlc](https://sqlc.dev) plugin that turns your SQL into Ruby methods, with RBS type signatures. -Works with `pg` (PostgreSQL), `mysql2` (MySQL) and `sqlite3` (SQLite), and Ruby 3.1 – 4.0. +

+ Write SQL. Get typed, idiomatic Ruby.
+ A sqlc plugin for PostgreSQL, MySQL and SQLite, with RBS signatures. +

-The plugin is written in pure Ruby and ships as one self-contained WebAssembly file -(the [ruby.wasm](https://github.com/ruby/ruby.wasm) interpreter with the plugin inside), so you don't need to install anything besides sqlc. +

+ Build + Tests +

+ +--- + +[sqlc](https://sqlc.dev) reads your schema and your queries and checks them against each other. This plugin turns +the result into plain Ruby files: one method per query, one class per result row, and `.rbs` signatures for all of it. +There is no ORM and no runtime library: the generated code only needs your database gem and `connection_pool`. + +## How it works + +**1. You write SQL** + +```sql +-- schema.sql +CREATE TABLE authors ( + id BIGSERIAL PRIMARY KEY, + name text NOT NULL, + bio text +); + +-- query.sql +-- name: GetAuthor :one +-- Returns the author with the given id, or nil when there is none. +SELECT * FROM authors WHERE id = $1 LIMIT 1; + +-- name: CreateAuthor :one +INSERT INTO authors (name, bio) VALUES ($1, $2) RETURNING *; +``` + +**2. `sqlc generate` writes Ruby, and RBS to go with it** (abridged) + +```ruby +# Returns the author with the given id, or nil when there is none. +def get_author(id:) + ... +end +``` +```rbs +class GetAuthorRow + attr_reader id: ::Integer + attr_reader name: ::String + attr_reader bio: ::String? +end + +def get_author: (id: ::Integer) -> GetAuthorRow? +def get_author!: (id: ::Integer) -> GetAuthorRow +``` + +**3. You call it** + +```ruby +require_relative 'app/db/query_sql' + +queries = PgCodegen::QuerySql.connect({ dbname: 'app', host: 'localhost' }, size: 5) # a connection pool + +author = queries.create_author(name: 'Seuss', bio: nil) # => # +queries.get_author(id: author.id) # the row, or nil +queries.get_author!(id: 42) # raises QuerySql::NotFound + +queries.transaction do |tx| # COMMIT at the end, ROLLBACK if it raises + tx.create_author(name: 'a', bio: nil) +end +``` + +The complete, committed output for every database is in [`examples/`](docs/Examples.md). + +## Install + +The plugin is one WebAssembly file, so the only thing to install is +[sqlc](https://docs.sqlc.dev/en/latest/overview/install.html). Add the plugin to `sqlc.yaml`; the URL and sha256 of +every version are in its [release notes](https://github.com/DaredevilOSS/sqlc-gen-ruby/releases): -## Quick start -**1. Point sqlc at the plugin** (`sqlc.yaml`; the URL and sha256 of each version are in its [release notes](https://github.com/DaredevilOSS/sqlc-gen-ruby/releases)): ```yaml version: "2" plugins: @@ -28,288 +98,108 @@ sql: driver: pg # or mysql2 / sqlite3 ``` -**2. Write queries** (`db/query.sql`): -```sql --- name: GetAuthor :one -SELECT * FROM authors WHERE id = $1; +Then: --- name: CreateAuthor :one -INSERT INTO authors (name, bio) VALUES ($1, $2) RETURNING *; -``` +1. run `sqlc generate`; +2. add `connection_pool` and your driver gem (`pg`, `mysql2` or `sqlite3`) to your Gemfile, or set + `generateGemfile: true` and the plugin writes one; +3. `require_relative 'app/db/query_sql'`. -**3. Run `sqlc generate` and use the code:** -```ruby -require_relative 'app/db/query_sql' +## What's supported -queries = PgCodegen::QuerySql.connect({ dbname: 'app', host: 'localhost' }, size: 5) # a connection pool +Ruby 3.1 – 4.0, with PostgreSQL (`pg`), MySQL (`mysql2`) and SQLite (`sqlite3`). -author = queries.create_author(name: 'Seuss', bio: nil) # => # -queries.get_author(id: author.id) # the row, or nil -queries.get_author!(id: 42) # raises QuerySql::NotFound instead of returning nil +### Queries -queries.transaction do |tx| # COMMIT when the block ends, ROLLBACK if it raises - tx.create_author(name: 'a', bio: nil) -end -``` -More: [the generated code for every driver](docs/Examples.md) (it is the plugin's real output, from -[these schemas and queries](examples/authors)). +Every sqlc annotation works on all three databases. Details: [Queries](docs/Queries.md). + +| Annotation | The method returns | +|---|---| +| `:one` | the row or `nil`; a second method, `get_author!`, raises instead of returning `nil` | +| `:many` | an `Array` of rows | +| `:exec` | `nil` | +| `:execrows` | the number of affected rows | +| `:execlastid` | the new id | +| `:execresult` | an `ExecResult` (`rows_affected`, `last_insert_id`) | +| `:batchexec` `:batchone` `:batchmany` | runs the query once per element of an `Array`, on one connection | +| `:copyfrom` | bulk insert, all rows or none: `COPY` (PostgreSQL), `LOAD DATA` (MySQL), multi-row `INSERT` (SQLite) | + +| Macro | What it does | +|---|---| +| `sqlc.arg(name)` | names a parameter | +| `sqlc.narg(name)` | names a nullable parameter, which defaults to `nil` | +| `sqlc.slice(ids)` | `IN (...)` with an `Array` (MySQL, SQLite; on PostgreSQL use `= ANY($1::bigint[])`) | +| `sqlc.embed(table)` | puts the table's model into the row | + +### Types + +Values are what the driver gem returns, so the Ruby type of a column depends on the database. +Details and the full list: [Types](docs/Types.md). + +| SQL | PostgreSQL | MySQL | SQLite | +|---|---|---|---| +| integers | `Integer` | `Integer` | `Integer` | +| `real`, `double` | `Float` | `Float` | `Float` | +| `numeric`, `decimal` | `BigDecimal` | `BigDecimal` | `Float` | +| text, binary | `String` | `String` | `String` | +| `boolean` | `true` / `false` | `Integer` (0 / 1) | `Integer` (0 / 1) | +| `date` | `Date` | `Date` | `String` | +| `timestamp`, `datetime` | `Time` | `Time` | `String` | +| JSON | parsed Ruby objects | `String` (JSON text) | `String` (JSON text) | +| enums | `String` + constants | `String` + constants | | +| arrays | `Array[T]` | | | +| `inet`, `cidr` | `IPAddr` | | | + +Nullable columns and parameters are `T?`. The declared type of any column can be changed with +[`overrides`](docs/Options.md#overrides). + +### At runtime + +| | | +|---|---| +| **Transactions** | `transaction` commits or rolls back, nests with savepoints and takes an isolation level. [More](docs/Transactions.md) | +| **Connections** | a `connection_pool` built for you, your own pool, or a connection shared with ActiveRecord / Sequel. [More](docs/Connections.md) | +| **Concurrency** | threads, forked workers (Puma, Sidekiq) and the Fiber scheduler (`async`). [More](docs/Connections.md#forked-processes) | +| **Failures** | timeouts, killed threads, lost connections and deadlocks never leave a connection inside a transaction. [More](docs/Transactions.md#when-something-goes-wrong) | +| **Logging** | pass `logger:` to log each query's name and duration; parameter values are never logged. [More](docs/Generated-Code.md#logging) | ## Options + +Set them under `options:` of the `codegen` entry. Details: [Options](docs/Options.md). + | Option | Default | What it does | |---|---|---| -| `driver` | required | `pg`, `mysql2` or `sqlite3`. Must match the sqlc `engine:`. | -| `rubyVersion` | `3.3` | Oldest Ruby the code must run on, `3.1` – `4.0`. Rows are `Data` classes from 3.2 and `Struct`s below; `3` means any 3.x, so it gets what Ruby 3.1 needs (`Struct`s, the Timeout handling below 3.3). | -| `generateTypes` | `true` | Also generate `.rbs` type signatures. | -| `generateGemfile` | `false` | Also generate a `Gemfile` with the gems the code needs. | -| `moduleName` | `Codegen` | The module the code lives in, e.g. `MyApp::Db`. | -| `overrides` | none | Change the type of columns / parameters, see [below](#overrides). | -| `rename` | none | Rename tables (model class) and columns (attribute), e.g. `{authors: Writer, bio: biography}`; `schema.table` works too. | -| `exactTableNames` | `false` | Keep table names as model names (`authors` → `Authors`) instead of singularizing them (`Author`). | -| `preparedStatements` | `true` | `pg` only (an error with other drivers). `false` sends the SQL with every call instead of preparing it (needed behind pgbouncer in transaction mode). Statements dropped by `DEALLOCATE ALL` / `DISCARD ALL`, and statements PostgreSQL refuses after a migration changed the type of a column they return (`PG::FeatureNotSupported`: cached plan must not change result type), are prepared again by their next call outside a transaction (inside one, that call raises and the transaction rolls back; the next call prepares the statement again). Statements are named `Module::Class#method` (with a hash when longer than PostgreSQL's 63 bytes), so classes sharing a pool never run each other's SQL. | - -Unknown or misspelled options are errors, with a hint for near misses (`generate_types` → `generateTypes`). -Options shared by several `sql:` blocks can go in the top-level `options:` of `sqlc.yaml` under the plugin's name -(`options: {ruby: {moduleName: MyApp::Db}}`); a codegen entry's own options win, and the `overrides` of both apply. - -### Overrides -```yaml -overrides: - - column: "GetAuthor:bio" # one column of one query (:) - ruby_type: { type: "Symbol", notNull: true } - - column: "authors.bio" # a table column, in its model and in every query (.. too) - ruby_type: { type: "Symbol" } - - column: "*:bio" # a column name in every query and model - ruby_type: { type: "Pathname", notNull: false } - - db_type: "uuid" # a database type, everywhere (also the elements of uuid[] arrays) - ruby_type: { type: "String" } -``` -The first match wins in this order: `:`, `
.`, `*:`, `db_type`. -Overrides only change the RBS types; the values are whatever the driver returns. Unknown keys are errors. - -## The generated code -For each query file you get `query_sql.rb` with a `QuerySql` class (one method per query; the file name in CamelCase: -`admin_users.sql` → `AdminUsersSql` in `admin_users_sql.rb`), plus `models.rb` (one class -per table, one module of constants per enum), `connection.rb` (shared helpers) and the matching `.rbs` files. - -* **Arguments** are keywords named after the parameters (`get_author(id:)`); `sqlc.arg(name)` names them yourself. - Missing or unknown ones raise `ArgumentError`. Nullable parameters named with `sqlc.arg` / `sqlc.narg` default - to `nil`; other nullable parameters (`VALUES ($1, $2)`) must be passed, even as `nil`. -* **Rows** are frozen `Data` objects from Ruby 3.2 (with `with` and `to_h`) or mutable keyword `Struct`s below Ruby 3.2; - both support pattern matching: `case row in { id:, name: }`. -* **Bang finders:** every `:one` query also gets a `get_author!` that raises `QuerySql::NotFound` instead of returning `nil`. -* **Connections:** `QuerySql.connect(params, **pool_options)` builds a [`connection_pool`](https://github.com/mperham/connection_pool). - To use a pool you already have (or a fake in tests) pass it to `QuerySql.new(pool)`; anything with `#with` works. - For SQLite pass a `SQLite3::Database` (opened with any options, `results_as_hash: true` included): `QuerySql.new(db)`. -* **Logging:** `connect(params, logger: Rails.logger)` / `new(pool, logger:)` logs `QuerySql#get_author (0.4ms)` at - debug level for each query and transaction. Parameter values are never logged. -* **Async / fibers:** it is plain synchronous Ruby and works with the Fiber scheduler ([`async`](https://github.com/socketry/async)): - inside `Sync { ... }`, pooled `pg` and `mysql2` queries run concurrently (`mysql2` blocks the thread on Windows). MySQL parameters are quoted client-side - with `Mysql2::Client#escape` and sent with `Client#query`, not server-side prepared statements. SQLite queries run - one at a time. -* **Style:** SQL lives in frozen heredoc constants (`GET_AUTHOR_SQL`), comments above a query become its doc comment, - table / column / enum comments of the schema become doc comments of the models. The code is RuboCop-clean; where a - name from your schema breaks a RuboCop naming rule (`"UserId"`, `naïve`), that class or method gets a `rubocop:disable` - (on SQLite keep the query files ASCII, see [Limitations](#limitations)). Query names such as `HasAuthors` / `IsActive` - keep their name (`has_authors`) with `Naming/PredicatePrefix` disabled. - A result column without a usable name (`SELECT 1, now()`) is named by its position: `column_1`, `column_2`, ... - -### Transactions -```ruby -queries.transaction(isolation: :serializable) do |tx| - id = tx.create_author_return_id(name: 'a', bio: nil) - tx.transaction { tx.create_book(name: 'b', author_id: id) } # nested: a SAVEPOINT - :done # the block's value is returned -end -``` -* Everything in the block runs on one connection. It commits when the block ends (also on `break`, `return`, `throw`) - and rolls back when it raises or the thread is killed. An interrupt (`Timeout`, `Thread#raise`, `Thread#kill`) that - arrives while `BEGIN`, `COMMIT` or `ROLLBACK` runs waits for it to finish, and a `BEGIN` / `COMMIT` that raises - itself (a stopped `Async` task) is rolled back: the connection never goes back to the pool inside a transaction. - On PostgreSQL a query interrupted by any of them (inside or outside a transaction) is cancelled at once, so the - connection goes back to the pool idle and a killed transaction rolls back without waiting for its query. - On MySQL a `Thread#kill` during a query closes that client (mysql2 cannot send `ROLLBACK` while it waits for the - query's result), so the server rolls the transaction back and the pool replaces the client. On Ruby <3.3 with timeout <0.4, `break`, `return`, and - `throw` also roll back because Timeout cannot be told apart; add `gem 'timeout', '>= 0.4'` to the generated Gemfile - to keep those exits committing; an old `Timeout` that interrupts a PostgreSQL query in a transaction still - raises `Timeout::Error` (not the `PG::InFailedSqlTransaction` of the cancelled query) after rolling back. A nested `transaction` rolls back only its own part. With `connection_pool` or - SQLite, parent-object calls from the same thread reuse the transaction connection; injected `#with` objects must do - the same. -* `tx` is a copy of the query object bound to the transaction: its queries stay in the transaction from any fiber or - thread (an `Enumerator`, an `Async` task, a `Thread`), one at a time, and raise `TransactionEnded` once the - transaction is over. Other query objects (the parent `queries`, another generated class sharing the pool) used from - another fiber or thread check out their own connection and run outside the transaction: pass `tx` instead. -* `isolation:` (`pg`, `mysql2`) takes `:read_uncommitted`, `:read_committed`, `:repeatable_read`, `:serializable`; - `mode:` (`sqlite3`) takes `:deferred`, `:immediate`, `:exclusive`. Other values raise `ArgumentError` before anything runs, and so - do `isolation:` / `mode:` on a nested `transaction` (a savepoint cannot change them). -* PostgreSQL: if a statement fails and the block rescues the error, PostgreSQL still rolls the whole transaction - back, so `transaction` raises `PG::InFailedSqlTransaction` instead of pretending it committed. In a nested - `transaction` only the savepoint is rolled back: the outer block can rescue that error and go on. -* MySQL: a deadlock (or a lock wait timeout with `innodb_rollback_on_timeout`) rolls the whole transaction back on - the server, its savepoints too. If the block rescues that error, every later query in the transaction and - `transaction` itself raise it again (a `Mysql2::Error` with the same `error_number`, its message prefixed with - `transaction rolled back:`) instead of running in autocommit mode and reporting a commit; retry the whole - `transaction`. A lock wait timeout without that setting fails only its statement: the block can rescue it and go on. -* A connection the server closed (restart, failover, killed backend) is reconnected (`pg`) or replaced in the pool - (`mysql2`) the next time it is checked out; the call that hit the dead connection still raises. Inside `transaction` - a lost `pg` connection is not reconnected: every later call in the block and `transaction` itself raise - `PG::ConnectionBad`, so nothing after the loss runs outside the transaction (the server rolled it back). A `mysql2` - client that mysql2 closed because a running query was interrupted (an `Async` task stopped, `Thread#raise`) is - replaced the same way, without failing the next call. -* SQLite: threads and fibers sharing one `SQLite3::Database` take turns; a transaction holds the database until it ends. -* `pg`: `QuerySql.connect` also takes a connection URL: `QuerySql.connect(ENV.fetch('DATABASE_URL'), size: 5)`. -* A connection shared with another library (ActiveRecord's `raw_connection`, Sequel, your own code) keeps its - settings: the generated queries pass their own type maps (`pg`; the connection's `type_map_for_results`, - `type_map_for_queries` and `field_name_type` are left as they are) or query options (`mysql2`: rows are read with - `as: :array, cast: true, cast_booleans: false`, whatever the client's or `Mysql2::Client.default_query_options`). - On SQLite `results_as_hash` and the other settings of the `SQLite3::Database` are kept too. -* Forked processes (Puma / Sidekiq / Resque workers, `Process.fork`): connections opened before the fork stay the - parent's. connection_pool's fork handling replaces the pooled ones in the child without closing the parent's - sessions, and the child lets go of every inherited connection the generated code has used (also ones another - thread had checked out at fork time, and ones the child never uses), so its exit does not end them either. A `pg` - connection used directly (not from a `ConnectionPool`) reconnects in the child; a `mysql2` client - used that way raises `Mysql2::Error::ConnectionError` in the child (open clients in the process that uses them). - SQLite: the sqlite3 gem closes writable databases a child inherits, so open the database again in the child. - On macOS, connecting from a forked child can crash in libpq's GSSAPI probe; set `gssencmode: 'disable'` (or - `PGGSSENCMODE=disable`), as usual for forking `pg` apps there. -* A `transaction` inside a transaction that another library opened on the same connection (`BEGIN`) is a savepoint - on PostgreSQL and SQLite. MySQL has no nested transactions: its `BEGIN` commits the open transaction first (see - [Limitations](#limitations)). - -## Supported features -### Query annotations -| Annotation | Returns | Notes | -|---|---|---| -| `:one` | the row or `nil` | plus a bang finder (`get_author!`) | -| `:many` | `Array` of rows | | -| `:exec` | `nil` | | -| `:execrows` | number of affected rows | MySQL: rows changed, not rows matched; connect with `flags: Mysql2::Client::FOUND_ROWS` to count matched rows | -| `:execlastid` | the new id | PostgreSQL / SQLite: needs `RETURNING id`; MySQL: the auto-increment id | -| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | `last_insert_id` is `nil` on PostgreSQL (use `RETURNING`) | -| `:batchexec` / `:batchone` / `:batchmany` | `nil` / `Array` of rows or `nil` / `Array` of `Array`s of rows | take an `Array` of `Args` and run the query once per element, on one connection | -| `:copyfrom` | `nil` | bulk insert of an `Array` (or any `Enumerable`, also lazy) of `Args`; all rows or none (see below) | - -`:copyfrom` uses `COPY ... FROM STDIN` on PostgreSQL and batched multi-row `INSERT`s on SQLite. On MySQL it uses -`LOAD DATA LOCAL INFILE`, which needs `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the -server; duplicate keys and invalid values raise `Mysql2::Error` instead of being skipped. Conflict clauses are kept: -`INSERT OR REPLACE` / `OR IGNORE` / `ON CONFLICT ... DO ...` on SQLite, `INSERT IGNORE` (duplicates skipped) and -`REPLACE` on MySQL. Clauses `COPY` / `LOAD DATA` cannot express (`ON CONFLICT` / `RETURNING` on PostgreSQL, -`ON DUPLICATE KEY UPDATE` on MySQL) are generation errors; use `:batchexec` for those. -* MySQL: `LOAD DATA` names the table and columns as the `INSERT` writes them (`` `MixedCase` ``): sqlc reports MySQL - names in lowercase, but table names are case-sensitive on Linux. -* MySQL: `LOAD DATA` commits an open transaction that was started with raw SQL (`BEGIN`); call `:copyfrom` inside - `transaction` instead. - -### Macros -| Macro | PostgreSQL | MySQL | SQLite | Notes | -|---|:---:|:---:|:---:|---| -| `sqlc.arg` | ✅ | ✅ | ✅ | names a parameter (unnamed ones such as `ANY($1::BIGINT[])` become `param1`, ...) | -| `sqlc.narg` | ✅ | ✅ | ✅ | a nullable parameter (`T?`, defaults to `nil`) | -| `sqlc.slice` | 🚫 | ✅ | ✅ | `IN (sqlc.slice(ids))` takes an `Array` (an empty one: `IN` matches nothing, `NOT IN` everything). PostgreSQL: use `= ANY($1::bigint[])` with an array | -| `sqlc.embed` | ✅ | ✅ | ✅ | the row gets an attribute holding the table's model (`author`, `author2` for the same table twice); `nil` when an outer join found no row | - -🚫 = not offered by sqlc for that database. +| `driver` | required | `pg`, `mysql2` or `sqlite3`; must match the sqlc `engine:` | +| `rubyVersion` | `3.3` | the oldest Ruby the code must run on (`3.1` – `4.0`) | +| `generateTypes` | `true` | also generate `.rbs` signatures | +| `generateGemfile` | `false` | also generate a `Gemfile` | +| `moduleName` | `Codegen` | the module the code lives in, e.g. `MyApp::Db` | +| `overrides` | none | change the declared Ruby type of columns and parameters | +| `rename` | none | rename tables and columns, e.g. `{authors: Writer, bio: biography}` | +| `exactTableNames` | `false` | name the model `Authors` instead of the singular `Author` | +| `preparedStatements` | `true` | `pg` only; set `false` behind pgbouncer in transaction mode | -### Types -* Types follow what the drivers return: `Integer`, `Float`, `BigDecimal` (`numeric` / `decimal`), `String`, `Date`, `Time`, - `IPAddr`, `true` / `false`; PostgreSQL `json` / `jsonb` are parsed (`untyped`; MySQL `JSON` columns and parameters are JSON text `String`s, as - `mysql2` returns them); nullable columns and parameters are `T?`. -* PostgreSQL arrays are `Array[T]` (`Array[Array[T]]` for `T[][]`); arrays of types the pg gem does not decode - (`uuid[]`, enum arrays, ...) are `Array[String]`, also for types created after the connection was opened (a - migration run while the application is up). Geometric, range, full-text, `hstore`, `ltree`, `timetz`, `reg*` and - composite types are `String`s; domains and unknown types are `untyped`. -* MySQL: a `Time` / `DateTime` parameter (also in `:copyfrom`) keeps its instant whatever its UTC offset: it is written - in the zone `mysql2` reads `DATETIME` / `TIMESTAMP` values back in, the process's local zone, or UTC for a client - created with `database_timezone: :utc` (`connect(params.merge(database_timezone: :utc))`). The server also converts - `TIMESTAMP` columns with the session `time_zone`, so keep that equal to the client's zone. -* PostgreSQL: a `Time` or `DateTime` parameter of a `timestamp` (without time zone) column (also in `:copyfrom` and - batches) keeps its instant whatever its UTC offset: it is written in the process's local zone, the zone the pg gem - reads `timestamp` values back in (PostgreSQL itself ignores the offset of a `timestamp` value). A `DateTime` of a - `timestamptz` column keeps its instant too (the pg gem alone would send its date only); the RBS still declares - `::Time`, the type the rows return. Elements of `timestamp[]` arrays are written as given. -* A column without a time zone (PostgreSQL `timestamp`, MySQL `DATETIME`) stores the local wall-clock time, so an - instant in the hour repeated when the local zone falls back from daylight saving time (e.g. 02:30 on the last - Sunday of October in Europe/Berlin happens twice) reads back as the first of the two, one hour off for the second. - Use `timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), to keep every instant. -* A `String` passed to a PostgreSQL `json` / `jsonb` parameter is JSON text (`'{"a":1}'`): pass `'"hi"'` or - `'hi'.to_json` for a JSON string. Hashes, Arrays, numbers, `true` / `false` and `nil` are encoded for you. -* Enums are `String`s, with constants in `models.rb` (`ExtendedBioType::MEMOIR`, `ExtendedBioType::ALL`). Values that - are not valid constant names still get one: `'end'` → `END_`, `'all'` → `ALL_`, `'2x'` → `VALUE_2X`, `'a b'` → `A_B`; - values that end up with the same name get `_2`, `_3`, ... -* SQLite follows its [type affinity rules](https://www.sqlite.org/datatype3.html#determination_of_column_affinity) - (`VARCHAR(255)` → `String`, `DECIMAL(10,5)` → `Float`, `UNSIGNED BIG INT` → `Integer`); `JSON` and `UUID` columns are `String`s, - also when SQLite stored numeric-looking text as a number (`'42'` → `"42"`, in SQLite's spelling: `'1.50'` → `"1.5"`). - The generated SQLite code uses `Monitor`: if you run Steep, add `library "monitor"` to your `Steepfile`. Code - generated for a `rubyVersion` below 3.3 also uses `Timeout` (see [Transactions](#transactions)): add `library "timeout"` too. - -### Names -* Query methods and keyword arguments are snake_cased, with acronyms kept together (`MIN_ID` → `min_id`, - `HTTPServer` → `http_server`); Ruby keywords get a trailing `_` (`end_:`). Row members use the column spelling - reported by sqlc for the engine, except names that conflict with row methods, including the private ones `Data` / `Struct` call while building and copying rows (`class_`, `hash_`, `initialize_`): - PostgreSQL and SQLite keep `"camelCase"` as `:camelCase`, while MySQL reports it as `:camelcase`. A column named - `end` is the row member `:end` but the argument `end_:`. `Struct` rows (`rubyVersion` below 3.2) have more methods - than `Data` rows, so more names get the `_`: a column `count` is `:count` from 3.2 but `:count_` below - (`Struct#count` would otherwise be `Enumerable#count`); check the generated `.rbs` when changing `rubyVersion`. -* A parameter used twice is one keyword when named with `sqlc.arg`; repeated `?` / `$n` of one column become `id:`, `id_2:`. -* Models may be named like Ruby or driver constants (tables `exceptions`, `data`, `threads`, `sets`, ...): the - generated code and `.rbs` refer to those as `::Exception`, `::Data`, `::Thread`, ... so the models cannot shadow them. -* SQLite: sqlc before 1.30 (including the pinned 1.27) reports a quoted table name with its quotes, so - `CREATE TABLE "Items"` gives the model `Items` there and `Item` with sqlc 1.30 and later (PostgreSQL / MySQL give - `Item` with any version). Use `rename` or `exactTableNames` to keep the name when upgrading sqlc. -* A table `authors` next to a table `author` keeps its plural model name (`Authors`). Query names that would replace - a generated method (`Transaction`, `Initialize`, ...) are errors. - -### Limitations -* Two result columns with the same name (e.g. `a.id, b.id`) are an error: alias one (`b.id AS book_id`) or use `sqlc.embed`. -* SQLite's `NUMERIC` affinity can return an `Integer` where `Float` is declared. -* SQLite: a statement takes at most 32,766 parameters (`SQLITE_MAX_VARIABLE_NUMBER`), so a `sqlc.slice` with more - elements raises `SQLite3::SQLException: too many SQL variables`; on MySQL the limit is the server's - `max_allowed_packet`. Load larger sets into a table (`:copyfrom`) and join it instead. -* `:batchexec` / `:batchone` / `:batchmany` are not atomic on their own: wrap them in `transaction` for all-or-nothing. -* MySQL: `transaction` on a client inside a transaction that was started with raw SQL or by another library commits - that transaction first (MySQL's `BEGIN` does), so a later `ROLLBACK` no longer undoes it. Run the outer work in - `transaction` too (inner calls become savepoints), or finish the other transaction first. -* SQLite: sqlc does not see parameters in `ON CONFLICT ... DO UPDATE SET` (it leaves `sqlc.arg(...)` in the SQL or - reports fewer parameters than there are `?`), so such queries are generation errors. Use `excluded.` - (`DO UPDATE SET n = counters.n + excluded.n`) or move the value into `VALUES`. -* SQLite: `busy_timeout=` makes the sqlite3 gem wait for a locked database while holding Ruby's GVL, so other threads - and fibers stop too; `busy_handler_timeout=` waits without blocking them. -* SQLite transactions belong to the thread that opened them; fibers sharing the same database take turns, and a - transaction holds it until it ends. Only the yielded `tx` runs inside the transaction from other fibers / threads. -* SQLite: sqlc (1.27 – 1.31 at least) misplaces the SQL of a query file that contains a non-ASCII character (in a - name, literal or comment): that query and the following ones lose characters at the end (`FROM user` instead of - `FROM users`). This is a sqlc bug, not a plugin one; keep SQLite query files ASCII and pass such values as parameters. -* MySQL: sqlc lowercases the table name that `sqlc.embed` expands to (`sqlc.embed(MixedItems)` becomes - `SELECT mixeditems.itemid, ...`). MySQL table names are case-sensitive on Linux, so a mixed-case table fails with - `Unknown column 'mixeditems.itemid'`. This is a sqlc limitation the plugin cannot undo (the request only carries the - lowercased name); give the table an alias and embed that: `` SELECT sqlc.embed(m) FROM `MixedItems` AS m ``. -* Not applicable to Ruby (they exist in the C# plugin): `withAsyncSuffix` / `withCancellationToken` (use the Fiber - scheduler), `useDapper`, target framework / csproj options (`moduleName` replaces the namespace). - -# Contributing -You need Ruby 3.3 (`bundle install`), [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html) (version in -[`scripts/tool-versions.env`](scripts/tool-versions.env)), `curl` + `tar` to build the WASM, and PostgreSQL / MySQL -servers (or Docker) for the end-to-end tests. +## Documentation -``` -bundle exec rake test # unit + golden tests (no database needed) -bundle exec rake rubocop steep # the generated code must stay RuboCop- and Steep-clean -rake sqlc_generate_process # regenerate examples/ with exe/plugin.rb -rake sqlc_generate_wasm # build dist/plugin.wasm, then regenerate examples/ with it -rake end2end_sqlite3 # end-to-end, no server needed -rake end2end_pg end2end_mysql2 # end-to-end, needs servers (see docs/Testing.md) -rake run_end2end_tests # all of it in Docker (docker-compose) -``` -`examples/` is the plugin's committed output and CI fails when it drifts, so regenerate it with every change to the -generated code. Pull requests need both CI workflows (`build.yml`, `tests.yml`) green. - -Releases are started by a maintainer (**Actions → Release → Run workflow**) and are automated from there: the full -test gate, the tag, the GitHub release with the tested `sqlc-gen-ruby.wasm` and its sha256, and a check of the published file. +**Using the plugin** -| Doc | What | +| | | +|---|---| +| [Generated code](docs/Generated-Code.md) | the files, classes and methods you get, and how they are named | +| [Queries](docs/Queries.md) | annotations, `:copyfrom`, macros, arguments | +| [Types](docs/Types.md) | the Ruby type of every SQL type, time zones, enums | +| [Connections](docs/Connections.md) | pools, shared connections, forked workers, fibers | +| [Transactions](docs/Transactions.md) | commit and rollback, savepoints, isolation, what happens on errors | +| [Options](docs/Options.md) | every option, `overrides` | +| [Troubleshooting](docs/Troubleshooting.md) | known errors and surprises per database, with their fixes | +| [Examples](docs/Examples.md) | schemas, queries and generated output for each database | + +**Working on the plugin** + +| | | |---|---| -| [docs/Architecture.md](docs/Architecture.md) | how the generator works, where to change what, adding a feature / driver / option | -| [docs/Testing.md](docs/Testing.md) | the test layers, CI workflows, what to run before pushing | -| [docs/Releasing.md](docs/Releasing.md) | the release process and versioning rules | -| [docs/Design.md](docs/Design.md) | the WASM packaging and design decisions | -| [docs/Examples.md](docs/Examples.md) | per-driver configs and links to the generated examples | -| [CLAUDE.md](CLAUDE.md) | the same orientation condensed for AI coding assistants | +| [Contributing](docs/Contributing.md) | setup, commands, pull requests | +| [Architecture](docs/Architecture.md) | how the generator works and how it is packed into WebAssembly | +| [Testing](docs/Testing.md) | test layers, CI, what to run before pushing | +| [Coverage](docs/Coverage.md) | every documented behavior and the test that proves it | +| [Releasing](docs/Releasing.md) | the release workflow and versioning | diff --git a/docs/Architecture.md b/docs/Architecture.md index 721f829..73e01bd 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -1,8 +1,18 @@ -# Architecture: how the generator works +# Architecture -A sqlc plugin is a program that gets a protobuf `GenerateRequest` on stdin (the parsed schema, the queries, the -plugin's `options:`) and writes a `GenerateResponse` (a list of files) to stdout. This plugin does exactly that in -pure Ruby; [Design.md](Design.md) explains how it is packed into a WASM module. +How the generator works, why it is built this way, and where to change what. + +## What a sqlc plugin is + +sqlc parses the schema and the queries, then hands the plugin a protobuf `GenerateRequest` on stdin (tables, enums, +queries with their typed parameters and columns, and the plugin's `options:`). The plugin answers on stdout with a +`GenerateResponse`: a list of files. sqlc can run a plugin as a process or as a WASI module. + +This plugin is pure Ruby. It runs as a process during development (`exe/plugin.rb`) and is released as one WASI module: +the [ruby.wasm](https://github.com/ruby/ruby.wasm) interpreter with the plugin inside +([Packaging](#packaging-as-webassembly)). + +## The pipeline ``` stdin ─► Proto (decode) ─► Generator ─┬─► Options / RubyVersion what the user asked for @@ -14,78 +24,123 @@ stdin ─► Proto (decode) ─► Generator ─┬─► Options / RubyVersion stdout ◄─ Proto (encode) ◄─ GenerateResponse (query_sql.rb, query_sql.rbs, models.rb, connection.rb, Gemfile, ...) ``` -## The pieces +Everything is in `lib/sqlc_gen_ruby/`: -| Piece | Responsibility | +| File | Responsibility | |---|---| | `exe/plugin.rb` | entry point. Errors go to stderr with exit code 1 (sqlc shows them); `SQLC_GEN_RUBY_DEBUG=1` adds a backtrace | -| `proto.rb`, `plugin_pb.rb` | a ~150 line protobuf wire codec and the subset of sqlc's `codegen.proto` we read. Field numbers must match upstream. `google-protobuf` is a C extension and cannot run inside ruby.wasm | -| `options.rb`, `ruby_version.rb` | parse + validate the `options:` JSON (`driver`, `rubyVersion`, `overrides`, `rename`, ...; unknown keys are errors) | -| `generator.rb` | the orchestrator: one file per query file (`query_sql.rb`), plus `models.rb`, `connection.rb` (per-connection state and helpers the query classes share), `Gemfile`; builds the module / class / `Data` skeleton | -| `method_gen.rb` | the code of one query: signature (keyword arguments), SQL constant, command (`:one`, `:many`, `:exec`, `:execrows`, `:execlastid`, `:execresult`, `:batchexec` / `:batchone` / `:batchmany`, `:copyfrom`), `sqlc.slice`, bang finder, and the RBS signature | +| `proto.rb`, `plugin_pb.rb` | a small protobuf wire codec and the subset of sqlc's `codegen.proto` the plugin reads. Field numbers must match upstream | +| `options.rb`, `ruby_version.rb` | parse and validate the `options:` JSON; unknown keys are errors | +| `generator.rb` | the orchestrator: one file per query file, plus `models.rb`, `connection.rb` and `Gemfile`; builds the module / class / row skeleton | +| `method_gen.rb` | the code of one query: signature, SQL constant, the annotation's behavior, `sqlc.slice`, the bang method, and the RBS signature | | `row_layout.rb` | turns result columns into row attributes; `sqlc.embed` collapses a table's columns into one model attribute | -| `type_resolver.rb` | RBS type of a column / parameter: driver type table, arrays, nullability (`T?`), `overrides:` (a `db_type` override types array elements too) | +| `type_resolver.rb` | the RBS type of a column or parameter: driver type table, arrays, nullability, `overrides` | | `catalog.rb` | the schema's tables and enums, and their Ruby names | -| `drivers/base.rb` | the driver interface and what all drivers share: `instrument` logging, `NotFound`, transaction RBS | -| `drivers/{pg,mysql2,sqlite3}.rb` | the per-library parts: type mapping, connecting / pooling, executing statements, transactions, `:copyfrom` | -| `code.rb` | text helpers: indentation, `snake_case`, `camelize`, Ruby-keyword-safe parameter names, `rubocop:disable` pairs for schema names that break a naming cop | +| `drivers/base.rb` | the driver interface and what all drivers share: logging, `NotFound`, transaction RBS | +| `drivers/{pg,mysql2,sqlite3}.rb` | the per-library parts: type mapping, connecting and pooling, executing statements, transactions, `:copyfrom` | +| `code.rb` | text helpers: indentation, `snake_case`, `camelize`, keyword-safe names, `rubocop:disable` pairs | + +**The one rule:** `MethodGen` and `Generator` decide *what* is generated and are the same for every database. A driver +only supplies *snippets* (how to prepare a statement, how to read the affected rows, ...) through the hooks of +`Drivers::Base`. A `case driver` in `MethodGen` belongs in a driver hook instead. -**Rule of thumb:** `MethodGen` / `Generator` decide *what* is generated and are the same for every database; a driver only -supplies *snippets* (how to prepare a statement, how to read the affected rows, ...). If you find a `case driver` in -`MethodGen`, it belongs in a driver hook instead. +## The generated code -## What the generated code looks like +Abridged from [`examples/pg/query_sql.rb`](../examples/pg/query_sql.rb), the plugin's real output: ```ruby module PgCodegen - class QuerySql - GET_AUTHOR_SQL = <<~'SQL' # raw heredoc: the SQL is never interpolated - SELECT id, name, bio FROM authors WHERE id = $1 LIMIT 1 - SQL - GetAuthorRow = Data.define(:id, :name, :bio) # Struct.new(...) below Ruby 3.2 + GET_AUTHOR_SQL = <<~'SQL' # raw heredoc: the SQL is never interpolated + SELECT id, name, bio FROM authors + WHERE id = $1 LIMIT 1 + SQL + class GetAuthorRow < ::Data.define(:id, :name, :bio) # a keyword Struct below Ruby 3.2 + end + + class QuerySql def get_author(id:) # a :one query returns the row or nil ... with_client('get_author') do |client| # one pooled connection + logging - ...prepare once per connection, exec_prepared... + query_params = [id] + result = run_prepared(client, 'PgCodegen::QuerySql#get_author', GET_AUTHOR_SQL, query_params) + row = result.first + row && GetAuthorRow.new(row['id'], row['name'], row['bio']) end end - def get_author!(id:) # ... and gets a bang variant - get_author(id: id) || raise(NotFound, 'GetAuthor returned no row') + def get_author!(...) # ... and gets a bang variant + get_author(...) || raise(NotFound, 'GetAuthor returned no row') end - - def transaction(isolation: nil) { |tx| ... } # commit / rollback, nested = SAVEPOINT end end ``` -Look at [`examples/pg/query_sql.rb`](../examples/pg/query_sql.rb) for a complete file; it is the plugin's real output. -Design decisions worth knowing (the *why* is in the comments of `drivers/base.rb` and `drivers/pg.rb`): +## Design decisions + +The *why* behind most of these is in the comments of `drivers/base.rb` and `drivers/pg.rb`. -* **PostgreSQL prepared statements are per connection.** A pooled query object is shared by many connections, and a +* **Idiomatic Ruby, not a port.** The plugin started as a port of + [sqlc-gen-csharp](https://github.com/DaredevilOSS/sqlc-gen-csharp). Its output now follows Ruby conventions instead: + keyword arguments rather than parameter objects, `Data` rows, implicit returns, bang methods, 2-space indentation. +* **Nothing from the plugin leaks into the application.** The generated code only requires the driver gem and + `connection_pool`. +* **A thin layer over the driver.** Values are never converted: `overrides` only change the RBS types. +* **Pool injection.** `QuerySql.new(pool)` accepts anything that responds to `#with`, so an application can share its + pool and tests can pass a fake. `connect(...)` is only a convenience that builds a `connection_pool`. +* **PostgreSQL prepared statements are per connection.** A query object is shared by many pooled connections, and a prepared statement exists only on the connection that prepared it, so each connection remembers its own - (`ConnectionState`). MySQL quotes values client-side and uses `Mysql2::Client#query` so queries cooperate with the - Fiber scheduler. -* **Pool injection.** `QuerySql.new(pool)` accepts anything that responds to `#with`, so an app can share its pool - and tests can pass a fake. `connect(...)` is only a convenience that builds a `connection_pool`. -* **SQLite statements are closed.** They are closed in `ensure` because an open statement blocks `COMMIT`. -* **Parameters are encoded by the driver.** `Driver#encode_param` (e.g. `bytea`, `json` and arrays on `pg`) is used by - every command, batches included; add a type's encoding there, not in `MethodGen`. -* **`:copyfrom` is all or nothing** on every driver (a savepoint / transaction around the load), and MySQL's + (`ConnectionState`, a module mixed into the connection). Statements are named per module and query class, so several + query objects can share a pool. +* **MySQL quotes parameters client-side** and uses `Mysql2::Client#query`, not server-side prepared statements, so + queries cooperate with the Fiber scheduler. +* **SQLite queries do not overlap.** A `SQLite3::Database` is one connection, so the generated code takes one lock per + database (held for a whole transaction) to keep threads and fibers from mixing their transactions. Statements are + closed in `ensure`, because an open statement blocks `COMMIT`. +* **Parameters are encoded by the driver.** `Driver#encode_param` (`bytea`, `json` and arrays on `pg`) is used by every + annotation, batches included. Add a type's encoding there, not in `MethodGen`. +* **`:copyfrom` is all or nothing** on every driver (a savepoint or transaction around the load), and MySQL's `LOAD DATA` warnings (skipped duplicates, coerced values) are raised as errors. -* **Nothing from the plugin leaks into the app.** The generated code only requires the driver gem and `connection_pool`. + +## Packaging as WebAssembly + +sqlc gives a WASI plugin no filesystem, no network and no way to install gems. That shapes the plugin: + +* `lib/` and `exe/` use only Ruby core and `json`. +* `google-protobuf` is a C extension and cannot be loaded inside ruby.wasm, so the plugin has its own codec + (`proto.rb`). + +`scripts/wasm/build.sh` builds `dist/plugin.wasm`: + +1. download the *prebuilt* `ruby-3.3-wasm32-unknown-wasip1-full` release of ruby.wasm (no need to compile CRuby); +2. strip the ~18 MB of DWARF debug info and names from the interpreter (`strip_wasm.rb`); +3. stage a slim stdlib, `lib/` and `exe/`, plus an empty `syntax_suggest/core_ext.rb` (the interpreter loads it to + format errors and would otherwise print a warning before the plugin's own error message); +4. embed that filesystem into the interpreter with `rbwasm pack` (wasi-vfs). + +**The argv trick:** sqlc runs the module as `plugin.wasm /plugin.CodegenService/Generate`, and the interpreter takes +that path as its script. The build embeds a file at exactly that path containing `load '/app/exe/plugin.rb'`. + +Tool versions (sqlc, ruby.wasm) are pinned in [`scripts/tool-versions.env`](../scripts/tool-versions.env). ## Adding things -**A new query feature** (an annotation or macro): -1. read what sqlc sends in a real request: `scripts/capture_request.rb` saves one to `test/fixtures/` (see [Testing](Testing.md)); -2. handle it in `MethodGen` (add a driver hook in `Base` if the code differs per database); -3. add a query using it to `examples/authors/*/{schema,query}.sql` for every driver that can run it; -4. `rake sqlc_generate_process`, review the diff of `examples/`, update the golden files, add unit and end-to-end tests. +**A query feature** (an annotation or macro): + +1. look at what sqlc sends: `scripts/capture_request.rb` saves a real request to `test/fixtures/`; +2. handle it in `MethodGen`, with a hook in `Drivers::Base` if the code differs per database; +3. add a query that uses it to `examples/authors/*/{schema,query}.sql` for every database that can run it; +4. run `rake sqlc_generate_process`, read the diff of `examples/`, update the golden files; +5. add end-to-end tests and list them in [Coverage](Coverage.md) ([Testing](Testing.md)); document the feature. + +**An option:** + +1. parse and default it in `options.rb`, and use it where needed; +2. document it in [Options](Options.md) and the README option table; +3. add a case to `test/options_test.rb` and, if it changes the output, to the `codegen-options` matrix in `build.yml`. -**A new driver** (e.g. `trilogy`): subclass `Drivers::Base`, implement the abstract hooks (the abstract ones raise -`NotImplementedError`), register it in `Generator#instantiate_driver` and `Options::DRIVERS`, add `examples/authors/` + an end-to-end test file `tests/end2end_.rb`, and a service -container job in `.github/workflows/tests.yml`. +**A driver** (for example `trilogy`): -**A new option:** `options.rb` (parse + default), use it where needed, document it in the README option table, and -add a case to `test/options_test.rb` and (if it changes output) to the `codegen-options` matrix in `build.yml`. +1. subclass `Drivers::Base` and implement its hooks (the required ones raise `NotImplementedError`); +2. register it in `Generator#instantiate_driver` and `Options::DRIVERS`; +3. add `examples/authors//`, an end-to-end suite `tests/end2end_.rb` and its job in + `.github/workflows/tests.yml`. diff --git a/docs/Connections.md b/docs/Connections.md new file mode 100644 index 0000000..6ec5f3d --- /dev/null +++ b/docs/Connections.md @@ -0,0 +1,89 @@ +# Connections + +A query object runs its queries on connections it takes from a pool. It is safe to create once and share between +threads. + +## Getting a query object + +**Let the plugin build the pool** (`pg`, `mysql2`). `connect` takes the driver's connection parameters and passes the +other options to [`connection_pool`](https://github.com/mperham/connection_pool): + +```ruby +queries = PgCodegen::QuerySql.connect({ dbname: 'app', host: 'localhost' }, size: 5, timeout: 5) +queries = PgCodegen::QuerySql.connect(ENV.fetch('DATABASE_URL'), size: 5) # pg also takes a URL +queries = Mysql2Codegen::QuerySql.connect({ database: 'app', host: 'localhost' }, size: 5) +``` + +**Bring your own pool.** Anything that responds to `#with` and yields a connection works, including a fake in tests: + +```ruby +queries = PgCodegen::QuerySql.new(pool) +``` + +**SQLite** has no `connect`: pass a `SQLite3::Database`, opened with any options (`results_as_hash: true` included). + +```ruby +queries = Sqlite3Codegen::QuerySql.new(SQLite3::Database.new('app.db')) +``` + +A `SQLite3::Database` is one connection, so threads and fibers sharing it take turns. + +Both `connect` and `new` take `logger:`, see [Logging](Generated-Code.md#logging). + +## Sharing a connection with another library + +A connection that ActiveRecord (`raw_connection`), Sequel or your own code also uses keeps its settings. The generated +queries pass what they need with every call instead of changing the connection: + +| Driver | Left untouched | What the generated queries use | +|---|---|---| +| `pg` | `type_map_for_results`, `type_map_for_queries`, `field_name_type` | their own type maps | +| `mysql2` | the client's query options and `Mysql2::Client.default_query_options` | `as: :array, cast: true, cast_booleans: false` | +| `sqlite3` | `results_as_hash` and the other settings of the `SQLite3::Database` | | + +For a transaction the other library opened on the same connection, see +[Transactions](Transactions.md#inside-another-librarys-transaction). + +## Forked processes + +Puma, Sidekiq and Resque workers, and anything else that calls `Process.fork`, can inherit a query object: + +* **Pooled connections:** connections opened before the fork stay the parent's. `connection_pool` replaces them in the + child, and the child lets go of every inherited connection without closing the parent's session, also ones another + thread had checked out at fork time and ones the child never uses. A child that exits does not end them either. +* **A `pg` connection used directly** (not from a `ConnectionPool`) reconnects in the child. +* **A `mysql2` client used directly** raises `Mysql2::Error::ConnectionError` in the child: open clients in the + process that uses them. +* **SQLite:** the sqlite3 gem closes writable databases a child inherits, so open the database again in the child. +* **macOS:** connecting from a forked child can crash in libpq's GSSAPI probe. Set `gssencmode: 'disable'` (or + `PGGSSENCMODE=disable`), as usual for forking `pg` applications there. + +## Fibers and Async + +The generated code is plain synchronous Ruby and works with the Fiber scheduler +([`async`](https://github.com/socketry/async)). Inside `Sync { ... }`: + +| Driver | | +|---|---| +| `pg` | pooled queries run concurrently | +| `mysql2` | pooled queries run concurrently (on Windows `mysql2` blocks the thread) | +| `sqlite3` | queries run one at a time | + +## Lost connections + +A connection the server closed (restart, failover, killed backend) is reconnected (`pg`) or replaced in the pool +(`mysql2`) the next time it is checked out. The call that hit the dead connection still raises. + +A `mysql2` client that mysql2 closed because a running query was interrupted (a stopped `Async` task, `Thread#raise`) +is replaced the same way, without failing the next call. + +Inside a transaction a lost connection is never replaced silently: see +[Transactions](Transactions.md#lost-connections). + +## Prepared statements + +* **PostgreSQL** prepares each query once per connection. Behind pgbouncer in transaction mode, turn that off with + [`preparedStatements: false`](Options.md#preparedstatements-pg). +* **MySQL** does not use server-side prepared statements: parameters are quoted client-side with + `Mysql2::Client#escape` and sent with `Client#query`. +* **SQLite** prepares and closes a statement for each call. diff --git a/docs/Contributing.md b/docs/Contributing.md new file mode 100644 index 0000000..5f310ee --- /dev/null +++ b/docs/Contributing.md @@ -0,0 +1,44 @@ +# Contributing + +Contributions are welcome. This page gets you set up; the other contributor docs go deeper. + +| Doc | What | +|---|---| +| [Architecture](Architecture.md) | how the generator works, the WASM packaging, adding a feature / option / driver | +| [Testing](Testing.md) | the test layers, CI workflows, what to run before pushing | +| [Coverage](Coverage.md) | every documented behavior mapped to its end-to-end test | +| [Releasing](Releasing.md) | the release workflow and versioning rules | +| [CLAUDE.md](../CLAUDE.md) | the same orientation, condensed for AI coding assistants | + +## Setup + +You need: + +* Ruby 3.3 and `bundle install`; +* [sqlc](https://docs.sqlc.dev/en/latest/overview/install.html), in the version pinned in + [`scripts/tool-versions.env`](../scripts/tool-versions.env); +* `curl` and `tar` to build the WASM; +* PostgreSQL and MySQL servers, or Docker, for the end-to-end tests. + +## Commands + +``` +bundle exec rake test # unit + golden tests (no database needed) +bundle exec rake rubocop steep # the generated code must stay RuboCop- and Steep-clean +rake sqlc_generate_process # regenerate examples/ with exe/plugin.rb +rake sqlc_generate_wasm # build dist/plugin.wasm, then regenerate examples/ with it +rake end2end_sqlite3 # end-to-end, no server needed +rake end2end_pg end2end_mysql2 # end-to-end, needs servers (see Testing.md) +rake run_end2end_tests # end-to-end in Docker (docker-compose) +``` + +## Making a change + +* **Generated output:** `examples/` is the plugin's committed output and CI fails when it drifts. When a change alters + the generated code, regenerate `examples/` and the golden files in the same commit, and read their diff. +* **Tests:** new behavior and bug fixes come with an end-to-end test and a row in [Coverage](Coverage.md). +* **Docs:** user-facing behavior is documented in `docs/`; the README stays a short overview that links there. +* **Pull requests** target `main`, need both CI workflows (`build.yml`, `tests.yml`) green, and get a label: `bug`, + `enhancement` or `documentation` (`breaking` for incompatible changes). + +Releases are started by a maintainer and automated from there: see [Releasing](Releasing.md). diff --git a/docs/Coverage.md b/docs/Coverage.md index 397ebb7..6927748 100644 --- a/docs/Coverage.md +++ b/docs/Coverage.md @@ -1,60 +1,110 @@ -# Supported feature coverage +# Coverage -Every README behavior is linked to the real-database test that exercises it for each applicable engine. `unit:` is used -only when the behavior is rejected or decided during generation and generated code cannot reach it. +Every behavior documented in the README and `docs/` is listed here with the test that proves it, for each database it +applies to. See [Testing](Testing.md) for the rules. -| README claim / feature | PostgreSQL | MySQL | SQLite | Unit-only checks | +* A cell links to the end-to-end test in `tests/` that runs against the real database. +* *Unit-only checks* are used only when the behavior is decided or rejected during generation, where the generated + code cannot reach it. +* `test/coverage_map_test.rb` checks that every linked test exists. + +## Options and generated files + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | |---|---|---|---|---| | Options: driver, Ruby target, types, Gemfile, module name, renames, exact names, overrides and global/local merge; CamelCase query file classes (`admin_users.sql` → `AdminUsersSql`); distinct prepared statements for modules sharing a pool (also names over 63 bytes) | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | unit: MySQL-specific `preparedStatements` is rejected | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | [`test_unknown_options_are_rejected_with_a_hint`](../test/options_test.rb#test_unknown_options_are_rejected_with_a_hint), [`test_override_forms`](../test/options_test.rb#test_override_forms), [`test_immutable_data_support`](../test/options_test.rb#test_immutable_data_support), [`test_omitted_parts_mean_the_oldest_covered_release`](../test/options_test.rb#test_omitted_parts_mean_the_oldest_covered_release), [`test_version_decisions_agree_for_every_form`](../test/options_test.rb#test_version_decisions_agree_for_every_form), [`test_major_only_version_orders_as_its_oldest_release`](../test/options_test.rb#test_major_only_version_orders_as_its_oldest_release), [`test_legacy_timeout_guard_is_emitted_only_for_pre_33_targets`](../test/features_test.rb#test_legacy_timeout_guard_is_emitted_only_for_pre_33_targets), [`test_query_file_names_become_camel_case_classes`](../test/features_test.rb#test_query_file_names_become_camel_case_classes), [`test_statement_names_of_different_classes_never_collide`](../test/features_test.rb#test_statement_names_of_different_classes_never_collide) | | `preparedStatements` disabled and configured `moduleName` | [`test_flow`](../tests/end2end_pg.rb#test_flow) (Ruby 4.0 variant) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow) (Ruby 4.0 variant) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow) (Ruby 4.0 variant) | [`test_pg_can_skip_prepared_statements`](../test/features_test.rb#test_pg_can_skip_prepared_statements) | | Generated query/model/connection files and generated `.rbs`; runtime return/attribute checks | [`test_models_are_generated_for_every_table`](../tests/end2end_pg.rb#test_models_are_generated_for_every_table), [`test_declared_types_match_runtime_values`](../tests/end2end_pg.rb#test_declared_types_match_runtime_values) | [`test_models_are_generated_for_every_table`](../tests/end2end_mysql2.rb#test_models_are_generated_for_every_table), [`test_declared_types_match_runtime_values`](../tests/end2end_mysql2.rb#test_declared_types_match_runtime_values), [`test_declared_types_match_runtime_values_for_nulls`](../tests/end2end_mysql2.rb#test_declared_types_match_runtime_values_for_nulls) | [`test_models_are_generated_for_every_table`](../tests/end2end_sqlite3.rb#test_models_are_generated_for_every_table), [`test_declared_types_match_runtime_values`](../tests/end2end_sqlite3.rb#test_declared_types_match_runtime_values) | [`test_unknown_types_in_queries_fall_back_to_untyped`](../test/features_test.rb#test_unknown_types_in_queries_fall_back_to_untyped), [`test_models_fall_back_to_untyped_for_unknown_types`](../test/features_test.rb#test_models_fall_back_to_untyped_for_unknown_types) | +| RuboCop style, query/schema comments and source SQL constants | [`test_flow`](../tests/end2end_pg.rb#test_flow) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow) | [`test_generated_ruby_is_formatted_like_ruby`](../test/features_test.rb#test_generated_ruby_is_formatted_like_ruby), [`test_ruby31_struct_rows_of_unnamed_columns_are_rubocop_clean`](../test/features_test.rb#test_ruby31_struct_rows_of_unnamed_columns_are_rubocop_clean), [`test_query_comments_become_doc_comments`](../test/features_test.rb#test_query_comments_become_doc_comments), [`test_table_column_and_enum_comments_become_doc_comments`](../test/features_test.rb#test_table_column_and_enum_comments_become_doc_comments), [`test_sql_is_a_frozen_raw_heredoc`](../test/features_test.rb#test_sql_is_a_frozen_raw_heredoc) | +| Query names like `HasAuthors` / `IsActive` keep their name and stay RuboCop-clean | [`test_predicate_like_query_names_are_kept`](../tests/end2end_pg.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_mysql2.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_sqlite3.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_rubocop_clean`](../test/features_test.rb#test_predicate_like_query_names_are_rubocop_clean) | + +## Queries + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| | `:one`, `:many`, `:exec`, nil for no row, bang finder methods and `NotFound` error | [`test_flow`](../tests/end2end_pg.rb#test_flow), [`test_one_on_empty_table_returns_nil`](../tests/end2end_pg.rb#test_one_on_empty_table_returns_nil), [`test_every_one_query_has_a_bang_method`](../tests/end2end_pg.rb#test_every_one_query_has_a_bang_method), [`test_bang_finder_raises_not_found`](../tests/end2end_pg.rb#test_bang_finder_raises_not_found) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow), [`test_one_on_empty_table_returns_nil`](../tests/end2end_mysql2.rb#test_one_on_empty_table_returns_nil), [`test_every_one_query_has_a_bang_method`](../tests/end2end_mysql2.rb#test_every_one_query_has_a_bang_method), [`test_bang_finder_raises_not_found`](../tests/end2end_mysql2.rb#test_bang_finder_raises_not_found) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow), [`test_every_one_query_has_a_bang_method`](../tests/end2end_sqlite3.rb#test_every_one_query_has_a_bang_method), [`test_bang_finder_raises_not_found`](../tests/end2end_sqlite3.rb#test_bang_finder_raises_not_found) | [`test_command_signatures`](../test/features_test.rb#test_command_signatures) | | `:execrows`, `:execlastid`, `:execresult` semantics | [`test_execrows_returns_affected_row_count`](../tests/end2end_pg.rb#test_execrows_returns_affected_row_count), [`test_execlastid_returns_the_returning_column`](../tests/end2end_pg.rb#test_execlastid_returns_the_returning_column), [`test_execresult_returns_rows_affected`](../tests/end2end_pg.rb#test_execresult_returns_rows_affected) | [`test_execrows_returns_affected_row_count`](../tests/end2end_mysql2.rb#test_execrows_returns_affected_row_count), [`test_execrows_counts_changed_rows_unless_found_rows_is_set`](../tests/end2end_mysql2.rb#test_execrows_counts_changed_rows_unless_found_rows_is_set), [`test_execlastid_returns_the_new_id`](../tests/end2end_mysql2.rb#test_execlastid_returns_the_new_id), [`test_execresult_returns_rows_affected_and_last_id`](../tests/end2end_mysql2.rb#test_execresult_returns_rows_affected_and_last_id) | [`test_execrows_returns_affected_row_count`](../tests/end2end_sqlite3.rb#test_execrows_returns_affected_row_count), [`test_execlastid_returns_the_returning_column`](../tests/end2end_sqlite3.rb#test_execlastid_returns_the_returning_column), [`test_execresult_returns_rows_affected_and_last_id`](../tests/end2end_sqlite3.rb#test_execresult_returns_rows_affected_and_last_id) | [`test_pg_and_sqlite_execlastid_need_returning`](../test/features_test.rb#test_pg_and_sqlite_execlastid_need_returning) | +| `:execresult` with `RETURNING` runs the statement | covered by `:execresult` semantics | covered by `:execresult` semantics | [`test_execresult_with_returning_runs_the_statement`](../tests/end2end_sqlite3.rb#test_execresult_with_returning_runs_the_statement) | | | `:batchexec`, `:batchone`, `:batchmany`; batches use one connection and are not atomic unless wrapped in a transaction | [`test_batch_commands_run_once_per_args`](../tests/end2end_pg.rb#test_batch_commands_run_once_per_args), [`test_batches_are_not_atomic_without_a_transaction`](../tests/end2end_pg.rb#test_batches_are_not_atomic_without_a_transaction) | [`test_batch_commands_run_once_per_args`](../tests/end2end_mysql2.rb#test_batch_commands_run_once_per_args), [`test_batches_are_not_atomic_without_a_transaction`](../tests/end2end_mysql2.rb#test_batches_are_not_atomic_without_a_transaction) | [`test_batch_commands_run_once_per_args`](../tests/end2end_sqlite3.rb#test_batch_commands_run_once_per_args), [`test_batches_are_not_atomic_without_a_transaction`](../tests/end2end_sqlite3.rb#test_batches_are_not_atomic_without_a_transaction) | [`test_batch_commands_take_an_array_of_args`](../test/features_test.rb#test_batch_commands_take_an_array_of_args) | | `:copyfrom` round trips, all-or-none behavior and conflict clauses | [`test_copyfrom_many_rows_and_none`](../tests/end2end_pg.rb#test_copyfrom_many_rows_and_none), [`test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic`](../tests/end2end_pg.rb#test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic) | [`test_copyfrom_loads_binary_values`](../tests/end2end_mysql2.rb#test_copyfrom_loads_binary_values), [`test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic`](../tests/end2end_mysql2.rb#test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic) | [`test_copyfrom_is_all_or_nothing`](../tests/end2end_sqlite3.rb#test_copyfrom_is_all_or_nothing), [`test_copyfrom_keeps_conflict_clauses`](../tests/end2end_sqlite3.rb#test_copyfrom_keeps_conflict_clauses) | [`test_copyfrom_keeps_or_rejects_conflict_clauses`](../test/features_test.rb#test_copyfrom_keeps_or_rejects_conflict_clauses) | +| `:copyfrom` keeps the case of quoted / mixed-case table and column names (MySQL `LOAD DATA`) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_pg.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_mysql2.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_sqlite3.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | | +| `:copyfrom` takes any `Enumerable` (empty, lazy, `Enumerator`) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_pg.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_mysql2.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_sqlite3.rb#test_copyfrom_takes_any_enumerable) | | | `sqlc.arg`, `sqlc.narg`, positional/named keyword args, optional nils and argument errors | [`test_named_nullable_arguments_are_optional`](../tests/end2end_pg.rb#test_named_nullable_arguments_are_optional), [`test_required_argument_after_optional_named_argument`](../tests/end2end_pg.rb#test_required_argument_after_optional_named_argument), [`test_missing_keyword_arguments_are_errors`](../tests/end2end_pg.rb#test_missing_keyword_arguments_are_errors) | [`test_nullable_named_arg`](../tests/end2end_mysql2.rb#test_nullable_named_arg), [`test_required_argument_after_optional_named_argument`](../tests/end2end_mysql2.rb#test_required_argument_after_optional_named_argument), [`test_missing_keyword_arguments_are_errors`](../tests/end2end_mysql2.rb#test_missing_keyword_arguments_are_errors) | [`test_nullable_named_arg`](../tests/end2end_sqlite3.rb#test_nullable_named_arg), [`test_required_argument_after_optional_named_argument`](../tests/end2end_sqlite3.rb#test_required_argument_after_optional_named_argument), [`test_missing_keyword_arguments_are_errors`](../tests/end2end_sqlite3.rb#test_missing_keyword_arguments_are_errors) | [`test_named_nullable_parameters_are_optional_keywords`](../test/features_test.rb#test_named_nullable_parameters_are_optional_keywords) | | `sqlc.slice`, placeholder order and repeated parameters | unit: PostgreSQL uses arrays instead of slices | [`test_slice`](../tests/end2end_mysql2.rb#test_slice) | [`test_slice`](../tests/end2end_sqlite3.rb#test_slice), [`test_numbered_placeholders_keep_their_bindings_after_slice_expansion`](../tests/end2end_sqlite3.rb#test_numbered_placeholders_keep_their_bindings_after_slice_expansion), [`test_non_slice_numbered_placeholders_keep_sql_order`](../tests/end2end_sqlite3.rb#test_non_slice_numbered_placeholders_keep_sql_order), [`test_batch_numbered_placeholders_keep_sql_order`](../tests/end2end_sqlite3.rb#test_batch_numbered_placeholders_keep_sql_order) | [`test_slices_are_rejected_for_postgres`](../test/features_test.rb#test_slices_are_rejected_for_postgres) | +| Empty `sqlc.slice`: `IN` matches nothing, `NOT IN` everything | unit: PostgreSQL uses arrays instead of slices | [`test_empty_slice_matches_nothing`](../tests/end2end_mysql2.rb#test_empty_slice_matches_nothing), [`test_not_in_an_empty_slice_matches_everything`](../tests/end2end_mysql2.rb#test_not_in_an_empty_slice_matches_everything) | [`test_empty_slice_matches_nothing`](../tests/end2end_sqlite3.rb#test_empty_slice_matches_nothing), [`test_not_in_an_empty_slice_matches_everything`](../tests/end2end_sqlite3.rb#test_not_in_an_empty_slice_matches_everything) | | | `sqlc.embed`, outer-join nil, same table embedded twice | [`test_embed`](../tests/end2end_pg.rb#test_embed), [`test_embed_on_the_outer_side_of_a_join_is_nil_without_a_match`](../tests/end2end_pg.rb#test_embed_on_the_outer_side_of_a_join_is_nil_without_a_match), [`test_same_table_embedded_twice`](../tests/end2end_pg.rb#test_same_table_embedded_twice) | [`test_embed`](../tests/end2end_mysql2.rb#test_embed), [`test_outer_join_embed_is_nil_without_a_match`](../tests/end2end_mysql2.rb#test_outer_join_embed_is_nil_without_a_match), [`test_same_table_embedded_twice`](../tests/end2end_mysql2.rb#test_same_table_embedded_twice) | [`test_embed`](../tests/end2end_sqlite3.rb#test_embed), [`test_outer_join_embed_is_nil_without_a_match`](../tests/end2end_sqlite3.rb#test_outer_join_embed_is_nil_without_a_match), [`test_same_table_embedded_twice`](../tests/end2end_sqlite3.rb#test_same_table_embedded_twice) | [`test_embedded_tables_become_model_attributes`](../test/features_test.rb#test_embedded_tables_become_model_attributes) | +| Unnamed result columns (`column_1`, ...) and aliases with quotes | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_pg.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_pg.rb#test_alias_with_a_quote_is_read) | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_mysql2.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_mysql2.rb#test_alias_with_a_quote_is_read) | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_sqlite3.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_sqlite3.rb#test_alias_with_a_quote_is_read) | | + +## Rows, models and names + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| | Frozen Ruby 3.2+ `Data`, mutable older `Struct`, `with`, `to_h`, pattern matching | [`test_rows_and_embedded_models_have_supported_layout_and_pattern_matching`](../tests/end2end_pg.rb#test_rows_and_embedded_models_have_supported_layout_and_pattern_matching) | [`test_rows_and_embedded_models_have_supported_layout_and_pattern_matching`](../tests/end2end_mysql2.rb#test_rows_and_embedded_models_have_supported_layout_and_pattern_matching) | [`test_rows_and_embedded_models_have_supported_layout_and_pattern_matching`](../tests/end2end_sqlite3.rb#test_rows_and_embedded_models_have_supported_layout_and_pattern_matching) | [`test_struct_classes_are_valid_and_usable`](../test/features_test.rb#test_struct_classes_are_valid_and_usable), [`test_data_classes_are_usable`](../test/features_test.rb#test_data_classes_are_usable) | +| Rows through `Marshal` (Rails.cache) and JSON (`to_h.to_json`) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_pg.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_mysql2.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_sqlite3.rb#test_rows_round_trip_through_marshal_and_json) | | +| Names: keywords, method collisions (also private row hooks such as `initialize`), acronyms, quoted mixed case, repeated `$n`, plural models | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_pg.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names), [`test_repeated_column_parameters_get_distinct_names`](../tests/end2end_pg.rb#test_repeated_column_parameters_get_distinct_names) | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_mysql2.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names), [`test_repeated_positional_parameters_get_distinct_names`](../tests/end2end_mysql2.rb#test_repeated_positional_parameters_get_distinct_names) | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_sqlite3.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names) | [`test_snake_case_keeps_acronyms_together`](../test/features_test.rb#test_snake_case_keeps_acronyms_together), [`test_query_names_cannot_replace_generated_methods`](../test/features_test.rb#test_query_names_cannot_replace_generated_methods), [`test_tables_with_the_same_singular_get_distinct_models`](../test/features_test.rb#test_tables_with_the_same_singular_get_distinct_models), [`test_columns_named_like_row_methods_still_build_and_read_rows`](../test/features_test.rb#test_columns_named_like_row_methods_still_build_and_read_rows) | +| Models named like Ruby / driver constants (`Exception`, `Data`, `Thread`, `Set`, `Monitor`, ...) do not shadow them | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_pg.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_mysql2.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_sqlite3.rb#test_models_named_like_core_constants_do_not_shadow_them) | | + +## Types + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| | Enums, constants (also for keyword / invalid names), database labels and MySQL `SET` | [`test_enum_column`](../tests/end2end_pg.rb#test_enum_column), [`test_enum_values_named_like_ruby_keywords_get_valid_constants`](../tests/end2end_pg.rb#test_enum_values_named_like_ruby_keywords_get_valid_constants) | [`test_enum_and_set_columns`](../tests/end2end_mysql2.rb#test_enum_and_set_columns), [`test_enum_values_named_like_ruby_keywords_get_valid_constants`](../tests/end2end_mysql2.rb#test_enum_values_named_like_ruby_keywords_get_valid_constants) | not supported by the SQLite schema | [`test_models_cover_tables_and_enums`](../test/features_test.rb#test_models_cover_tables_and_enums) | +| Top-level Float JSON parameters, Symbol enum values | [`test_top_level_float_json_parameters_round_trip_exactly`](../tests/end2end_pg.rb#test_top_level_float_json_parameters_round_trip_exactly) | [`test_symbol_enum_value_is_bound_as_its_name`](../tests/end2end_mysql2.rb#test_symbol_enum_value_is_bound_as_its_name) | not SQLite (no enums / JSON encoding) | | | PostgreSQL arrays, JSON/JSONB, type round trips and RBS; a `Time` keeps its instant in a `timestamp` column whatever its offset; a `DateTime` keeps its instant in `timestamp` / `timestamptz` | [`test_array_parameters`](../tests/end2end_pg.rb#test_array_parameters), [`test_numeric_types_round_trip`](../tests/end2end_pg.rb#test_numeric_types_round_trip), [`test_string_types_round_trip`](../tests/end2end_pg.rb#test_string_types_round_trip), [`test_datetime_types_round_trip`](../tests/end2end_pg.rb#test_datetime_types_round_trip), [`test_array_types_round_trip`](../tests/end2end_pg.rb#test_array_types_round_trip), [`test_special_types_round_trip`](../tests/end2end_pg.rb#test_special_types_round_trip), [`test_type_families_round_trip_nil`](../tests/end2end_pg.rb#test_type_families_round_trip_nil), [`test_declared_types_match_runtime_values`](../tests/end2end_pg.rb#test_declared_types_match_runtime_values), [`test_declared_types_match_runtime_values_for_nulls`](../tests/end2end_pg.rb#test_declared_types_match_runtime_values_for_nulls), [`test_timestamp_parameters_keep_their_instant_whatever_their_offset`](../tests/end2end_pg.rb#test_timestamp_parameters_keep_their_instant_whatever_their_offset), [`test_datetime_parameters_keep_their_instant`](../tests/end2end_pg.rb#test_datetime_parameters_keep_their_instant), [`test_json_and_jsonb_parameters_round_trip_all_json_value_types`](../tests/end2end_pg.rb#test_json_and_jsonb_parameters_round_trip_all_json_value_types) | not PostgreSQL types | not PostgreSQL types | [`test_multi_dimensional_arrays`](../test/features_test.rb#test_multi_dimensional_arrays), [`test_more_postgres_types_composites_and_domains`](../test/features_test.rb#test_more_postgres_types_composites_and_domains) | -| MySQL unsigned integers, BIT, YEAR, DATETIME/TIME (a `Time` keeps its instant in any zone, also in `:copyfrom`), JSON, enums/SET and client quoting | not MySQL | [`test_client_side_quoting_round_trips_supported_values`](../tests/end2end_mysql2.rb#test_client_side_quoting_round_trips_supported_values), [`test_time_values_keep_their_instant_whatever_their_offset`](../tests/end2end_mysql2.rb#test_time_values_keep_their_instant_whatever_their_offset), [`test_time_values_follow_a_utc_database_timezone`](../tests/end2end_mysql2.rb#test_time_values_follow_a_utc_database_timezone), [`test_declared_types_match_runtime_values`](../tests/end2end_mysql2.rb#test_declared_types_match_runtime_values) | not MySQL | [`test_mysql_uses_client_side_binding_and_sqlite_closes_prepared_statements`](../test/features_test.rb#test_mysql_uses_client_side_binding_and_sqlite_closes_prepared_statements) | -| SQLite affinity, JSON/UUID, and NUMERIC integer/float results | not SQLite | not SQLite | [`test_declared_types_match_runtime_values`](../tests/end2end_sqlite3.rb#test_declared_types_match_runtime_values), [`test_declared_types_with_parameters`](../tests/end2end_sqlite3.rb#test_declared_types_with_parameters) | [`test_sqlite_declared_types_follow_type_affinity`](../test/features_test.rb#test_sqlite_declared_types_follow_type_affinity), [`test_sqlite_json_and_uuid_declarations_are_strings`](../test/features_test.rb#test_sqlite_json_and_uuid_declarations_are_strings) | -| SQLite `busy_handler_timeout=` waits for another connection without blocking other threads | not SQLite | not SQLite | [`test_busy_handler_timeout_waits_for_another_connection_without_blocking_threads`](../tests/end2end_sqlite3.rb#test_busy_handler_timeout_waits_for_another_connection_without_blocking_threads) | | -| Names: keywords, method collisions (also private row hooks such as `initialize`), acronyms, quoted mixed case, repeated `$n`, plural models | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_pg.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names), [`test_repeated_column_parameters_get_distinct_names`](../tests/end2end_pg.rb#test_repeated_column_parameters_get_distinct_names) | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_mysql2.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names), [`test_repeated_positional_parameters_get_distinct_names`](../tests/end2end_mysql2.rb#test_repeated_positional_parameters_get_distinct_names) | [`test_keyword_and_method_names_keep_distinct_row_and_argument_names`](../tests/end2end_sqlite3.rb#test_keyword_and_method_names_keep_distinct_row_and_argument_names) | [`test_snake_case_keeps_acronyms_together`](../test/features_test.rb#test_snake_case_keeps_acronyms_together), [`test_query_names_cannot_replace_generated_methods`](../test/features_test.rb#test_query_names_cannot_replace_generated_methods), [`test_tables_with_the_same_singular_get_distinct_models`](../test/features_test.rb#test_tables_with_the_same_singular_get_distinct_models), [`test_columns_named_like_row_methods_still_build_and_read_rows`](../test/features_test.rb#test_columns_named_like_row_methods_still_build_and_read_rows) | +| PostgreSQL arrays of enums created after the connection was opened | [`test_enum_arrays_of_a_type_created_after_connecting_are_decoded`](../tests/end2end_pg.rb#test_enum_arrays_of_a_type_created_after_connecting_are_decoded) | not MySQL | not SQLite | | +| MySQL unsigned integers, BIT, YEAR, DATETIME/TIME (a `Time` keeps its instant in any zone, also in `:copyfrom`), JSON, enums/SET and client quoting | not PostgreSQL | [`test_client_side_quoting_round_trips_supported_values`](../tests/end2end_mysql2.rb#test_client_side_quoting_round_trips_supported_values), [`test_time_values_keep_their_instant_whatever_their_offset`](../tests/end2end_mysql2.rb#test_time_values_keep_their_instant_whatever_their_offset), [`test_time_values_follow_a_utc_database_timezone`](../tests/end2end_mysql2.rb#test_time_values_follow_a_utc_database_timezone), [`test_declared_types_match_runtime_values`](../tests/end2end_mysql2.rb#test_declared_types_match_runtime_values) | not MySQL | [`test_mysql_uses_client_side_binding_and_sqlite_closes_prepared_statements`](../test/features_test.rb#test_mysql_uses_client_side_binding_and_sqlite_closes_prepared_statements) | +| SQLite affinity, JSON/UUID, and NUMERIC integer/float results | not PostgreSQL | not MySQL | [`test_declared_types_match_runtime_values`](../tests/end2end_sqlite3.rb#test_declared_types_match_runtime_values), [`test_declared_types_with_parameters`](../tests/end2end_sqlite3.rb#test_declared_types_with_parameters) | [`test_sqlite_declared_types_follow_type_affinity`](../test/features_test.rb#test_sqlite_declared_types_follow_type_affinity), [`test_sqlite_json_and_uuid_declarations_are_strings`](../test/features_test.rb#test_sqlite_json_and_uuid_declarations_are_strings) | +| SQLite `JSON` / `UUID` columns are `String`s, also numeric-looking text | not PostgreSQL | not MySQL | [`test_json_and_uuid_columns_return_strings_that_look_numeric`](../tests/end2end_sqlite3.rb#test_json_and_uuid_columns_return_strings_that_look_numeric) | | + +## Connections + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| | Connections, injected pools, reconnect behavior and SQLite serialization | [`test_queries_can_share_an_injected_pool`](../tests/end2end_pg.rb#test_queries_can_share_an_injected_pool), [`test_a_connection_closed_by_the_server_is_reconnected`](../tests/end2end_pg.rb#test_a_connection_closed_by_the_server_is_reconnected), [`test_connection_lost_inside_transaction_is_not_reconnected_until_the_transaction_ends`](../tests/end2end_pg.rb#test_connection_lost_inside_transaction_is_not_reconnected_until_the_transaction_ends) | [`test_queries_can_share_an_injected_pool`](../tests/end2end_mysql2.rb#test_queries_can_share_an_injected_pool), [`test_a_connection_closed_by_the_server_is_replaced`](../tests/end2end_mysql2.rb#test_a_connection_closed_by_the_server_is_replaced), [`test_a_stopped_async_task_does_not_leave_a_closed_connection_in_the_pool`](../tests/end2end_mysql2.rb#test_a_stopped_async_task_does_not_leave_a_closed_connection_in_the_pool) | [`test_fibers_sharing_a_database_take_turns`](../tests/end2end_sqlite3.rb#test_fibers_sharing_a_database_take_turns), [`test_threads_sharing_a_database_take_turns_in_transactions`](../tests/end2end_sqlite3.rb#test_threads_sharing_a_database_take_turns_in_transactions), [`test_a_database_returning_hash_rows_still_builds_rows`](../tests/end2end_sqlite3.rb#test_a_database_returning_hash_rows_still_builds_rows) | [`test_engine_must_match_the_driver`](../test/features_test.rb#test_engine_must_match_the_driver) | -| Transaction result, commit/rollback, nested savepoint, exit semantics, rescued PostgreSQL error and rescued MySQL deadlock / lock wait timeout | [`test_nested_savepoint_rolls_back_while_outer_transaction_commits`](../tests/end2end_pg.rb#test_nested_savepoint_rolls_back_while_outer_transaction_commits), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_pg.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version), [`test_transaction_whose_failed_statement_was_rescued_raises_instead_of_committing`](../tests/end2end_pg.rb#test_transaction_whose_failed_statement_was_rescued_raises_instead_of_committing) | [`test_nested_savepoint_rolls_back_while_outer_transaction_commits`](../tests/end2end_mysql2.rb#test_nested_savepoint_rolls_back_while_outer_transaction_commits), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_mysql2.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version), [`test_rescued_deadlock_in_a_transaction_raises_instead_of_committing`](../tests/end2end_mysql2.rb#test_rescued_deadlock_in_a_transaction_raises_instead_of_committing), [`test_rescued_deadlock_in_a_nested_transaction_raises_instead_of_committing`](../tests/end2end_mysql2.rb#test_rescued_deadlock_in_a_nested_transaction_raises_instead_of_committing), [`test_rescued_lock_wait_timeout_rolls_back_only_its_statement`](../tests/end2end_mysql2.rb#test_rescued_lock_wait_timeout_rolls_back_only_its_statement) | [`test_nested_transaction_rolls_back_to_a_savepoint`](../tests/end2end_sqlite3.rb#test_nested_transaction_rolls_back_to_a_savepoint), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_sqlite3.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version) | [`test_nested_transactions_reject_isolation_and_mode`](../test/features_test.rb#test_nested_transactions_reject_isolation_and_mode) | -| Isolation applied; invalid isolation/mode rejected with `ArgumentError` before begin; SQLite locks | [`test_transaction_isolation_level`](../tests/end2end_pg.rb#test_transaction_isolation_level) | [`test_transaction_isolation_level_is_applied`](../tests/end2end_mysql2.rb#test_transaction_isolation_level_is_applied) | [`test_transaction_mode`](../tests/end2end_sqlite3.rb#test_transaction_mode), [`test_immediate_and_exclusive_transactions_block_a_second_connection`](../tests/end2end_sqlite3.rb#test_immediate_and_exclusive_transactions_block_a_second_connection), [`test_unknown_transaction_mode_is_rejected_before_beginning`](../tests/end2end_sqlite3.rb#test_unknown_transaction_mode_is_rejected_before_beginning) | [`test_nested_transactions_reject_isolation_and_mode`](../test/features_test.rb#test_nested_transactions_reject_isolation_and_mode) | -| Logger debug format, transaction logs, secret-free parameters | [`test_logger_receives_query_names_and_durations`](../tests/end2end_pg.rb#test_logger_receives_query_names_and_durations) | [`test_logger_receives_query_names_and_durations`](../tests/end2end_mysql2.rb#test_logger_receives_query_names_and_durations) | [`test_logger_receives_query_names_and_durations`](../tests/end2end_sqlite3.rb#test_logger_receives_query_names_and_durations) | [`test_logger_support`](../test/features_test.rb#test_logger_support) | -| Fiber scheduler concurrency / MySQL no prepared-statement leaks | [`test_queries_run_concurrently_under_the_fiber_scheduler`](../tests/end2end_pg.rb#test_queries_run_concurrently_under_the_fiber_scheduler) | [`test_generated_queries_overlap_under_sync_including_transactions`](../tests/end2end_mysql2.rb#test_generated_queries_overlap_under_sync_including_transactions), [`test_generated_queries_do_not_create_server_side_prepared_statements`](../tests/end2end_mysql2.rb#test_generated_queries_do_not_create_server_side_prepared_statements) | [`test_fibers_sharing_a_database_take_turns`](../tests/end2end_sqlite3.rb#test_fibers_sharing_a_database_take_turns) | | -| Duplicate result columns, unsupported options, invalid `:copyfrom` clauses and generation errors | E2E cannot reach generation-time rejection | E2E cannot reach generation-time rejection | E2E cannot reach generation-time rejection | [`test_duplicate_result_columns_are_rejected`](../test/features_test.rb#test_duplicate_result_columns_are_rejected), [`test_copyfrom_keeps_or_rejects_conflict_clauses`](../test/features_test.rb#test_copyfrom_keeps_or_rejects_conflict_clauses), [`test_unknown_options_are_rejected_with_a_hint`](../test/options_test.rb#test_unknown_options_are_rejected_with_a_hint), [`test_query_names_cannot_replace_generated_methods`](../test/features_test.rb#test_query_names_cannot_replace_generated_methods) | -| RuboCop style, query/schema comments and source SQL constants | [`test_flow`](../tests/end2end_pg.rb#test_flow) | [`test_flow`](../tests/end2end_mysql2.rb#test_flow) | [`test_flow`](../tests/end2end_sqlite3.rb#test_flow) | [`test_generated_ruby_is_formatted_like_ruby`](../test/features_test.rb#test_generated_ruby_is_formatted_like_ruby), [`test_ruby31_struct_rows_of_unnamed_columns_are_rubocop_clean`](../test/features_test.rb#test_ruby31_struct_rows_of_unnamed_columns_are_rubocop_clean), [`test_query_comments_become_doc_comments`](../test/features_test.rb#test_query_comments_become_doc_comments), [`test_table_column_and_enum_comments_become_doc_comments`](../test/features_test.rb#test_table_column_and_enum_comments_become_doc_comments), [`test_sql_is_a_frozen_raw_heredoc`](../test/features_test.rb#test_sql_is_a_frozen_raw_heredoc) | -| Unnamed result columns (`column_1`, ...) and aliases with quotes | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_pg.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_pg.rb#test_alias_with_a_quote_is_read) | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_mysql2.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_mysql2.rb#test_alias_with_a_quote_is_read) | [`test_unnamed_result_columns_are_read_by_position`](../tests/end2end_sqlite3.rb#test_unnamed_result_columns_are_read_by_position), [`test_alias_with_a_quote_is_read`](../tests/end2end_sqlite3.rb#test_alias_with_a_quote_is_read) | | -| Empty `sqlc.slice`: `IN` matches nothing, `NOT IN` everything | unit: PostgreSQL uses arrays instead of slices | [`test_empty_slice_matches_nothing`](../tests/end2end_mysql2.rb#test_empty_slice_matches_nothing), [`test_not_in_an_empty_slice_matches_everything`](../tests/end2end_mysql2.rb#test_not_in_an_empty_slice_matches_everything) | [`test_empty_slice_matches_nothing`](../tests/end2end_sqlite3.rb#test_empty_slice_matches_nothing), [`test_not_in_an_empty_slice_matches_everything`](../tests/end2end_sqlite3.rb#test_not_in_an_empty_slice_matches_everything) | | -| A transaction ended by the database keeps the original error; a query abandoned by Timeout is cancelled; a Timeout in a PostgreSQL transaction raises `Timeout::Error`, also with Ruby 3.1's bundled timeout 0.2.0 | [`test_query_abandoned_by_a_timeout_is_cancelled`](../tests/end2end_pg.rb#test_query_abandoned_by_a_timeout_is_cancelled), [`test_transaction_timeout_rolls_back_and_keeps_connection_usable`](../tests/end2end_pg.rb#test_transaction_timeout_rolls_back_and_keeps_connection_usable) | [`test_deadlock_in_a_nested_transaction_raises_the_deadlock_error`](../tests/end2end_mysql2.rb#test_deadlock_in_a_nested_transaction_raises_the_deadlock_error) | [`test_conflict_that_ends_the_transaction_raises_the_constraint_error`](../tests/end2end_sqlite3.rb#test_conflict_that_ends_the_transaction_raises_the_constraint_error) | | -| Top-level Float JSON parameters, Symbol enum values | [`test_top_level_float_json_parameters_round_trip_exactly`](../tests/end2end_pg.rb#test_top_level_float_json_parameters_round_trip_exactly) | [`test_symbol_enum_value_is_bound_as_its_name`](../tests/end2end_mysql2.rb#test_symbol_enum_value_is_bound_as_its_name) | not SQLite (no enums / JSON encoding) | | -| Modules sharing a connection keep separate prepared statements | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | not MySQL (no server-side prepared statements) | not SQLite (statements are per call) | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) | -| SQLite parameters sqlc does not report (`ON CONFLICT ... DO UPDATE SET`) are generation errors | not PostgreSQL | not MySQL | unit: rejected at generation | [`test_sqlite_parameters_sqlc_does_not_report_are_generation_errors`](../test/features_test.rb#test_sqlite_parameters_sqlc_does_not_report_are_generation_errors), [`test_sqlite_placeholders_in_literals_and_comments_are_not_parameters`](../test/features_test.rb#test_sqlite_placeholders_in_literals_and_comments_are_not_parameters) | -| Forked processes (Puma / Sidekiq workers): pooled connections are replaced in the child without ending the parent's sessions; `pg` direct connections reconnect, `mysql2` direct clients raise, SQLite children open their own database | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_pg.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_mysql2.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable`](../tests/end2end_sqlite3.rb#test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable) | | -| A forked child's plain `exit` leaves the parent's sessions open: a pooled connection another thread has checked out at fork time, a direct connection the child never uses | [pg](../tests/end2end_pg.rb) `test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open` · [mysql2](../tests/end2end_mysql2.rb) `test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open` | -| A connection shared with another library keeps its settings (pg type maps / `field_name_type`, mysql2 query options and `default_query_options`, SQLite `results_as_hash`) | [`test_a_connection_shared_with_another_library_keeps_its_settings`](../tests/end2end_pg.rb#test_a_connection_shared_with_another_library_keeps_its_settings) | [`test_a_client_with_other_query_options_still_returns_typed_rows`](../tests/end2end_mysql2.rb#test_a_client_with_other_query_options_still_returns_typed_rows) | [`test_a_database_shared_with_another_library_keeps_its_settings`](../tests/end2end_sqlite3.rb#test_a_database_shared_with_another_library_keeps_its_settings) | | | `pg`: `QuerySql.connect` takes a connection URL (`DATABASE_URL`) | [`test_connect_accepts_a_connection_url`](../tests/end2end_pg.rb#test_connect_accepts_a_connection_url) | not MySQL (mysql2 takes no URL) | not SQLite (no `connect`) | | -| `transaction` inside a transaction opened by another library: a savepoint (PostgreSQL, SQLite); MySQL limitation: commits it | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_pg.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | [`test_transaction_inside_a_transaction_opened_by_another_library_commits_it`](../tests/end2end_mysql2.rb#test_transaction_inside_a_transaction_opened_by_another_library_commits_it) | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_sqlite3.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | | +| A connection shared with another library keeps its settings (pg type maps / `field_name_type`, mysql2 query options and `default_query_options`, SQLite `results_as_hash`) | [`test_a_connection_shared_with_another_library_keeps_its_settings`](../tests/end2end_pg.rb#test_a_connection_shared_with_another_library_keeps_its_settings) | [`test_a_client_with_other_query_options_still_returns_typed_rows`](../tests/end2end_mysql2.rb#test_a_client_with_other_query_options_still_returns_typed_rows) | [`test_a_database_shared_with_another_library_keeps_its_settings`](../tests/end2end_sqlite3.rb#test_a_database_shared_with_another_library_keeps_its_settings) | | +| Forked processes (Puma / Sidekiq workers): pooled connections are replaced in the child without ending the parent's sessions; `pg` direct connections reconnect, `mysql2` direct clients raise, SQLite children open their own database | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_pg.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_mysql2.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable`](../tests/end2end_sqlite3.rb#test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable) | | +| A forked child's plain `exit` leaves the parent's sessions open: a pooled connection another thread has checked out at fork time, a direct connection the child never uses | [`test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open`](../tests/end2end_pg.rb#test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open) | [`test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open`](../tests/end2end_mysql2.rb#test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open) | not SQLite (a child opens its own database) | | | Schema-per-tenant `search_path` switching with prepared statements | [`test_queries_follow_the_search_path_of_each_tenant`](../tests/end2end_pg.rb#test_queries_follow_the_search_path_of_each_tenant) | not MySQL (databases, no search path) | not SQLite (no schemas) | | | Primary + read-replica query objects; a write to the read-only replica raises and leaves it usable | [`test_primary_and_read_replica_query_objects`](../tests/end2end_pg.rb#test_primary_and_read_replica_query_objects) | [`test_primary_and_read_replica_query_objects`](../tests/end2end_mysql2.rb#test_primary_and_read_replica_query_objects) | [`test_primary_and_read_replica_query_objects`](../tests/end2end_sqlite3.rb#test_primary_and_read_replica_query_objects) | | | Memoized frozen query objects, subclasses and prepended wrappers forwarding `...` / `**`, `SimpleDelegator`, shared across threads | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_pg.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_mysql2.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_sqlite3.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | | -| Rows through `Marshal` (Rails.cache) and JSON (`to_h.to_json`) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_pg.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_mysql2.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_sqlite3.rb#test_rows_round_trip_through_marshal_and_json) | | +| Fiber scheduler concurrency / MySQL no prepared-statement leaks | [`test_queries_run_concurrently_under_the_fiber_scheduler`](../tests/end2end_pg.rb#test_queries_run_concurrently_under_the_fiber_scheduler) | [`test_generated_queries_overlap_under_sync_including_transactions`](../tests/end2end_mysql2.rb#test_generated_queries_overlap_under_sync_including_transactions), [`test_generated_queries_do_not_create_server_side_prepared_statements`](../tests/end2end_mysql2.rb#test_generated_queries_do_not_create_server_side_prepared_statements) | [`test_fibers_sharing_a_database_take_turns`](../tests/end2end_sqlite3.rb#test_fibers_sharing_a_database_take_turns) | | +| SQLite `busy_handler_timeout=` waits for another connection without blocking other threads | not PostgreSQL | not MySQL | [`test_busy_handler_timeout_waits_for_another_connection_without_blocking_threads`](../tests/end2end_sqlite3.rb#test_busy_handler_timeout_waits_for_another_connection_without_blocking_threads) | | -CI runs `test/coverage_map_test.rb`, which checks every linked test method exists in its referenced `tests/` or `test/` file. -| Models named like Ruby / driver constants (`Exception`, `Data`, `Thread`, `Set`, `Monitor`, ...) do not shadow them | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_pg.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_mysql2.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_sqlite3.rb#test_models_named_like_core_constants_do_not_shadow_them) | | +## PostgreSQL prepared statements + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| Modules sharing a connection keep separate prepared statements | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | not MySQL (no server-side prepared statements) | not SQLite (statements are per call) | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) | +| PostgreSQL statements dropped by `DEALLOCATE ALL` / `DISCARD ALL`, or refused after a column type change (cached plan must not change result type), are prepared again | [`test_statements_are_prepared_again_after_deallocate_all`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_deallocate_all), [`test_statements_are_prepared_again_after_a_column_type_change`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_a_column_type_change) | not MySQL (no server-side prepared statements) | not SQLite | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) | +| PostgreSQL `PREPARE` interrupted (Timeout / `Thread#raise`, an exception raised by it) after the server ran it leaves the statement usable | [`test_an_interrupt_during_prepare_leaves_the_statement_usable`](../tests/end2end_pg.rb#test_an_interrupt_during_prepare_leaves_the_statement_usable), [`test_a_statement_left_prepared_by_a_failed_prepare_is_used`](../tests/end2end_pg.rb#test_a_statement_left_prepared_by_a_failed_prepare_is_used) | not MySQL (no server-side prepared statements) | not SQLite | | + +## Transactions + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| Transaction result, commit/rollback, nested savepoint, exit semantics, rescued PostgreSQL error and rescued MySQL deadlock / lock wait timeout | [`test_nested_savepoint_rolls_back_while_outer_transaction_commits`](../tests/end2end_pg.rb#test_nested_savepoint_rolls_back_while_outer_transaction_commits), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_pg.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version), [`test_transaction_whose_failed_statement_was_rescued_raises_instead_of_committing`](../tests/end2end_pg.rb#test_transaction_whose_failed_statement_was_rescued_raises_instead_of_committing) | [`test_nested_savepoint_rolls_back_while_outer_transaction_commits`](../tests/end2end_mysql2.rb#test_nested_savepoint_rolls_back_while_outer_transaction_commits), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_mysql2.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version), [`test_rescued_deadlock_in_a_transaction_raises_instead_of_committing`](../tests/end2end_mysql2.rb#test_rescued_deadlock_in_a_transaction_raises_instead_of_committing), [`test_rescued_deadlock_in_a_nested_transaction_raises_instead_of_committing`](../tests/end2end_mysql2.rb#test_rescued_deadlock_in_a_nested_transaction_raises_instead_of_committing), [`test_rescued_lock_wait_timeout_rolls_back_only_its_statement`](../tests/end2end_mysql2.rb#test_rescued_lock_wait_timeout_rolls_back_only_its_statement) | [`test_nested_transaction_rolls_back_to_a_savepoint`](../tests/end2end_sqlite3.rb#test_nested_transaction_rolls_back_to_a_savepoint), [`test_transaction_nonlocal_exit_semantics_follow_timeout_version`](../tests/end2end_sqlite3.rb#test_transaction_nonlocal_exit_semantics_follow_timeout_version) | [`test_nested_transactions_reject_isolation_and_mode`](../test/features_test.rb#test_nested_transactions_reject_isolation_and_mode) | +| Isolation applied; invalid isolation/mode rejected with `ArgumentError` before begin; SQLite locks | [`test_transaction_isolation_level`](../tests/end2end_pg.rb#test_transaction_isolation_level) | [`test_transaction_isolation_level_is_applied`](../tests/end2end_mysql2.rb#test_transaction_isolation_level_is_applied) | [`test_transaction_mode`](../tests/end2end_sqlite3.rb#test_transaction_mode), [`test_immediate_and_exclusive_transactions_block_a_second_connection`](../tests/end2end_sqlite3.rb#test_immediate_and_exclusive_transactions_block_a_second_connection), [`test_unknown_transaction_mode_is_rejected_before_beginning`](../tests/end2end_sqlite3.rb#test_unknown_transaction_mode_is_rejected_before_beginning) | [`test_nested_transactions_reject_isolation_and_mode`](../test/features_test.rb#test_nested_transactions_reject_isolation_and_mode) | +| `transaction` inside a transaction opened by another library: a savepoint (PostgreSQL, SQLite); MySQL limitation: commits it | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_pg.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | [`test_transaction_inside_a_transaction_opened_by_another_library_commits_it`](../tests/end2end_mysql2.rb#test_transaction_inside_a_transaction_opened_by_another_library_commits_it) | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_sqlite3.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | | | Transaction `tx` used from an `Enumerator`, `Fiber`, `Thread`, `Async` task and nested savepoint; `TransactionEnded` after the transaction | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_pg.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_mysql2.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | [`test_transaction_query_object_works_from_other_fibers_and_threads`](../tests/end2end_sqlite3.rb#test_transaction_query_object_works_from_other_fibers_and_threads) | | -| PostgreSQL arrays of enums created after the connection was opened | [`test_enum_arrays_of_a_type_created_after_connecting_are_decoded`](../tests/end2end_pg.rb#test_enum_arrays_of_a_type_created_after_connecting_are_decoded) | not MySQL | not SQLite | | + +## Errors and interrupts + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| A transaction ended by the database keeps the original error; a query abandoned by Timeout is cancelled; a Timeout in a PostgreSQL transaction raises `Timeout::Error`, also with Ruby 3.1's bundled timeout 0.2.0 | [`test_query_abandoned_by_a_timeout_is_cancelled`](../tests/end2end_pg.rb#test_query_abandoned_by_a_timeout_is_cancelled), [`test_transaction_timeout_rolls_back_and_keeps_connection_usable`](../tests/end2end_pg.rb#test_transaction_timeout_rolls_back_and_keeps_connection_usable) | [`test_deadlock_in_a_nested_transaction_raises_the_deadlock_error`](../tests/end2end_mysql2.rb#test_deadlock_in_a_nested_transaction_raises_the_deadlock_error) | [`test_conflict_that_ends_the_transaction_raises_the_constraint_error`](../tests/end2end_sqlite3.rb#test_conflict_that_ends_the_transaction_raises_the_constraint_error) | | | Interrupts at `BEGIN` / `COMMIT` (`Thread#raise`, raised by the statement) roll back and leave no open transaction | [`test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction`](../tests/end2end_pg.rb#test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction) | [`test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction`](../tests/end2end_mysql2.rb#test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction) | [`test_interrupts_around_begin_and_commit_never_leave_the_database_in_a_transaction`](../tests/end2end_sqlite3.rb#test_interrupts_around_begin_and_commit_never_leave_the_database_in_a_transaction) | | | `Thread#kill` inside a transaction rolls back; on PostgreSQL a kill during a query (inside or outside a transaction) cancels it at once and leaves the connection idle; on MySQL a kill during a query replaces the client and ends its transaction | [`test_thread_kill_rolls_back_and_leaves_connection_idle`](../tests/end2end_pg.rb#test_thread_kill_rolls_back_and_leaves_connection_idle), [`test_thread_kill_during_a_query_cancels_it_and_leaves_the_connection_idle`](../tests/end2end_pg.rb#test_thread_kill_during_a_query_cancels_it_and_leaves_the_connection_idle) | [`test_thread_kill_rolls_back_and_leaves_connection_outside_transaction`](../tests/end2end_mysql2.rb#test_thread_kill_rolls_back_and_leaves_connection_outside_transaction), [`test_thread_kill_during_a_query_replaces_the_connection_and_ends_its_transaction`](../tests/end2end_mysql2.rb#test_thread_kill_during_a_query_replaces_the_connection_and_ends_its_transaction) | [`test_thread_kill_rolls_back_and_leaves_connection_inactive`](../tests/end2end_sqlite3.rb#test_thread_kill_rolls_back_and_leaves_connection_inactive) | | -| PostgreSQL statements dropped by `DEALLOCATE ALL` / `DISCARD ALL`, or refused after a column type change (cached plan must not change result type), are prepared again | [`test_statements_are_prepared_again_after_deallocate_all`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_deallocate_all), [`test_statements_are_prepared_again_after_a_column_type_change`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_a_column_type_change) | not MySQL (no server-side prepared statements) | not SQLite | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) | -| `:execresult` with `RETURNING` runs the statement | covered by `:execresult` semantics | covered by `:execresult` semantics | [`test_execresult_with_returning_runs_the_statement`](../tests/end2end_sqlite3.rb#test_execresult_with_returning_runs_the_statement) | | -| `:copyfrom` keeps the case of quoted / mixed-case table and column names (MySQL `LOAD DATA`) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_pg.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_mysql2.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | [`test_copyfrom_keeps_the_case_of_table_and_column_names`](../tests/end2end_sqlite3.rb#test_copyfrom_keeps_the_case_of_table_and_column_names) | | -| `:copyfrom` takes any `Enumerable` (empty, lazy, `Enumerator`) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_pg.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_mysql2.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_sqlite3.rb#test_copyfrom_takes_any_enumerable) | | -| Query names like `HasAuthors` / `IsActive` keep their name and stay RuboCop-clean | [`test_predicate_like_query_names_are_kept`](../tests/end2end_pg.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_mysql2.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_sqlite3.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_rubocop_clean`](../test/features_test.rb#test_predicate_like_query_names_are_rubocop_clean) | -| SQLite `JSON` / `UUID` columns are `String`s, also numeric-looking text | not PostgreSQL | not MySQL | [`test_json_and_uuid_columns_return_strings_that_look_numeric`](../tests/end2end_sqlite3.rb#test_json_and_uuid_columns_return_strings_that_look_numeric) | | -| PostgreSQL `PREPARE` interrupted (Timeout / `Thread#raise`, an exception raised by it) after the server ran it leaves the statement usable | [`test_an_interrupt_during_prepare_leaves_the_statement_usable`](../tests/end2end_pg.rb#test_an_interrupt_during_prepare_leaves_the_statement_usable), [`test_a_statement_left_prepared_by_a_failed_prepare_is_used`](../tests/end2end_pg.rb#test_a_statement_left_prepared_by_a_failed_prepare_is_used) | not MySQL (no server-side prepared statements) | not SQLite | | + +## Logging + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| Logger debug format, transaction logs, secret-free parameters | [`test_logger_receives_query_names_and_durations`](../tests/end2end_pg.rb#test_logger_receives_query_names_and_durations) | [`test_logger_receives_query_names_and_durations`](../tests/end2end_mysql2.rb#test_logger_receives_query_names_and_durations) | [`test_logger_receives_query_names_and_durations`](../tests/end2end_sqlite3.rb#test_logger_receives_query_names_and_durations) | [`test_logger_support`](../test/features_test.rb#test_logger_support) | + +## Generation errors + +| Documented behavior | PostgreSQL | MySQL | SQLite | Unit-only checks | +|---|---|---|---|---| +| Duplicate result columns, unsupported options, invalid `:copyfrom` clauses and generation errors | E2E cannot reach generation-time rejection | E2E cannot reach generation-time rejection | E2E cannot reach generation-time rejection | [`test_duplicate_result_columns_are_rejected`](../test/features_test.rb#test_duplicate_result_columns_are_rejected), [`test_copyfrom_keeps_or_rejects_conflict_clauses`](../test/features_test.rb#test_copyfrom_keeps_or_rejects_conflict_clauses), [`test_unknown_options_are_rejected_with_a_hint`](../test/options_test.rb#test_unknown_options_are_rejected_with_a_hint), [`test_query_names_cannot_replace_generated_methods`](../test/features_test.rb#test_query_names_cannot_replace_generated_methods) | +| SQLite parameters sqlc does not report (`ON CONFLICT ... DO UPDATE SET`) are generation errors | not PostgreSQL | not MySQL | unit: rejected at generation | [`test_sqlite_parameters_sqlc_does_not_report_are_generation_errors`](../test/features_test.rb#test_sqlite_parameters_sqlc_does_not_report_are_generation_errors), [`test_sqlite_placeholders_in_literals_and_comments_are_not_parameters`](../test/features_test.rb#test_sqlite_placeholders_in_literals_and_comments_are_not_parameters) | diff --git a/docs/Design.md b/docs/Design.md deleted file mode 100644 index 585a1e5..0000000 --- a/docs/Design.md +++ /dev/null @@ -1,51 +0,0 @@ -# Design: a sqlc plugin written in Ruby, shipped as WASM - -sqlc plugins receive a protobuf `GenerateRequest` on stdin and answer with a `GenerateResponse` on stdout, either as a -process or as a WASI module. This plugin is pure Ruby and is distributed as a WASI module (`plugin.wasm`). - -## Source layout -What each file in `lib/sqlc_gen_ruby/` does is in [Architecture.md](Architecture.md). Two constraints shape it: -* `google-protobuf` is a C extension and cannot be loaded inside ruby.wasm, so the plugin brings its own ~150 line - protobuf codec (`proto.rb`) and the subset of sqlc's `codegen.proto` it reads (`plugin_pb.rb`, field numbers must - match upstream). -* Driver differences are data, not copy/paste: `Drivers::Base` defines hooks (how to prepare and run a statement, how - rows are read - `row_data_type` `:hash` for `row['col']` vs `:array` for `row[0]` -, how `:execlastid` / - `:execresult` / `:copyfrom` / transactions work) and a column mapping table; `Pg`, `Mysql2` and `Sqlite3` fill them in. - -## The WASM build (`scripts/wasm/build.sh`) - -1. downloads the *prebuilt* `ruby-3.3-wasm32-unknown-wasip1-full` release from ruby/ruby.wasm (this avoids compiling - CRuby with a wasi-sdk); -2. strips the ~18 MB of DWARF debug info / name section from the interpreter (`strip_wasm.rb`); -3. stages a slim stdlib (the plugin only uses core + `json`) + `lib/` + `exe/`, plus an empty - `syntax_suggest/core_ext.rb` (the interpreter loads it to format errors and would otherwise print a warning before - the plugin's error message); -4. embeds that filesystem into the interpreter with `rbwasm pack` (wasi-vfs), because sqlc gives plugins no filesystem. - -**The argv trick:** sqlc runs the module as `plugin.wasm /plugin.CodegenService/Generate`, so the interpreter treats that -path as its script. The build embeds a file at exactly that path containing `load '/app/exe/plugin.rb'`. - -## Verification -How the plugin is tested (golden files from real sqlc requests, `sqlc diff` over `examples/`, end-to-end tests -against real databases) is described in [Testing.md](Testing.md); how it works inside in [Architecture.md](Architecture.md). - -## What the generated code looks like (and why) -Generated code follows what Ruby code usually looks like rather than mirroring the C# plugin: keyword arguments instead of -parameter objects, `Data` rows, implicit returns, `result.map { ... }`, 2-space indentation, a frozen-string-literal -header, SQL in raw heredocs, `connection_pool` for pooling with an injectable pool, nested transactions via savepoints. -PostgreSQL prepared statements are named per query class and remembered per *connection* (a module mixed into each -connection), so several query objects can share a pool. MySQL quotes parameters client-side and sends queries through -`Mysql2::Client#query` rather than server-side prepared statements. - -## History -The plugin started as a port of the C# `sqlc-gen-csharp` plugin and reproduced its output byte for byte (verified -against the C# binary; see the git history before the commit that removed the C# code). It has since gained the sqlc features of -that plugin (`:execrows`, `:execlastid` for every driver, `:copyfrom`, `sqlc.slice`, `sqlc.embed`, enums, models, -transactions, nullability, `overrides`) and Ruby conventions (bang finders, logger, pooling), so the output is -now idiomatic Ruby rather than a copy of the C# one. - -## Known limitations -See [Limitations in the README](../README.md#limitations). Design choices that look like limitations: -* Overrides only change the RBS types; values are never converted, so the generated code stays a thin layer over the driver. -* SQLite queries do not overlap: a `SQLite3::Database` is one connection, so the generated code takes one lock per - database (held for a whole transaction) to keep threads and fibers from mixing their transactions. diff --git a/docs/Examples.md b/docs/Examples.md index 024acc1..97eb154 100644 --- a/docs/Examples.md +++ b/docs/Examples.md @@ -1,9 +1,17 @@ # Examples -## Engine `postgresql`: [examples/pg](../examples/pg) -### [Schema](../examples/authors/postgresql/schema.sql) | [Queries](../examples/authors/postgresql/query.sql) | [End2End Test](../tests/end2end_pg.rb) +The same example application for each database: a schema, queries that use every supported feature, and the code +the plugin generates from them. The generated code is committed, and the end-to-end tests run against it. -### Config +In each generated directory, `query_sql.rb` has the query methods, `models.rb` the table and enum models, +`connection.rb` the shared helpers, and the `.rbs` files their signatures +(see [The generated code](Generated-Code.md)). + +## PostgreSQL (`pg`) + +[Schema](../examples/authors/postgresql/schema.sql) · [Queries](../examples/authors/postgresql/query.sql) · [Generated code](../examples/pg) · [End-to-end tests](../tests/end2end_pg.rb) + +Options: ```yaml driver: pg rubyVersion: "3.3" @@ -11,11 +19,11 @@ generateTypes: true generateGemfile: false ``` -## Engine `mysql`: [examples/mysql2](../examples/mysql2) +## MySQL (`mysql2`) -### [Schema](../examples/authors/mysql/schema.sql) | [Queries](../examples/authors/mysql/query.sql) | [End2End Test](../tests/end2end_mysql2.rb) +[Schema](../examples/authors/mysql/schema.sql) · [Queries](../examples/authors/mysql/query.sql) · [Generated code](../examples/mysql2) · [End-to-end tests](../tests/end2end_mysql2.rb) -### Config +Options: ```yaml driver: mysql2 rubyVersion: "3.3" @@ -23,11 +31,11 @@ generateTypes: true generateGemfile: false ``` -## Engine `sqlite`: [examples/sqlite3](../examples/sqlite3) +## SQLite (`sqlite3`) -### [Schema](../examples/authors/sqlite/schema.sql) | [Queries](../examples/authors/sqlite/query.sql) | [End2End Test](../tests/end2end_sqlite3.rb) +[Schema](../examples/authors/sqlite/schema.sql) · [Queries](../examples/authors/sqlite/query.sql) · [Generated code](../examples/sqlite3) · [End-to-end tests](../tests/end2end_sqlite3.rb) -### Config +Options: ```yaml driver: sqlite3 rubyVersion: "3.3" diff --git a/docs/Generated-Code.md b/docs/Generated-Code.md new file mode 100644 index 0000000..8a4fdfc --- /dev/null +++ b/docs/Generated-Code.md @@ -0,0 +1,107 @@ +# The generated code + +What `sqlc generate` writes, and how the classes, methods and attributes are named. +The real output for every database is in [Examples](Examples.md). + +## Files + +| File | Contents | +|---|---| +| `query_sql.rb` | one per query file: the `QuerySql` class (one method per query) and the row classes of its queries. The names follow the file: `admin_users.sql` → `AdminUsersSql` in `admin_users_sql.rb` | +| `models.rb` | one class per table, one module of constants per enum | +| `connection.rb` | helpers the query classes share | +| `*.rbs` | signatures for each of the above (unless `generateTypes: false`) | +| `Gemfile` | the gems the code needs (only with `generateGemfile: true`) | + +Everything lives in one module: `PgCodegen`, `Mysql2Codegen` or `Sqlite3Codegen`, or the +[`moduleName`](Options.md) you set. + +## The query class + +```ruby +queries = PgCodegen::QuerySql.connect({ dbname: 'app' }, size: 5) # see Connections + +queries.get_author(id: 1) # => # or nil +queries.get_author!(id: 1) # raises PgCodegen::QuerySql::NotFound instead of returning nil +queries.list_authors # => [#, ...] +``` + +* One method per query, named after it in snake_case. What it returns depends on the query's annotation: see + [Queries](Queries.md). +* Arguments are keywords: see [Arguments](Queries.md#arguments). +* Every `:one` query also gets a bang method (`get_author!`) that raises `QuerySql::NotFound` when there is no row. +* `transaction`, `connect` and `new` are described in [Transactions](Transactions.md) and [Connections](Connections.md). + +## Rows and models + +A query returns `Row` objects (`GetAuthorRow`); `sqlc.embed` and `models.rb` use one class per table (`Author`). + +| `rubyVersion` | Rows are | | +|---|---|---| +| 3.2 and later | `Data` | frozen; `with` and `to_h` | +| below 3.2 | keyword `Struct` | mutable | + +Both support pattern matching: + +```ruby +case queries.get_author(id: 1) +in { id:, name: } then puts "#{id}: #{name}" +in nil then puts 'no such author' +end +``` + +## Names + +**Methods and arguments** + +* Query names and parameters are snake_cased, with acronyms kept together (`MIN_ID` → `min_id`, `HTTPServer` → + `http_server`). +* A Ruby keyword gets a trailing `_` as an argument (`end_:`). +* A parameter used twice is one keyword when it is named with `sqlc.arg`; repeated `?` / `$n` of one column become + `id:`, `id_2:`. +* A query whose name would replace a generated method (`Transaction`, `Initialize`, ...) is a generation error. + +**Row attributes** + +* An attribute has the column's name as sqlc reports it: PostgreSQL and SQLite keep `"camelCase"` as `camelCase`, MySQL + reports `camelcase`. +* A result column without a usable name (`SELECT 1, now()`) is named by its position: `column_1`, `column_2`, ... +* A name that would replace a method of the row gets a trailing `_` (`class_`, `hash_`, `initialize_`). `Struct` rows + have more methods than `Data` rows, so a column `count` is `count` from `rubyVersion` 3.2 and `count_` below: check + the generated `.rbs` when you change `rubyVersion`. +* Keywords are fine as attributes: a column `end` is the attribute `end`, but the argument `end_:`. + +**Models** + +* A table's model is its singular name (`authors` → `Author`). A table `authors` next to a table `author` keeps the + plural (`Authors`). Change this with [`rename` or `exactTableNames`](Options.md). +* A model may have the name of a Ruby or driver constant (tables `exceptions`, `data`, `threads`, `sets`, ...). The + generated code refers to the originals as `::Exception`, `::Data`, `::Thread`, ..., so the models never shadow them. +* SQLite: sqlc before 1.30 reports a quoted table name with its quotes, so `CREATE TABLE "Items"` gives the model + `Items` there and `Item` from sqlc 1.30 on. Use `rename` or `exactTableNames` to keep the name when you upgrade sqlc. + +Enum constants are described in [Types](Types.md#enums). + +## Logging + +Pass a logger (anything with `#debug`) and every query and transaction is logged at debug level with its name and +duration, for example `PgCodegen::QuerySql#get_author (0.4ms)`: + +```ruby +PgCodegen::QuerySql.connect(params, logger: Rails.logger) +PgCodegen::QuerySql.new(pool, logger: Rails.logger) +``` + +Parameter values are never logged. + +## Style and static checks + +* The SQL of each query is a frozen heredoc constant (`GET_AUTHOR_SQL`). +* A comment above a query becomes the doc comment of its method. Table, column and enum comments of the schema become + doc comments of the models. +* **RuboCop:** the code is RuboCop-clean. Where a name from your schema breaks a naming rule (`"UserId"`, `naïve`), + that class or method gets a `rubocop:disable`. Query names such as `HasAuthors` / `IsActive` keep their name + (`has_authors`), with `Naming/PredicatePrefix` disabled. +* **Steep:** the `.rbs` files type-check with Steep. Add the standard libraries the code uses to your `Steepfile`: + `library "monitor"` for SQLite, and `library "timeout"` for a `rubyVersion` below 3.3. This repository's + [`Steepfile`](../Steepfile) shows the full list for all three drivers. diff --git a/docs/Options.md b/docs/Options.md new file mode 100644 index 0000000..1449ef8 --- /dev/null +++ b/docs/Options.md @@ -0,0 +1,70 @@ +# Options + +All options go in the `options:` of a `codegen:` entry in `sqlc.yaml`: + +```yaml +codegen: + - plugin: ruby + out: app/db + options: + driver: pg + rubyVersion: "3.3" + moduleName: MyApp::Db +``` + +| Option | Default | What it does | +|---|---|---| +| `driver` | required | `pg`, `mysql2` or `sqlite3`. Must match the sqlc `engine:`. | +| `rubyVersion` | `3.3` | Oldest Ruby the code must run on, `3.1` – `4.0`. Rows are `Data` classes from 3.2 and `Struct`s below; `3` means any 3.x, so it gets what Ruby 3.1 needs (`Struct`s, the [Timeout handling](Transactions.md#ruby-below-33-with-timeout--04) below 3.3). | +| `generateTypes` | `true` | Also generate `.rbs` type signatures. | +| `generateGemfile` | `false` | Also generate a `Gemfile` with the gems the code needs. | +| `moduleName` | `Codegen` | The module the code lives in, e.g. `MyApp::Db`. | +| `overrides` | none | Change the type of columns / parameters, see [Overrides](#overrides). | +| `rename` | none | Rename tables (model class) and columns (attribute), e.g. `{authors: Writer, bio: biography}`; `schema.table` works too. See [Names](Generated-Code.md#names). | +| `exactTableNames` | `false` | Keep table names as model names (`authors` → `Authors`) instead of singularizing them (`Author`). | +| `preparedStatements` | `true` | `pg` only (an error with other drivers). See [below](#preparedstatements-pg). | + +Unknown or misspelled options are errors, with a hint for near misses (`generate_types` → `generateTypes`). + +## Sharing options between `sql:` blocks + +Options used by several `sql:` blocks can go in the top-level `options:` of `sqlc.yaml`, under the plugin's name: + +```yaml +options: + ruby: + moduleName: MyApp::Db +``` + +A codegen entry's own options win, and the `overrides` of both apply. + +## `preparedStatements` (pg) + +`false` sends the SQL with every call instead of preparing it (needed behind pgbouncer in transaction mode). + +With the default (`true`): + +* Statements dropped by `DEALLOCATE ALL` / `DISCARD ALL`, and statements PostgreSQL refuses after a migration changed + the type of a column they return (`PG::FeatureNotSupported`: cached plan must not change result type), are prepared + again by their next call outside a transaction. Inside a transaction, that call raises and the transaction rolls + back; the next call prepares the statement again. +* Statements are named `Module::Class#method` (with a hash when longer than PostgreSQL's 63 bytes), so classes sharing + a pool never run each other's SQL. + +## Overrides + +```yaml +overrides: + - column: "GetAuthor:bio" # one column of one query (:) + ruby_type: { type: "Symbol", notNull: true } + - column: "authors.bio" # a table column, in its model and in every query (.
. too) + ruby_type: { type: "Symbol" } + - column: "*:bio" # a column name in every query and model + ruby_type: { type: "Pathname", notNull: false } + - db_type: "uuid" # a database type, everywhere (also the elements of uuid[] arrays) + ruby_type: { type: "String" } +``` + +The first match wins in this order: `:`, `
.`, `*:`, `db_type`. +Overrides only change the declared (RBS) types: values are never converted and stay what the driver returns (see +[Types](Types.md)). Unknown keys are errors. diff --git a/docs/Queries.md b/docs/Queries.md new file mode 100644 index 0000000..2b292ef --- /dev/null +++ b/docs/Queries.md @@ -0,0 +1,88 @@ +# Queries + +Each query in your `.sql` files becomes one method. Its annotation (`-- name: GetAuthor :one`) decides what the +method returns. + +## Annotations + +| Annotation | Returns | Notes | +|---|---|---| +| `:one` | the row or `nil` | also generates `get_author!`, which raises `QuerySql::NotFound` instead of returning `nil` | +| `:many` | `Array` of rows | | +| `:exec` | `nil` | | +| `:execrows` | number of affected rows | MySQL counts rows changed, not rows matched; connect with `flags: Mysql2::Client::FOUND_ROWS` to count matched rows | +| `:execlastid` | the new id | PostgreSQL / SQLite: the query needs `RETURNING id`. MySQL: the auto-increment id | +| `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | `last_insert_id` is `nil` on PostgreSQL; use `RETURNING` there | +| `:batchexec` | `nil` | see [Batches](#batches) | +| `:batchone` | `Array` of rows or `nil` | | +| `:batchmany` | `Array` of `Array`s of rows | | +| `:copyfrom` | `nil` | see [`:copyfrom`](#copyfrom) | + +## Arguments + +```ruby +queries.get_author(id: 1) +queries.create_author(name: 'Seuss', bio: nil) +``` + +* Arguments are keywords named after the parameters. Name one yourself with `sqlc.arg(name)`; a parameter sqlc cannot + name (`ANY($1::BIGINT[])`) becomes `param1`, `param2`, ... +* Missing or unknown keywords raise `ArgumentError`. +* A nullable parameter named with `sqlc.arg` / `sqlc.narg` is optional and defaults to `nil`. Other nullable + parameters (`VALUES ($1, $2)`) must be passed, even as `nil`. + +## Macros + +| Macro | PostgreSQL | MySQL | SQLite | What it does | +|---|:---:|:---:|:---:|---| +| `sqlc.arg(name)` | ✅ | ✅ | ✅ | names a parameter | +| `sqlc.narg(name)` | ✅ | ✅ | ✅ | names a nullable parameter (`T?`, defaults to `nil`) | +| `sqlc.slice(ids)` | 🚫 | ✅ | ✅ | `IN (sqlc.slice(ids))` takes an `Array`. With an empty one, `IN` matches nothing and `NOT IN` everything | +| `sqlc.embed(table)` | ✅ | ✅ | ✅ | the row gets an attribute holding the table's model (`author`; `author2` when the table is embedded twice). It is `nil` when an outer join found no row | + +🚫 sqlc does not offer `sqlc.slice` for PostgreSQL. Use an array parameter instead: `WHERE id = ANY($1::bigint[])`. + +Two result columns with the same name (`a.id, b.id`) are a generation error: alias one (`b.id AS book_id`) or use +`sqlc.embed`. + +## Batches + +`:batchexec`, `:batchone` and `:batchmany` take an `Array` of `Args` and run the query once per element, on one +connection: + +```ruby +queries.create_authors_batch([ + PgCodegen::CreateAuthorsBatchArgs.new(name: 'a', bio: nil), + PgCodegen::CreateAuthorsBatchArgs.new(name: 'b', bio: nil) +]) +``` + +A batch is not atomic on its own: wrap it in [`transaction`](Transactions.md) for all-or-nothing. + +## `:copyfrom` + +A bulk insert of an `Array`, or any `Enumerable` (also a lazy one), of `Args`. Either all rows are inserted or +none. + +| Database | Implementation | Requirements | +|---|---|---| +| PostgreSQL | `COPY ... FROM STDIN` | | +| MySQL | `LOAD DATA LOCAL INFILE` | `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the server | +| SQLite | batched multi-row `INSERT`s | | + +**Conflict clauses** + +| Database | Kept | Generation error (use `:batchexec` instead) | +|---|---|---| +| PostgreSQL | | `ON CONFLICT`, `RETURNING` | +| MySQL | `INSERT IGNORE` (duplicates are skipped), `REPLACE` | `ON DUPLICATE KEY UPDATE` | +| SQLite | `INSERT OR REPLACE`, `INSERT OR IGNORE`, `ON CONFLICT ... DO ...` | | + +**MySQL notes** + +* Without `INSERT IGNORE`, duplicate keys and invalid values raise `Mysql2::Error` instead of being skipped or coerced + as `LOAD DATA` would do by itself. +* `LOAD DATA` commits a transaction that was opened with raw SQL (`BEGIN`). Call `:copyfrom` inside + [`transaction`](Transactions.md) instead. +* Table and column names are written as in your `INSERT` (`` `MixedCase` ``), so mixed-case tables work on Linux, where + MySQL table names are case-sensitive. diff --git a/docs/Testing.md b/docs/Testing.md index 8ce6c1f..0459f7f 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -1,67 +1,109 @@ # Testing and CI -Every README claim and supported feature needs a real-database E2E test for each applicable engine, listed in -[`Coverage.md`](Coverage.md). A bug fix starts with a real-database E2E regression that fails before the fix. Use unit -tests only where E2E cannot reach the behavior, such as option validation or generation errors. CI runs the full E2E -suites on every supported Ruby version plus full-suite option variants; generator-options E2E runs on Ruby 3.1 and 4.0. +## The rules -There are four layers. Each one answers a different question, and each runs in CI. +* Every documented behavior (README and `docs/`) needs an end-to-end test against a real database, for each database + it applies to, and a row in [Coverage](Coverage.md). +* A bug fix starts with an end-to-end test that fails before the fix. +* Unit tests are for what end-to-end tests cannot reach: option validation and generation errors. -| Layer | Command | Needs | Question it answers | +## The layers + +Each layer answers a different question, and each runs in CI. + +| Layer | Command | Needs | Question | |---|---|---|---| | Unit + golden | `bundle exec rake test` | nothing | does the plugin produce the expected files? | | Generated-code checks | `bundle exec rake rubocop steep` | nothing | is the generated code clean Ruby with correct types? | -| `sqlc diff` | `rake sqlc_generate_process` / `sqlc_generate_wasm` | sqlc | do the committed `examples/` match the plugin, through the real sqlc and the real WASM? | -| End-to-end | `rake end2end_sqlite3` / `end2end_pg` / `end2end_mysql2` | databases | does the *generated code* behave correctly against a real database? | - -## Unit and golden tests (`test/`) -* `golden_test.rb` runs the plugin on **real requests captured from sqlc** (`test/fixtures/*.bin`) and compares with - the checked-in `examples/`. `extended_golden_test.rb` does the same for option variants (Ruby 3.1/3.2, no types, - `generateGemfile`, ...) and synthetic requests (`synthetic_requests.rb`) against `test/golden/`. - After an intended output change: `UPDATE_GOLDEN=1 bundle exec ruby -Itest test/extended_golden_test.rb`, then **read the diff**. -* `features_test.rb` / `options_test.rb`: one focused test per feature or option. -* `codec_test.rb` cross-checks the hand-written protobuf codec against Google's `google-protobuf` (test-only gem). -* Refreshing a fixture after a new sqlc version or schema change: `scripts/capture_request.rb` (see its header). - -## `examples/` is both documentation and a test -`examples/authors//{schema,query}.sql` are the inputs; `examples//` is the plugin's output, committed. -CI runs `sqlc diff` against it, so a plugin change that is not accompanied by regenerated examples fails the build. +| `sqlc diff` | `sqlc -f sqlc.local.yaml diff` | sqlc | do the committed `examples/` match what the plugin generates now? | +| End-to-end | `rake end2end_sqlite3 end2end_pg end2end_mysql2` | databases | does the generated code behave correctly against a real database? | + +### Unit and golden tests (`test/`) + +| File | What it checks | +|---|---| +| `golden_test.rb` | runs the plugin on real requests captured from sqlc (`test/fixtures/*.bin`) and compares with the committed `examples/` | +| `extended_golden_test.rb` | the same for option variants (Ruby 3.1 / 3.2, no types, `generateGemfile`, ...) and synthetic requests (`synthetic_requests.rb`), against `test/golden/` | +| `features_test.rb`, `options_test.rb` | one focused test per feature or option | +| `codec_test.rb` | the hand-written protobuf codec against Google's `google-protobuf` (a test-only gem) | +| `coverage_map_test.rb` | every test linked from [Coverage](Coverage.md) exists | + +* After an intended output change, rewrite the golden files and **read the diff**: + `UPDATE_GOLDEN=1 bundle exec ruby -Itest test/extended_golden_test.rb` +* After a new sqlc version or a schema change, refresh the fixtures with `scripts/capture_request.rb` (see its header). + +### `examples/` and `sqlc diff` + +`examples/authors//{schema,query}.sql` are the inputs. `examples//` is the plugin's output, committed. +CI runs `sqlc diff` against it with both the process plugin and the WASM, so a plugin change without regenerated +examples fails the build. + Regenerate with `rake sqlc_generate_process` (pure Ruby, fast) or `rake sqlc_generate_wasm` (builds the WASM first). -## End-to-end tests (`tests/`) -They `require` the generated code from `examples/` and run it against a real database, with Minitest. -`rbs_types.rb` additionally checks that returned values match the generated `.rbs` types. -* SQLite needs nothing: `rake end2end_sqlite3`. -* PostgreSQL / MySQL: provide a server, load `examples/authors//schema.sql`, and set `DB_NAME DB_USER DB_PASS - POSTGRES_HOST MYSQL_HOST` (optionally `*_PORT`). MySQL needs `local_infile=1` for `:copyfrom`, and the MySQL tests - connect without a password (`DB_PASS` is only used for PostgreSQL), so use a user without one. - Or run everything in Docker: `rake run_end2end_tests` (docker-compose). -* They cover every query command (batches with encoded `bytea` / `json` values too), transactions (nested, - isolation, `break` / `throw`, a rescued failed statement on PostgreSQL, SQLite threads sharing a database, the - yielded `tx` used from other fibers / threads), models named like Ruby constants (`Exception`, `Data`, ...), pool - injection, logging, bang finders, row members named like row methods (`initialize`), MySQL changed vs. matched - rows, SQLite busy waiting, `:copyfrom` with awkward and binary values, lazy enumerators and its all-or-nothing behaviour, - PostgreSQL `PREPARE`s interrupted after the server ran them, SQLite JSON / UUID text that looks numeric, - Fiber-scheduler concurrency (`async` gem) and statement-leak checks. +### End-to-end tests (`tests/`) + +Minitest suites that `require` the generated code from `examples/` and run it against a real database. +`rbs_types.rb` also checks that returned values match the generated `.rbs` types. + +| Suite | Needs | +|---|---| +| `rake end2end_sqlite3` | nothing | +| `rake end2end_pg` | a PostgreSQL server with `examples/authors/postgresql/schema.sql` loaded | +| `rake end2end_mysql2` | a MySQL server with `examples/authors/mysql/schema.sql` loaded and `local_infile=1` (for `:copyfrom`) | +| `tests/end2end_options.rb` | sqlc and PostgreSQL; generates code with other options and runs it (PostgreSQL, SQLite) | +| `rake run_end2end_tests` | Docker: starts the servers with docker-compose and runs the suites | + +The servers are configured with `DB_NAME`, `DB_USER`, `DB_PASS`, `POSTGRES_HOST`, `MYSQL_HOST` (and optionally +`*_PORT`). The MySQL tests connect without a password (`DB_PASS` is only used for PostgreSQL), so use a MySQL user +that has none. + +What each test proves is listed in [Coverage](Coverage.md). ## CI workflows (`.github/workflows/`) -| Workflow | When | What | + +Every job has `timeout-minutes: 10`. Pull requests need `build.yml` and `tests.yml` green; both skip Markdown-only +changes. + +| Workflow | Runs on | What | |---|---|---| -| `build.yml` | push to `main`, pull requests (not for Markdown-only changes), release | build `plugin.wasm`; lint; `sqlc diff` with the WASM and the process plugin; Steep; generate with option combinations (Ruby 3.1/3.2/3.3/4.0 × types/Gemfile) and check the output with the matching Ruby | -| `tests.yml` | same | unit tests; plugin compatibility on Ruby 3.1 / 3.2 / 3.4 / 4.0; full E2E matrix on Ruby 3.1 / 3.2 / 3.3 / 3.4 / 4.0; Ruby 4.0 module-name variants for MySQL and SQLite and module-name + no-prepared-statements variant for PostgreSQL; a Ruby 3.1 PostgreSQL variant with Ruby's bundled timeout 0.2.0 (`E2E_TIMEOUT_GEM`, `Gemfile.e2e`); PostgreSQL 16.2, MySQL 8.3.0, and SQLite; generator-options E2E on Ruby 3.1 / 4.0 | -| `cross-platform.yml` | called by `build.yml` after `build-wasm` | macOS and Windows × Ruby 3.1 / 4.0 (oldest and newest supported), as two parallel jobs each. *Codegen*: unit + golden tests; `sqlc diff` with the Linux-built `plugin.wasm` and with the process plugin (`exe\plugin.cmd` on Windows); WASM codegen check. *End-to-End*: full E2E for SQLite, PostgreSQL (16 on macOS, 17 on Windows) and MySQL 8.4, installed natively (`ankane/setup-postgres`, `ankane/setup-mysql`, the image's MySQL on Windows), using the generated Gemfiles (installed gems cached per runner image); generator-options E2E | -| `docs.yml` | `examples/` change on `main`, manual | regenerates `docs/Examples.md` and opens a PR | -| `release.yml` | manual only | see [Releasing](Releasing.md) | +| `build.yml` | pull requests, pushes to `main`, releases | builds `plugin.wasm`; lint; `sqlc diff` with the WASM and with the process plugin; Steep; the option matrix; then calls `cross-platform.yml` | +| `tests.yml` | the same | unit tests; plugin compatibility; the end-to-end matrix | +| `cross-platform.yml` | called by `build.yml` | the Linux-built WASM, the process plugin and all end-to-end suites on macOS and Windows | +| `docs.yml` | changes of `examples/authors/` or `sqlc.ci.yaml` on `main`, manually | regenerates `docs/Examples.md` and opens a pull request | +| `release.yml` | manually | see [Releasing](Releasing.md) | + +**`build.yml` option matrix:** generates with combinations of `rubyVersion` (3.1, 3.2, 3.3, 4.0), `generateTypes` and +`generateGemfile`, and checks the output with the matching Ruby. + +**`tests.yml`:** -Tool versions (sqlc, ruby.wasm) live in `scripts/tool-versions.env`; dependabot updates actions and gems weekly. +| Job | Matrix | +|---|---| +| Unit tests | Ruby 3.3 | +| Plugin compatibility | the plugin itself on Ruby 3.1, 3.2, 3.4, 4.0 | +| End-to-end | Ruby 3.1, 3.2, 3.3, 3.4, 4.0 × PostgreSQL 16.2, MySQL 8.3.0, SQLite | +| End-to-end variants | Ruby 4.0 with a custom `moduleName` (all three), plus `preparedStatements: false` on PostgreSQL; Ruby 3.1 on PostgreSQL with Ruby's bundled timeout 0.2.0 (`E2E_TIMEOUT_GEM`, `Gemfile.e2e`) | +| Generator options end-to-end | Ruby 3.1 and 4.0 | + +**`cross-platform.yml`:** macOS and Windows × Ruby 3.1 and 4.0 (the oldest and newest supported), two jobs each. + +* *Codegen:* unit and golden tests; `sqlc diff` with the Linux-built `plugin.wasm` and with the process plugin + (`exe\plugin.cmd` on Windows); a WASM codegen check. +* *End-to-End:* all suites against natively installed databases (PostgreSQL 16 on macOS and 17 on Windows, MySQL 8.4, + SQLite), using the generated Gemfiles. + +Dependabot updates actions and gems weekly; tool versions (sqlc, ruby.wasm) are pinned in `scripts/tool-versions.env`. ## Before you push + ``` bundle exec rake test rubocop steep sqlc -f sqlc.local.yaml diff # committed examples must be current -scripts/tests/run_codegen.rb sqlc.local.yaml 3.1 true true -rake end2end_sqlite3 end2end_pg end2end_mysql2 # full Ruby 3.1–4.0 matrix, plus Ruby 4.0 variants -BUNDLE_GEMFILE=Gemfile.options.e2e bundle exec ruby tests/end2end_options.rb # Ruby 3.1 and 4.0 +scripts/tests/run_codegen.rb sqlc.local.yaml 3.1 true true # generate for Ruby 3.1 with types + Gemfile, check it +rake end2end_sqlite3 end2end_pg end2end_mysql2 +BUNDLE_GEMFILE=Gemfile.options.e2e bundle exec ruby tests/end2end_options.rb ``` -CI also runs the unit suite without Bundler on Ruby 3.1 and 4.0. For Ruby 3.1/3.2 E2E, generate scratch examples -with their matching `rubyVersion` and install the generated Gemfile before running the suite. + +To run the end-to-end suites for Ruby 3.1 / 3.2 locally, generate scratch examples with the matching `rubyVersion` +(`ruby scripts/tests/generate_examples.rb 3.1 sqlc.local.yaml --generate-gemfile`), point `EXAMPLES_DIR` at the +`.examples-ruby-3.1` directory it writes, and install the generated Gemfile first. diff --git a/docs/Transactions.md b/docs/Transactions.md new file mode 100644 index 0000000..5b827ce --- /dev/null +++ b/docs/Transactions.md @@ -0,0 +1,99 @@ +# Transactions + +```ruby +queries.transaction(isolation: :serializable) do |tx| + id = tx.create_author_return_id(name: 'a', bio: nil) + tx.transaction { tx.create_book(name: 'b', author_id: id) } # nested: a SAVEPOINT + :done # the block's value is returned +end +``` + +## The basics + +* Everything in the block runs on one connection. +* **Commit:** when the block ends, also with `break`, `return` or `throw`. +* **Rollback:** when the block raises or its thread is killed. +* **Nesting:** a `transaction` inside a transaction is a savepoint and rolls back only its own part. +* **Result:** `transaction` returns the value of the block. + +### Isolation + +| Driver | Keyword | Values | +|---|---|---| +| `pg`, `mysql2` | `isolation:` | `:read_uncommitted`, `:read_committed`, `:repeatable_read`, `:serializable` | +| `sqlite3` | `mode:` | `:deferred`, `:immediate`, `:exclusive` | + +Any other value raises `ArgumentError` before anything runs. So does `isolation:` / `mode:` on a nested `transaction`, +because a savepoint cannot change them. + +### Use `tx` inside the block + +`tx` is a copy of the query object that is bound to the transaction: + +* Its queries stay in the transaction from any fiber or thread (an `Enumerator`, an `Async` task, a `Thread`), one at + a time. +* After the transaction is over it raises `TransactionEnded`. + +The original `queries` object also joins the transaction when it is called from the thread that opened it (with +`connection_pool` or SQLite; a pool you inject must hand out the same connection again inside `#with`). From another +fiber or thread, `queries` and other query classes sharing the pool check out their own connection and run **outside** +the transaction, so pass `tx` there. + +### Inside another library's transaction + +When another library already opened a transaction on the same connection (`BEGIN`): + +* **PostgreSQL, SQLite:** `transaction` is a savepoint in it. +* **MySQL** has no nested transactions, and its `BEGIN` commits the open transaction first. A later rollback no longer + undoes the earlier work. Run the outer work in `transaction` too (inner calls then become savepoints), or finish + the other transaction first. + +## When something goes wrong + +The rule behind everything below: a connection never goes back to the pool inside a transaction, and `transaction` +never reports a commit that did not happen. + +### A statement fails and the block rescues the error + +* **PostgreSQL** aborts the whole transaction when a statement fails, so `transaction` raises + `PG::InFailedSqlTransaction` at the end instead of pretending it committed. To carry on after an error, run the + statement in a nested `transaction`: only that savepoint is rolled back, and the outer block can rescue the error. +* **MySQL** rolls the whole transaction back on a deadlock (or a lock wait timeout with + `innodb_rollback_on_timeout`), savepoints included. Every later query in the block and `transaction` itself then + raise the error again: a `Mysql2::Error` with the same `error_number` and a message that starts with + `transaction rolled back:`. Retry the whole `transaction`. A lock wait timeout without that setting fails only its + statement, and the block can rescue it and go on. + +### Interrupts + +`Timeout`, `Thread#raise`, `Thread#kill` and a stopped `Async` task can arrive at any moment: + +* **During `BEGIN`, `COMMIT` or `ROLLBACK`:** the interrupt waits for the statement to finish. A `BEGIN` / `COMMIT` + that raises itself (a stopped `Async` task) is rolled back. +* **During a query on PostgreSQL** (inside or outside a transaction): the query is cancelled at once. The connection + goes back to the pool idle, and a killed transaction rolls back without waiting for its query. +* **`Thread#kill` during a query on MySQL:** the client is closed, because mysql2 cannot send `ROLLBACK` while it + waits for the query's result. The server rolls the transaction back and the pool replaces the client. + +### Ruby below 3.3 with timeout < 0.4 + +The `timeout` gem bundled with Ruby before 3.3 leaves a block with `throw`, which cannot be told apart from your own +`break`, `return` or `throw`. With that gem all of them **roll back** instead of committing. + +Use `gem 'timeout', '>= 0.4'` to keep those exits committing. A generated Gemfile (`generateGemfile: true`) for a +`rubyVersion` below 3.3 already has it. + +With the old gem, a `Timeout` that interrupts a PostgreSQL query in a transaction still raises `Timeout::Error` (not +the `PG::InFailedSqlTransaction` of the cancelled query) after rolling back. + +### Lost connections + +A `pg` connection lost inside `transaction` is not reconnected. Every later call in the block and `transaction` itself +raise `PG::ConnectionBad`, so nothing after the loss runs outside the transaction; the server has rolled it back. + +Outside a transaction, lost connections are replaced: see [Connections](Connections.md#lost-connections). + +### SQLite + +A SQLite transaction belongs to the thread that opened it and holds the database until it ends. Other threads and +fibers sharing the same `SQLite3::Database` wait for it. diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md new file mode 100644 index 0000000..8f3e48e --- /dev/null +++ b/docs/Troubleshooting.md @@ -0,0 +1,53 @@ +# Troubleshooting + +Known errors and surprises, with their cause and fix. Most of them come from sqlc or from the database itself, not +from the plugin. + +## Any database + +| You see | Why | Do this | +|---|---|---| +| Generation error: two result columns with the same name (`a.id, b.id`) | a row has one member per name | alias one (`b.id AS book_id`) or use `sqlc.embed` | +| Generation error: a query named `Transaction`, `Initialize`, ... | it would replace a generated method | rename the query | +| Half the rows of a batch were written before an error | `:batchexec` / `:batchone` / `:batchmany` are not atomic on their own | wrap them in `transaction` | +| A timestamp comes back one hour off around a daylight-saving change | `timestamp` / `DATETIME` store the local wall-clock time, so the repeated hour is ambiguous | use `timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), see [Types](Types.md#dates-and-time-zones) | +| A huge `sqlc.slice` fails | database limits (see MySQL and SQLite below) | `:copyfrom` the values into a table and join it | +| `break` / `return` inside `transaction` rolls back | Ruby below 3.3 with its bundled `timeout` gem | `gem 'timeout', '>= 0.4'`, see [Transactions](Transactions.md#ruby-below-33-with-timeout--04) | +| Steep cannot find `Monitor` / `Timeout` | the generated code uses them (`Monitor` on SQLite, `Timeout` for a `rubyVersion` below 3.3) | add `library "monitor"` / `library "timeout"` to your `Steepfile` | + +## PostgreSQL + +| You see | Why | Do this | +|---|---|---| +| `sqlc.slice` is not available | sqlc does not offer it for PostgreSQL | `WHERE id = ANY($1::bigint[])` with an array | +| Prepared statement errors behind pgbouncer in transaction mode | statements are prepared per connection | `preparedStatements: false` ([Options](Options.md#preparedstatements-pg)) | +| Generation error in a `:copyfrom` query with `ON CONFLICT` / `RETURNING` | `COPY` cannot express them | use `:batchexec` | +| `last_insert_id` of `:execresult` is `nil` | PostgreSQL has no last-insert id | use `RETURNING` (`:one` or `:execlastid`) | + +## MySQL + +| You see | Why | Do this | +|---|---|---| +| A rollback did not undo work done before `transaction` | `transaction` on a client inside a transaction started with raw SQL or by another library commits it first (MySQL's `BEGIN` does) | run the outer work in `transaction` too (inner calls become savepoints), or finish the other transaction first, see [Transactions](Transactions.md#inside-another-librarys-transaction) | +| `Unknown column 'mixeditems.itemid'` with `sqlc.embed(MixedItems)` | sqlc lowercases the table name it expands to, and MySQL table names are case-sensitive on Linux; the plugin only receives the lowercased name | give the table an alias and embed that: `` SELECT sqlc.embed(m) FROM `MixedItems` AS m `` | +| `:copyfrom` is refused / fails | `LOAD DATA LOCAL INFILE` needs `local_infile` | `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the server, see [Queries](Queries.md#copyfrom) | +| A `sqlc.slice` fails with a packet error | the statement exceeds the server's `max_allowed_packet` | `:copyfrom` into a table and join it | +| `:execrows` counts changed rows, not matched rows | MySQL's default | connect with `flags: Mysql2::Client::FOUND_ROWS` | +| `:copyfrom` committed my raw-SQL transaction | `LOAD DATA` commits an open `BEGIN` transaction | call `:copyfrom` inside `transaction` | + +## SQLite + +| You see | Why | Do this | +|---|---|---| +| Queries lose their last characters (`FROM user` instead of `FROM users`) | sqlc (1.27 – 1.31 at least) misplaces the SQL of a query file that contains a non-ASCII character in a name, literal or comment. A sqlc bug, not a plugin one | keep SQLite query files ASCII and pass such values as parameters | +| Generation error with `sqlc.arg(...)` in `ON CONFLICT ... DO UPDATE SET` | sqlc does not see parameters there (it leaves `sqlc.arg(...)` in the SQL or reports fewer parameters than there are `?`) | use `excluded.` (`DO UPDATE SET n = counters.n + excluded.n`) or move the value into `VALUES` | +| `SQLite3::SQLException: too many SQL variables` | a statement takes at most 32,766 parameters (`SQLITE_MAX_VARIABLE_NUMBER`) | `:copyfrom` into a table and join it | +| An `Integer` where `Float` is declared | `NUMERIC` affinity: SQLite stores `5.0` as `5` | call `to_f` on the value, or declare the column `Numeric` with [`overrides`](Options.md#overrides) | +| Other threads freeze while the database is locked | `busy_timeout=` waits while holding Ruby's GVL | use `busy_handler_timeout=`, which waits without blocking them | +| A fiber or thread waits for a transaction | a SQLite transaction holds the database until it ends, and belongs to the thread that opened it | keep transactions short; inside one, use the yielded `tx` from other fibers / threads | +| The model is named `Items` instead of `Item` | sqlc before 1.30 (including the pinned 1.27) reports a quoted table name with its quotes | `rename` or `exactTableNames`, see [Names](Generated-Code.md#names) | + +## Coming from sqlc-gen-csharp + +These options of the C# plugin do not exist here: `withAsyncSuffix` / `withCancellationToken` (use the Fiber scheduler), `useDapper` and the target +framework / csproj options. `moduleName` replaces the namespace option. diff --git a/docs/Types.md b/docs/Types.md new file mode 100644 index 0000000..e2af99f --- /dev/null +++ b/docs/Types.md @@ -0,0 +1,93 @@ +# Types + +The generated code does not convert values: a column has the Ruby class its driver gem returns, and the `.rbs` +declares that class. Nullable columns and parameters are `T?`. + +To declare a different type for a column, use [`overrides`](Options.md#overrides). + +## Type mapping + +| SQL | PostgreSQL | MySQL | SQLite | +|---|---|---|---| +| integers | `Integer` | `Integer` (also `YEAR`) | `Integer` | +| `real`, `float`, `double` | `Float` | `Float` | `Float` | +| `numeric`, `decimal` | `BigDecimal` | `BigDecimal` | `Float` | +| `boolean` | `true` / `false` | `Integer` (0 / 1) | `Integer` (0 / 1) | +| text (`char`, `varchar`, `text`, ...) | `String` | `String` | `String` | +| binary (`bytea`, `blob`, `binary`, ...) | `String` | `String` | `String` | +| `date` | `Date` | `Date` | `String` | +| `timestamp`, `timestamptz`, `datetime` | `Time` | `Time` | `String` | +| `time` | `String` | `Time` | `String` | +| JSON | parsed (`untyped`) | `String` (JSON text) | `String` (JSON text) | +| `uuid` | `String` | | `String` | +| enums | `String` | `String` (`ENUM`, `SET`) | | +| arrays | `Array[T]` | | | +| `inet`, `cidr` | `IPAddr` | | | +| `bit` | `String` | `String` | | + +### PostgreSQL + +* **JSON:** `json` / `jsonb` columns are parsed. For a parameter, Hashes, Arrays, numbers, `true` / `false` and `nil` + are encoded for you. A `String` is taken as JSON text (`'{"a":1}'`), so pass `'"hi"'` or `'hi'.to_json` for a JSON + string. +* **Arrays** are `Array[T]`, and `Array[Array[T]]` for `T[][]`. Arrays of types the pg gem does not decode (`uuid[]`, + enum arrays, ...) are `Array[String]`, also for types created after the connection was opened (a migration that + runs while the application is up). +* **`String`:** `money`, `interval`, `xml`, `macaddr`, geometric, range, full-text, `hstore`, `ltree`, `timetz`, + `reg*` and composite types. +* **`untyped`:** domains and types the plugin does not know. + +### MySQL + +* `BOOLEAN` is `TINYINT(1)`, so it is an `Integer`. +* `JSON` columns and parameters are JSON text, as `mysql2` returns them. + +### SQLite + +SQLite has no fixed column types: the declared type only gives a column an +[affinity](https://www.sqlite.org/datatype3.html#determination_of_column_affinity), and the plugin follows the same +rules (`VARCHAR(255)` → `String`, `DECIMAL(10,5)` → `Float`, `UNSIGNED BIG INT` → `Integer`). + +* Dates, times and booleans are stored as text or numbers, so they are `String` or `Integer`. +* A `NUMERIC` / `DECIMAL` column is declared `Float`, but SQLite can return an `Integer` (it stores `5.0` as `5`). +* `JSON` and `UUID` columns are always `String`s, also when SQLite stored numeric-looking text as a number. The text + then comes back in SQLite's spelling: `'42'` → `"42"`, `'1.50'` → `"1.5"`. + +## Dates and time zones + +A `Time` or `DateTime` parameter keeps its instant whatever its UTC offset, also in `:copyfrom` and batches: + +* **PostgreSQL `timestamp`** (without time zone): the value is written in the process's local zone, the zone the pg + gem reads `timestamp` values back in. Elements of `timestamp[]` arrays are written as given. +* **PostgreSQL `timestamptz`:** a `DateTime` keeps its instant too. The RBS declares `::Time`, the class rows return. +* **MySQL `DATETIME` / `TIMESTAMP`:** the value is written in the zone `mysql2` reads values back in: the process's + local zone, or UTC for a client created with `database_timezone: :utc`. The server also converts `TIMESTAMP` columns + with the session `time_zone`, so keep that equal to the client's zone. + +**Daylight saving time.** A column without a time zone (PostgreSQL `timestamp`, MySQL `DATETIME`) stores the local +wall-clock time. When the local zone falls back from daylight saving time, one hour happens twice (02:30 on the last +Sunday of October in Europe/Berlin), and an instant in the second of the two reads back as the first, one hour off. +Use `timestamptz` / `TIMESTAMP`, or run the process in UTC (`TZ=UTC`), to keep every instant. + +## Enums + +An enum column is a `String`. `models.rb` has a module of constants for each enum: + +```ruby +PgCodegen::ExtendedBioType::MEMOIR # => "Memoir" +PgCodegen::ExtendedBioType::ALL # => ["Autobiography", "Biography", "Memoir"] +``` + +On PostgreSQL the module is named after the enum type. MySQL enums belong to a column, so the module is named after +the table and column: `Mysql2Codegen::ExtendedBiosBioType`. A MySQL `SET` column gets constants the same way. + +Values that are not valid constant names still get a constant: + +| Value | Constant | +|---|---| +| `'end'` | `END_` | +| `'all'` | `ALL_` | +| `'2x'` | `VALUE_2X` | +| `'a b'` | `A_B` | + +Values that end up with the same name get `_2`, `_3`, ... diff --git a/scripts/docs/generate_examples.rb b/scripts/docs/generate_examples.rb index 767838f..f27f9ab 100644 --- a/scripts/docs/generate_examples.rb +++ b/scripts/docs/generate_examples.rb @@ -8,6 +8,8 @@ doc_file = ARGV.fetch(0) config = YAML.safe_load_file(File.expand_path('../../sqlc.ci.yaml', __dir__)) +ENGINE_NAMES = { 'postgresql' => 'PostgreSQL', 'mysql' => 'MySQL', 'sqlite' => 'SQLite' }.freeze + def yaml_value(value) = value.is_a?(String) && value.match?(/\A[\d.]+\z/) ? %("#{value}") : value sections = config['sql'].map do |sql| @@ -15,11 +17,12 @@ def yaml_value(value) = value.is_a?(String) && value.match?(/\A[\d.]+\z/) ? %("# driver = options['driver'] yaml = options.map { |key, value| "#{key}: #{yaml_value(value)}" }.join("\n") [ - "## Engine `#{sql['engine']}`: [examples/#{driver}](../examples/#{driver})", + "## #{ENGINE_NAMES.fetch(sql['engine'])} (`#{driver}`)", '', - "### [Schema](../#{sql['schema']}) | [Queries](../#{sql['queries']}) | [End2End Test](../tests/end2end_#{driver}.rb)", + "[Schema](../#{sql['schema']}) · [Queries](../#{sql['queries']}) · [Generated code](../examples/#{driver}) · " \ + "[End-to-end tests](../tests/end2end_#{driver}.rb)", '', - '### Config', + 'Options:', '```yaml', yaml, '```', @@ -27,4 +30,16 @@ def yaml_value(value) = value.is_a?(String) && value.match?(/\A[\d.]+\z/) ? %("# ].join("\n") end -File.write(doc_file, "# Examples\n#{sections.join("\n")}") +intro = <<~MARKDOWN + # Examples + + The same example application for each database: a schema, queries that use every supported feature, and the code + the plugin generates from them. The generated code is committed, and the end-to-end tests run against it. + + In each generated directory, `query_sql.rb` has the query methods, `models.rb` the table and enum models, + `connection.rb` the shared helpers, and the `.rbs` files their signatures + (see [The generated code](Generated-Code.md)). + +MARKDOWN + +File.write(doc_file, intro + sections.join("\n"))