diff --git a/DESIGN.md b/DESIGN.md index f3cd0c6..0a9d5d5 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -14,6 +14,19 @@ The implementation favors inspectable data structures over production database f Invalid syntax, unknown tables or fields, unmatched quotes or parentheses, and corrupt files fail closed. An invalid `WHERE` expression never falls back to an unfiltered query. +## Source layout + +| Component | Responsibility | +| --- | --- | +| `includes/tokenizer` | Converts command text into lexical tokens with a state machine. | +| `includes/parser` | Validates the supported grammar and builds a parse tree. | +| `includes/shunting_yard` | Converts compound filter expressions to postfix form. | +| `includes/map` | Implements B+ tree-backed maps and multimaps. | +| `includes/binary_files` | Reads the legacy v1 fixed-width format. | +| `includes/storage` | Owns v2 pages, persistent indexes, checksums, WAL, and recovery. | +| `includes/table` | Owns table persistence, indexes, projections, and selections. | +| `includes/sql` | Coordinates parsing and table operations through the public `SQL` interface. | + ## Storage format Version 2 stores a table in one 4096-byte paged `.sql2` file. @@ -53,4 +66,4 @@ The command-line process reads and writes table files in its current working dir The CTest suite covers end-to-end SQL behavior, parser safety, persistent randomized B+ tree images, page and WAL corruption, crash recovery, v1 migration equivalence, large rows, and deterministic in-memory B+ tree properties. CI runs a release build and the CTest suite on Linux. -The AddressSanitizer and UndefinedBehaviorSanitizer build in the README is a local memory-safety check, not evidence of production database readiness. +The AddressSanitizer and UndefinedBehaviorSanitizer build in [docs/usage.md](docs/usage.md#sanitizer-build) is a local memory-safety check, not evidence of production database readiness. diff --git a/README.md b/README.md index 71bf335..f97ae55 100644 --- a/README.md +++ b/README.md @@ -1,151 +1,68 @@ # SQL in C++ -SQL in C++ 2.0 is an educational relational database engine and command-line DSL implemented from first principles. -It parses a focused subset of SQL, persists records in checksummed pages, maintains persistent B+ tree indexes, recovers atomic single-writer statements through a WAL, and evaluates compound filters with a shunting-yard pipeline. +A small relational database engine written from scratch in C++17. +It parses a focused subset of SQL, stores each table in checksummed 4 KB pages, keeps a B+ tree index for every field, and recovers each statement atomically after a crash through a write-ahead log. I wrote the first version in 2023 while learning C++ and rebuilt the storage engine in 2026. The [`original-2023`](https://github.com/hskl18/SQL/tree/original-2023) tag keeps the first version. -![Command-line preview](image/preview.png) +![A CLI session: an insert that prints the table, a select of two fields filtered by major, and a compound filter with AND that returns one row.](docs/media/cli-session.png) -## What it supports - -- `CREATE TABLE` with an ordered field list. -- `INSERT INTO` with quoted or unquoted values. -- `SELECT *` and projected field lists. -- `WHERE` filters using `=`, `!=`, `<`, `<=`, `>`, and `>=`. -- Compound `AND` and `OR` expressions with parentheses. -- Checksummed paged persistence across separate CLI sessions. -- Persistent per-field B+ tree indexes. -- Crash recovery for atomic single-writer statements. -- Copy-first migration from the v1 fixed-width format. -- Batch execution from the sample command files in `batch/`. - -This project is intentionally smaller than a production SQL database. -It does not implement joins, multi-statement transactions, schemas, query planning, updates, or deletes. -Values are stored and ordered as strings, so numeric-looking values use lexical comparison semantics. - -## Architecture - -| Component | Responsibility | -| --- | --- | -| `includes/tokenizer` | Converts command text into lexical tokens with a state machine. | -| `includes/parser` | Validates the supported grammar and builds a parse tree. | -| `includes/shunting_yard` | Converts compound filter expressions to postfix form. | -| `includes/map` | Implements B+ tree-backed maps and multimaps. | -| `includes/binary_files` | Reads the legacy v1 fixed-width format. | -| `includes/storage` | Owns v2 pages, persistent indexes, checksums, WAL, and recovery. | -| `includes/table` | Owns table persistence, indexes, projections, and selections. | -| `includes/sql` | Coordinates parsing and table operations through the public `SQL` interface. | +*A session in the CLI from a local Release build, November 13, 2026.* -Storage invariants, query semantics, and deliberate limits are documented in [DESIGN.md](DESIGN.md). +## Quick start -## Build - -You need a C++17 compiler and CMake 3.20 or newer. +You need a C++17 compiler and CMake 3.20 or newer; the build has no third-party dependency and downloads nothing. ```bash cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --parallel -``` - -The build has no downloaded test framework or other third-party dependency. - -## Test - -```bash ctest --test-dir build --output-on-failure -``` - -The test executables verify SQL behavior, persistent incremental B+ tree splits, stable untouched pages, WAL ordering, dynamically enumerated crash boundaries, corruption rejection, idempotent recovery, migration equivalence, maximum-width rows, CLI inspection, parser safety, and both repository batch fixtures. -Tests run in an isolated build directory, so they do not leave database files in the source tree. - -To run the tests under AddressSanitizer and UndefinedBehaviorSanitizer: - -```bash -cmake -S . -B build-sanitized \ - -DCMAKE_BUILD_TYPE=Debug \ - -DBUILD_TESTING=ON \ - -DSQL_ENABLE_SANITIZERS=ON \ - -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -cmake --build build-sanitized --parallel -ctest --test-dir build-sanitized --output-on-failure -``` - -## Run the CLI - -```bash ./build/run_sql ``` -Type `end` to exit and `cls` to clear the terminal. -The CLI writes each table to `
.sql2` and uses `
.sql2-wal` during a mutating statement. -It reports missing tables, invalid projections, wrong row arity, duplicate schemas, malformed filters, corrupt pages, and corrupt WAL records without terminating the session. - -Example session: +Type `end` to exit and `cls` to clear the screen. ```text SQL>> create table student fields first, last, major, age -SQL>> insert into student values Ada, Lovelace, Math, 36 SQL>> insert into student values Grace, Hopper, CS, 30 SQL>> select first, major from student where major = CS -SQL>> end -``` - -## Query grammar - -```text -create table
fields [, ...] -insert into
values [, ...] -select <* | field [, field ...]> from
- [where - [ ...]] -``` - -Quote values that contain spaces, such as `"Mary Ann"`. -Quoted values may contain spaces, numbers, and punctuation, but escaped double quotes are not supported. - -## Version and storage inspection - -```bash -./build/run_sql --version -./build/run_sql --inspect-storage student ``` -Inspection validates checksums and structural bounds without applying or clearing a pending WAL. +## Architecture -## Migrate a v1 table + + + A command goes from the CLI through the tokenizer and parser to the table, which filters through the shunting-yard step and a B+ tree index per field. The storage layer keeps checksummed 4 KB pages; it writes the after-image to the WAL and syncs it, then writes the dirty pages to the table file, and opening a table recovers from a complete WAL record. The v1 migration tool copies old .bin files into the new format. + -Inspect a legacy table without writing anything: +The table answers a `WHERE` filter from the per-field B+ tree indexes after the shunting-yard step turns the filter into postfix form. +Each statement that changes a table first writes its complete after-image to `
.sql2-wal` and syncs it, and only then writes the changed pages to `
.sql2`. +A crash before the WAL write leaves the old table, a crash after the WAL is synced recovers the new table on open, and a torn WAL record makes the open fail with a corruption error instead of guessing. -```bash -./build/sql_migrate_v1 --dry-run student -``` +## What it supports -Migrate it to `student.sql2`: +- `CREATE TABLE`, `INSERT INTO`, and `SELECT` with `*` or a field list. +- `WHERE` with `=`, `!=`, `<`, `<=`, `>`, `>=`, and `AND` and `OR` with parentheses. +- Tables that persist across CLI sessions, with page, index, and WAL corruption rejected on open. +- Batch runs from the sample command files in `batch/`. +- Copy-first migration from the 2023 fixed-width format. -```bash -./build/sql_migrate_v1 student -``` +It has no joins, multi-statement transactions, schemas, query planning, updates, or deletes, and it has no locking, so run only one writer at a time. +Values are stored and compared as strings, so `9` sorts after `10`. -The migration reads the v1 `.bin` and `_fields.bin` files read-only, writes an exclusively owned temporary v2 image, reopens and compares every row, rechecks the source bytes, then atomically links the verified destination without replacement. -It never modifies or deletes the v1 source. -Use `--destination ` to choose another destination and note that existing destinations are rejected. +## Tests -## Persistence boundaries +The CTest suite covers SQL behavior, parser safety, B+ tree splits, stable untouched pages, WAL ordering, a crash at every traced write and fsync, corruption rejection, recovery that can run twice, v1 migration, maximum-width rows, the CLI, and both batch fixtures. +Tests run in the build directory and leave no database files in the source tree. +CI runs a Release build and the suite on Linux. -- Tables contain between 1 and 64 uniquely named fields. -- Identifiers contain letters, digits, and underscores and cannot start with a digit. -- Each value is at most 100 bytes, and rows may cross page boundaries. -- An insert must provide exactly one value per field. -- Values longer than 100 bytes are rejected before a write. -- Existing tables cannot be overwritten by another `CREATE TABLE` command. -- Page, index, schema, row, and WAL corruption fail closed when a table is opened. -- V1 tables can be queried read-only and must be migrated before new inserts. -- The durability scope is one process, one writer, and one atomic statement at a time. +## Documentation -The exact byte layout, recovery ordering, checksums, and compatibility boundary are documented in [docs/storage-v2.md](docs/storage-v2.md). +- [Usage](docs/usage.md): the full grammar, the CLI's files and errors, storage inspection, v1 migration, limits, and the sanitizer build. +- [Design and invariants](DESIGN.md): the command path, source layout, and storage and query rules. +- [Storage format v2](docs/storage-v2.md): the byte layout, checksums, and recovery order. ## License -This project is available under the [MIT License](LICENSE). +[MIT](LICENSE) diff --git a/docs/architecture-dark.svg b/docs/architecture-dark.svg new file mode 100644 index 0000000..65cc479 --- /dev/null +++ b/docs/architecture-dark.svg @@ -0,0 +1 @@ +StorageQuery pathWHERE1. after-image, fsync2. dirty pagesrecover on openread-only, copy-firstrows and index pagesrun_sql CLIor batch fileTokenizer andparserstate machine,parse treeTableinsert, select,projectionShunting-yardpostfix filterB+ tree indexone per fieldPage bufferchecksummed4 KB pagestable.sql2-waltable.sql2v1 .bin filessql_migrate_v1 \ No newline at end of file diff --git a/docs/architecture-light.svg b/docs/architecture-light.svg new file mode 100644 index 0000000..2860c6b --- /dev/null +++ b/docs/architecture-light.svg @@ -0,0 +1 @@ +StorageQuery pathWHERE1. after-image, fsync2. dirty pagesrecover on openread-only, copy-firstrows and index pagesrun_sql CLIor batch fileTokenizer andparserstate machine,parse treeTableinsert, select,projectionShunting-yardpostfix filterB+ tree indexone per fieldPage bufferchecksummed4 KB pagestable.sql2-waltable.sql2v1 .bin filessql_migrate_v1 \ No newline at end of file diff --git a/docs/architecture.mmd b/docs/architecture.mmd new file mode 100644 index 0000000..e285328 --- /dev/null +++ b/docs/architecture.mmd @@ -0,0 +1,18 @@ +flowchart TB + subgraph query["Query path"] + direction LR + CLI["run_sql CLI
or batch file"] --> Parser["Tokenizer and parser
state machine, parse tree"] + Parser --> Table["Table
insert, select, projection"] + Table -->|"WHERE"| Filter["Shunting-yard
postfix filter"] + Filter --> Index["B+ tree index
one per field"] + end + subgraph storage["Storage"] + direction LR + Pages["Page buffer
checksummed
4 KB pages"] -->|"1. after-image, fsync"| WAL[("table.sql2-wal")] + Pages -->|"2. dirty pages"| Data[("table.sql2")] + WAL -.->|"recover on open"| Data + Legacy[("v1 .bin files")] -->|"read-only, copy-first"| Migrate["sql_migrate_v1"] + Migrate --> Data + end + query -->|"rows and index pages"| storage +%% Edit this file; the owner renders architecture-light.svg and architecture-dark.svg from it with a shared theme. diff --git a/docs/media/cli-session.png b/docs/media/cli-session.png new file mode 100644 index 0000000..9e829b6 Binary files /dev/null and b/docs/media/cli-session.png differ diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..fdd11ca --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,93 @@ +# Using SQL in C++ + +This page holds the details behind the [README](../README.md): the full grammar, the CLI's files and errors, storage inspection, v1 migration, persistence limits, and the sanitizer build. + +## Run the CLI + +```bash +./build/run_sql +``` + +Type `end` to exit and `cls` to clear the terminal. +The CLI writes each table to `
.sql2` and uses `
.sql2-wal` during a mutating statement. +It reports missing tables, invalid projections, wrong row arity, duplicate schemas, malformed filters, corrupt pages, and corrupt WAL records without terminating the session. + +Example session: + +```text +SQL>> create table student fields first, last, major, age +SQL>> insert into student values Ada, Lovelace, Math, 36 +SQL>> insert into student values Grace, Hopper, CS, 30 +SQL>> select first, major from student where major = CS +SQL>> end +``` + +## Query grammar + +```text +create table
fields [, ...] +insert into
values [, ...] +select <* | field [, field ...]> from
+ [where + [ ...]] +``` + +`` is one of `=`, `!=`, `<`, `<=`, `>`, or `>=`, and parentheses group `and` and `or` terms. + +Quote values that contain spaces, such as `"Mary Ann"`. +Quoted values may contain spaces, numbers, and punctuation, but escaped double quotes are not supported. + +## Version and storage inspection + +```bash +./build/run_sql --version +./build/run_sql --inspect-storage student +``` + +Inspection validates checksums and structural bounds without applying or clearing a pending WAL. + +## Migrate a v1 table + +Inspect a legacy table without writing anything: + +```bash +./build/sql_migrate_v1 --dry-run student +``` + +Migrate it to `student.sql2`: + +```bash +./build/sql_migrate_v1 student +``` + +The migration reads the v1 `.bin` and `_fields.bin` files read-only, writes an exclusively owned temporary v2 image, reopens and compares every row, rechecks the source bytes, then atomically links the verified destination without replacement. +It never modifies or deletes the v1 source. +Use `--destination ` to choose another destination and note that existing destinations are rejected. + +## Persistence boundaries + +- Tables contain between 1 and 64 uniquely named fields. +- Identifiers contain letters, digits, and underscores and cannot start with a digit. +- Each value is at most 100 bytes, and rows may cross page boundaries. +- An insert must provide exactly one value per field. +- Values longer than 100 bytes are rejected before a write. +- Existing tables cannot be overwritten by another `CREATE TABLE` command. +- Page, index, schema, row, and WAL corruption fail closed when a table is opened. +- V1 tables can be queried read-only and must be migrated before new inserts. +- The durability scope is one process, one writer, and one atomic statement at a time. + +The exact byte layout, recovery ordering, checksums, and compatibility boundary are documented in [storage-v2.md](storage-v2.md). + +## Sanitizer build + +To run the tests under AddressSanitizer and UndefinedBehaviorSanitizer: + +```bash +cmake -S . -B build-sanitized \ + -DCMAKE_BUILD_TYPE=Debug \ + -DBUILD_TESTING=ON \ + -DSQL_ENABLE_SANITIZERS=ON \ + -DCMAKE_COMPILE_WARNING_AS_ERROR=ON +cmake --build build-sanitized --parallel +ctest --test-dir build-sanitized --output-on-failure +``` diff --git a/image/preview.png b/image/preview.png deleted file mode 100644 index 193e6d9..0000000 Binary files a/image/preview.png and /dev/null differ