SQL in C++ v2.0.0 paged storage engine

CMakeLists.txt

@@ -1,6 +1,10 @@
 cmake_minimum_required(VERSION 3.20)
 
-project(SQLInCPP VERSION 1.0.0 LANGUAGES CXX)
+project(SQLInCPP VERSION 2.0.0 LANGUAGES CXX)
+
+if(NOT UNIX)
+    message(FATAL_ERROR "Storage format v2 requires POSIX file and process primitives")
+endif()
 
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
@@ -22,14 +26,24 @@ set(SQL_CORE_SOURCES
     includes/parser/parser.cpp
     includes/shunting_yard/shunting_yard.cpp
     includes/sql/sql.cpp
+    includes/storage/page_buffer.cpp
+    includes/storage/storage_v2.cpp
     includes/table/table.cpp
     includes/tokenizer/ftokenize.cpp
     includes/tokenizer/state_machine_functions.cpp
     includes/tokenizer/stokenize.cpp
 )
 
 add_library(sql_core STATIC ${SQL_CORE_SOURCES})
-target_include_directories(sql_core PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}")
+configure_file(
+    "${CMAKE_CURRENT_SOURCE_DIR}/includes/version.h.in"
+    "${CMAKE_CURRENT_BINARY_DIR}/generated/version.h"
+    @ONLY
+)
+target_include_directories(sql_core PUBLIC
+    "${CMAKE_CURRENT_SOURCE_DIR}"
+    "${CMAKE_CURRENT_BINARY_DIR}/generated"
+)
 
 function(sql_enable_warnings target)
     if(MSVC)
@@ -45,6 +59,10 @@ add_executable(run_sql run_sql.cpp)
 target_link_libraries(run_sql PRIVATE sql_core)
 sql_enable_warnings(run_sql)
 
+add_executable(sql_migrate_v1 tools/sql_migrate_v1.cpp)
+target_link_libraries(sql_migrate_v1 PRIVATE sql_core)
+sql_enable_warnings(sql_migrate_v1)
+
 include(CTest)
 
 if(BUILD_TESTING)
@@ -72,4 +90,51 @@ if(BUILD_TESTING)
     target_link_libraries(bplustree_tests PRIVATE sql_core)
     sql_enable_warnings(bplustree_tests)
     add_test(NAME bplustree_evidence COMMAND bplustree_tests)
+
+    add_executable(storage_v2_tests tests/storage_v2_tests.cpp)
+    target_link_libraries(storage_v2_tests PRIVATE sql_core)
+    sql_enable_warnings(storage_v2_tests)
+    add_test(NAME storage_v2_integrity COMMAND storage_v2_tests)
+
+    add_executable(storage_crash_worker tests/storage_crash_worker.cpp)
+    target_link_libraries(storage_crash_worker PRIVATE sql_core)
+    sql_enable_warnings(storage_crash_worker)
+
+    add_executable(storage_crash_tests tests/storage_crash_tests.cpp)
+    target_link_libraries(storage_crash_tests PRIVATE sql_core)
+    target_compile_definitions(storage_crash_tests PRIVATE
+        STORAGE_CRASH_WORKER_PATH="$<TARGET_FILE:storage_crash_worker>"
+        RUN_SQL_PATH="$<TARGET_FILE:run_sql>"
+    )
+    add_dependencies(storage_crash_tests storage_crash_worker run_sql)
+    sql_enable_warnings(storage_crash_tests)
+    add_test(NAME storage_crash_recovery COMMAND storage_crash_tests)
+    set_tests_properties(storage_crash_recovery PROPERTIES TIMEOUT 60 LABELS crash)
+
+    add_executable(migration_tests tests/migration_tests.cpp)
+    target_link_libraries(migration_tests PRIVATE sql_core)
+    target_compile_definitions(migration_tests PRIVATE
+        MIGRATION_TOOL_PATH="$<TARGET_FILE:sql_migrate_v1>"
+    )
+    add_dependencies(migration_tests sql_migrate_v1)
+    sql_enable_warnings(migration_tests)
+    add_test(NAME migration_v1_equivalence COMMAND migration_tests)
+
+    add_executable(cli_tests tests/cli_tests.cpp)
+    target_link_libraries(cli_tests PRIVATE sql_core)
+    target_compile_definitions(cli_tests PRIVATE
+        RUN_SQL_PATH="$<TARGET_FILE:run_sql>"
+    )
+    add_dependencies(cli_tests run_sql)
+    sql_enable_warnings(cli_tests)
+    add_test(NAME cli_storage_inspection COMMAND cli_tests)
+
+    add_test(NAME cli_version COMMAND run_sql --version)
+    set_tests_properties(cli_version PROPERTIES
+        PASS_REGULAR_EXPRESSION "SQL in C\\+\\+ 2.0.0 \\(storage format 2\\)"
+    )
+    add_test(NAME migration_cli_version COMMAND sql_migrate_v1 --version)
+    set_tests_properties(migration_cli_version PROPERTIES
+        PASS_REGULAR_EXPRESSION "sql_migrate_v1 2.0.0 \\(storage format 2\\)"
+    )
 endif()

DESIGN.md

@@ -1,6 +1,6 @@
 # Design and invariants
 
-SQL in C++ is an educational database engine with a deliberately small grammar and fixed-width local persistence.
+SQL in C++ is an educational database engine with a deliberately small grammar and checksummed paged local persistence.
 The implementation favors inspectable data structures over production database features.
 
 ## Command path
@@ -16,35 +16,41 @@ An invalid `WHERE` expression never falls back to an unfiltered query.
 
 ## Storage format
 
-Each field occupies one zero-padded 100-byte slot.
-The schema file stores the field count followed by one slot per field name.
-The data file stores each row as exactly `field_count * 100` bytes.
+Version 2 stores a table in one 4096-byte paged `<table>.sql2` file.
+Every page has an explicit type, identifier, payload bound, next pointer, and CRC32.
+Schema and row streams span checked page chains, so the maximum supported 64-field row can safely exceed one page.
+Each mutating statement updates transaction-owned row and B+ tree pages, tracks a deterministic dirty-page set, and builds a complete recovery image before any database page is written.
 
-The format retains the repository's existing fixed-width layout.
-Writes now use explicit zero-initialized buffers instead of reading beyond a string allocation.
-Values longer than 100 bytes, rows with the wrong number of values, duplicate fields, partial rows, and invalid metadata are rejected before they can be treated as valid data.
+The checksummed WAL stores that complete image and a commit footer.
+The writer fsyncs the WAL before database pages, fsyncs the database, then truncates and fsyncs the WAL.
+Recovery validates the whole WAL before redo and rejects incomplete or corrupt records.
+Redo is idempotent and the engine never acknowledges a statement before the WAL, database, and cleared-WAL boundaries complete.
+
+The legacy v1 fixed-width pair remains available through a read-only compatibility path and the copy-first migration tool.
+Migration never modifies or deletes its v1 source.
 
 Tables support between 1 and 64 uniquely named fields.
 Table and field identifiers use letters, digits, and underscores, and cannot begin with a digit.
 
 ## Indexes and query semantics
 
-Every field has an in-memory `MMap` backed by the repository's B+ tree implementation.
-Indexes are rebuilt from the data file when a table is opened.
+Every field has a persistent page-backed B+ tree ordered by `(value, record_id)`.
+Inserts mutate the persisted search path incrementally, preserve untouched page identifiers, split bounded leaf and internal pages, and replace roots through the index directory.
+Opening and querying a v2 table validates and reads persisted index entries without inserting them into the legacy in-memory `MMap`.
 Equality and range filters return record numbers, while logical expressions combine those sets with union or intersection.
 
 Values are strings.
 Ordering is lexical rather than numeric, so `"10"` sorts before `"2"`.
-The current `lower_bound` and `upper_bound` implementations scan the linked leaf sequence and are not a logarithmic query-planning claim.
+The current query layer loads persistent leaves into its compatibility map and is not a query-planning or logarithmic performance claim.
 
 ## Deliberate limits
 
-The engine has no transactions, concurrency control, write-ahead log, schema migration, joins, updates, deletes, query optimizer, or SQL compatibility guarantee.
+The engine has atomic single-writer statements and crash recovery, but no multi-statement transactions, concurrency control, joins, updates, deletes, query optimizer, or broader SQL compatibility guarantee.
 It is not designed for untrusted multi-user workloads.
 The command-line process reads and writes table files in its current working directory.
 
 ## Evidence
 
-The CTest suite covers end-to-end SQL behavior, parser safety, and deterministic B+ tree insertion and lookup properties.
+The CTest suite covers end-to-end SQL behavior, parser safety, persistent randomized B+ tree images, page and WAL corruption, crash recovery, v1 migration equivalence, large rows, and deterministic in-memory B+ tree properties.
 CI runs release builds on Linux and macOS plus an AddressSanitizer and UndefinedBehaviorSanitizer build on Linux.
 The sanitizer job is a memory-safety regression gate, not evidence of production database readiness.

README.md

@@ -1,7 +1,7 @@
 # SQL in C++
 
-SQL in C++ is an educational relational database engine and command-line DSL implemented from first principles.
-It parses a focused subset of SQL, persists records to binary files, indexes field values with B+ trees, and evaluates compound filters with a shunting-yard pipeline.
+SQL in C++ 2.0 is an educational relational database engine and command-line DSL implemented from first principles.
+It parses a focused subset of SQL, persists records in checksummed pages, maintains persistent B+ tree indexes, recovers atomic single-writer statements through a WAL, and evaluates compound filters with a shunting-yard pipeline.
 
 ![Command-line preview](image/preview.png)
 
@@ -12,11 +12,14 @@ It parses a focused subset of SQL, persists records to binary files, indexes fie
 - `SELECT *` and projected field lists.
 - `WHERE` filters using `=`, `!=`, `<`, `<=`, `>`, and `>=`.
 - Compound `AND` and `OR` expressions with parentheses.
-- Binary persistence across separate CLI sessions.
+- Checksummed paged persistence across separate CLI sessions.
+- Persistent per-field B+ tree indexes.
+- Crash recovery for atomic single-writer statements.
+- Copy-first migration from the v1 fixed-width format.
 - Batch execution from the sample command files in `batch/`.
 
 This project is intentionally smaller than a production SQL database.
-It does not implement joins, transactions, schemas, query planning, updates, or deletes.
+It does not implement joins, multi-statement transactions, schemas, query planning, updates, or deletes.
 Values are stored and ordered as strings, so numeric-looking values use lexical comparison semantics.
 
 ## Architecture
@@ -27,7 +30,8 @@ Values are stored and ordered as strings, so numeric-looking values use lexical
 | `includes/parser` | Validates the supported grammar and builds a parse tree. |
 | `includes/shunting_yard` | Converts compound filter expressions to postfix form. |
 | `includes/map` | Implements B+ tree-backed maps and multimaps. |
-| `includes/binary_files` | Reads and writes fixed-format binary records. |
+| `includes/binary_files` | Reads the legacy v1 fixed-width format. |
+| `includes/storage` | Owns v2 pages, persistent indexes, checksums, WAL, and recovery. |
 | `includes/table` | Owns table persistence, indexes, projections, and selections. |
 | `includes/sql` | Coordinates parsing and table operations through the public `SQL` interface. |
 
@@ -51,7 +55,7 @@ The build has no downloaded test framework or other third-party dependency.
 ctest --test-dir build --output-on-failure
 ```
 
-The test executables verify inserts, projections, compound filters, persistence, table and row validation, corrupt-file handling, quote and parenthesis safety, deterministic B+ tree properties, and both repository batch fixtures.
+The test executables verify SQL behavior, persistent incremental B+ tree splits, stable untouched pages, WAL ordering, dynamically enumerated crash boundaries, corruption rejection, idempotent recovery, migration equivalence, maximum-width rows, CLI inspection, parser safety, and both repository batch fixtures.
 Tests run in an isolated build directory, so they do not leave database files in the source tree.
 
 To run the same AddressSanitizer and UndefinedBehaviorSanitizer gate used by CI:
@@ -73,8 +77,8 @@ ctest --test-dir build-sanitized --output-on-failure
 ```
 
 Type `end` to exit and `cls` to clear the terminal.
-The CLI writes each table to `<table>.bin` and `<table>_fields.bin` in the current working directory.
-It reports missing tables, invalid projections, wrong row arity, duplicate schemas, malformed filters, and corrupt fixed-width files without terminating the session.
+The CLI writes each table to `<table>.sql2` and uses `<table>.sql2-wal` during a mutating statement.
+It reports missing tables, invalid projections, wrong row arity, duplicate schemas, malformed filters, corrupt pages, and corrupt WAL records without terminating the session.
 
 Example session:
 
@@ -99,17 +103,46 @@ select <* | field [, field ...]> from <table>
 Quote values that contain spaces, such as `"Mary Ann"`.
 Quoted values may contain spaces, numbers, and punctuation, but escaped double quotes are not supported.
 
+## Version and storage inspection
+
+```bash
+./build/run_sql --version
+./build/run_sql --inspect-storage student
+```
+
+Inspection validates checksums and structural bounds without applying or clearing a pending WAL.
+
+## Migrate a v1 table
+
+Inspect a legacy table without writing anything:
+
+```bash
+./build/sql_migrate_v1 --dry-run student
+```
+
+Migrate it to `student.sql2`:
+
+```bash
+./build/sql_migrate_v1 student
+```
+
+The migration reads the v1 `.bin` and `_fields.bin` files read-only, writes an exclusively owned temporary v2 image, reopens and compares every row, rechecks the source bytes, then atomically links the verified destination without replacement.
+It never modifies or deletes the v1 source.
+Use `--destination <path>` to choose another destination and note that existing destinations are rejected.
+
 ## Persistence boundaries
 
 - Tables contain between 1 and 64 uniquely named fields.
 - Identifiers contain letters, digits, and underscores and cannot start with a digit.
-- Each stored field occupies one zero-padded 100-byte slot.
+- Each value is at most 100 bytes, and rows may cross page boundaries.
 - An insert must provide exactly one value per field.
 - Values longer than 100 bytes are rejected before a write.
 - Existing tables cannot be overwritten by another `CREATE TABLE` command.
-- Partial rows and invalid schema metadata fail closed when a table is opened.
+- Page, index, schema, row, and WAL corruption fail closed when a table is opened.
+- V1 tables can be queried read-only and must be migrated before new inserts.
+- The durability scope is one process, one writer, and one atomic statement at a time.
 
-The fixed-width layout is intentionally simple and is not a transactional or crash-recovery format.
+The exact byte layout, recovery ordering, checksums, and compatibility boundary are documented in [docs/storage-v2.md](docs/storage-v2.md).
 
 ## License
 

docs/release-checklist-v2.md

@@ -0,0 +1,15 @@
+# Version 2.0.0 release checklist
+
+This checklist prepares the repository for a later release without creating a tag or GitHub Release.
+
+- [ ] Confirm the CMake project version is `2.0.0`.
+- [ ] Confirm `run_sql --version` reports CLI `2.0.0` and storage format `2`.
+- [ ] Run the complete Release CTest suite on Linux and macOS.
+- [ ] Run AddressSanitizer and UndefinedBehaviorSanitizer with warnings as errors.
+- [ ] Migrate the canonical v1 fixtures and compare schemas, rows, and indexed range-query results.
+- [ ] Run every deterministic write and recovery crash point.
+- [ ] Run superblock, page, WAL, child-pointer, and partial-record corruption tests.
+- [ ] Confirm persistent indexes reopen without a row-derived rebuild.
+- [ ] Confirm the v1 source files remain byte-identical after migration.
+- [ ] Review the documented single-process and single-writer compatibility boundary.
+- [ ] Review the pull request CI evidence before a maintainer creates `v2.0.0`.