Rewrite scanner and parser from re2c/lemon to PHP

.github/workflows/main.yml

@@ -101,9 +101,48 @@ jobs:
         with:
           composer-options: "--prefer-dist"
 
+      # ScannerTest checks keyword matching under a Turkish locale
+      - name: Generate the tr_TR.UTF-8 locale
+        run: sudo locale-gen tr_TR.UTF-8
+
       - name: Run tests
         run: composer test-unit
 
+  differential:
+    name: "Differential vs ext-phalcon"
+    needs: phpcs
+    runs-on: ubuntu-latest
+
+    permissions:
+      contents: read
+
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Setup PHP with ext-phalcon
+        uses: shivammathur/setup-php@eb7c497e18156a6bbabfef1d3a82760b9eda3962 # v2
+        with:
+          php-version: '8.3'
+          extensions: mbstring, phalcon-5.23.0
+          coverage: none
+          tools: composer:v2
+        env:
+          COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Install dependencies
+        uses: ramsey/composer-install@26d8a556604053a9612623447203a691f406fbe6 # v4
+        with:
+          composer-options: "--prefer-dist"
+
+      # tests/differential.php exits 0 when the extension is missing
+      - name: Assert the extension is actually loaded
+        run: php -m | grep -qi '^phalcon$'
+
+      - name: Compare against the C parser (corpus + seeded fuzz)
+        run: php tests/differential.php --fuzz=5000 --seed=1
+
   coverage:
     name: Code Coverage
     needs: tests

CHANGELOG.md

@@ -0,0 +1,38 @@
+# Changelog
+
+All notable changes are documented here. The format is based on [Keep a Changelog][keep_a_changelog] and this project adheres to [Semantic Versioning][semantic_versioning].
+
+## [Unreleased](https://github.com/phalcon/phql/releases/tag/vx.x.x) (2026-xx-xx)
+
+### Changed
+
+- The scanner and the parser are written by hand instead of being transliterated from cphalcon's re2c scanner and Lemon parser; `resources/files/parser.php`, the global `phql_*` functions and the `c/` sources are gone.
+- `Phalcon\Phql\Parser::parse()` returns `?array`: `null` for a statement without any token (empty, blank, or starting with a NUL byte), as cphalcon does.
+- `Phalcon\Phql\Scanner\Scanner` was rewritten: `new Scanner(string $phql)`, `next(): Token` and `cursor(): int` replace the `State`-based `scanForToken()` / `getToken()`.
+- `Phalcon\Phql\Scanner\Token` holds an `int` opcode (an `Opcode` value or `Token::EOF` / `Token::SCAN_ERROR`) and a value; its `Opcode` type and `$length` property are gone.
+- `Phalcon\Phql\Tokens` is `final`, and its `$names` table became `Tokens::name()`.
+- `Phalcon\Phql\Scanner\Opcode` is unchanged.
+- `ext-mbstring` is no longer required: the scanner works on bytes.
+- Parsing is linear in the statement's length: a 40,000-item `IN` list parses in about 0.15 s instead of a minute.
+
+### Removed
+
+- `Phalcon\Phql\Parser\Status`, `Phalcon\Phql\Scanner\State` and `Phalcon\Phql\Scanner\ScannerStatus`.
+
+### Added
+
+- Parity tests against ext-phalcon: a statement corpus with goldens recorded from cphalcon (`tests/generate-golden.php`), `FixturesTest`, and `tests/differential.php` with a seeded fuzzer, run in CI.
+
+### Fixed
+
+Parsing now matches cphalcon exactly. Previously:
+
+- an empty statement threw "PHQL statement cannot be NULL" and a blank one "Syntax error, unexpected EOF", where cphalcon returns no result;
+- syntax errors quoted one character instead of the rest of the statement, never showed the token's value, and lacked the statement's length;
+- scanning errors did not skip the offending byte, cut the excerpt at the wrong length, and counted characters instead of bytes;
+- string literals kept their escape sequences (cphalcon resolves `\n`, `\r`, `\t`, `\\`, `\'`, `\"`);
+- deeply nested statements were accepted instead of failing with "Parsing failed, the PHQL statement is nested too deeply";
+- parsing time grew quadratically with the length of the statement.
+
+[keep_a_changelog]: https://keepachangelog.com/en/1.1.0/
+[semantic_versioning]: https://semver.org/spec/v2.0.0.html

README.md

@@ -17,6 +17,153 @@
 
 This repository consists of the PHQL parser, for the Phalcon Query Language used by the Phalcon Framework.
 
+## What this is
+
+Phalcon v6 is a pure-PHP rewrite of [cphalcon](https://github.com/phalcon/cphalcon),
+the C extension behind Phalcon v5. This package is the PHQL parser for v6, and
+it is a behavioural port of cphalcon's: the same intermediate representation
+(key order and integer types included), the same error messages on the same
+tokens, and the same limit on how deeply a statement may nest. A statement that
+parses (or fails) one way under v5 parses (or fails) the same way here.
+
+The scanner and the parser are written by hand. cphalcon generates its own from
+a re2c scanner and a Lemon grammar; earlier versions of this package carried PHP
+transliterations of those generated artefacts.
+
+## Requirements
+
+* PHP `>= 8.1 < 9.0`; no extensions
+
+## Installation
+
+```bash
+composer require phalcon/phql
+```
+
+## Usage
+
+```php
+use Phalcon\Phql\Parser;
+
+$ast = (new Parser())->parse('SELECT r.name FROM Robots r WHERE r.id = :id:');
+```
+
+`parse()` returns the statement's AST, `null` for a statement without any token
+(empty, blank, or starting with a NUL byte), or throws `Phalcon\Phql\Exception`
+with cphalcon's message. `setEnableLiterals(false)` rejects numbers, strings,
+`TRUE` and `FALSE` (cphalcon's `phalcon.orm.enable_literals`). The node `type`
+values are the cases of `Phalcon\Phql\Scanner\Opcode`.
+
+The parser keeps no state between calls. cphalcon's per-request AST cache and
+the `id` it adds to each AST belong to `Phalcon\Mvc\Model\Query\Lang` in
+phalcon/phalcon, which also turns parse errors into
+`Phalcon\Mvc\Model\Exception`.
+
+## How it works
+
+| Class | Role | cphalcon counterpart |
+|---|---|---|
+| `Scanner\Scanner` | Turns the statement into tokens, one per call | `scanner.re` |
+| `Parser\TokenStream` | Feeds the parser one token at a time, runs the literal and opcode checks, builds every error message | the token loop in `base.c`, `%syntax_error` |
+| `Parser\StatementParser` | Statements and clauses, by recursive descent - one method per group of grammar rules | `parser.php.lemon` |
+| `Parser\ExpressionParser` | Expressions, by precedence climbing over the grammar's own precedence table | `parser.php.lemon` |
+| `Parser\Cursor` | One token of lookahead plus a count of Lemon's parser stack | the generated `parser.c` |
+| `Parser\Ast` | Builds the nodes | `parser.php.inc.h` |
+
+Tokens are scanned on demand rather than up front: cphalcon checks each token
+(literals, unknown opcodes) and reports scanning errors only when it reaches
+them, so a later token must not pre-empt an error cphalcon reports earlier.
+
+Scanning works on bytes with `strspn()`/`strcspn()` runs from the cursor, and
+the grammar's left-recursive lists and left-associative operator chains are
+loops, so parsing time grows linearly with the statement (a 40,000-item `IN`
+list parses in about 0.15 s).
+
+## Compatibility notes
+
+Everything below is cphalcon's behaviour, reproduced on purpose and pinned by
+the parity tests.
+
+* **Operator precedence follows the grammar, not SQL.** From loosest to
+  tightest: `AGAINST`; `BETWEEN`, `NOT BETWEEN`; comparisons and `@@ @> <@ &&`;
+  `AND`, `OR`; `LIKE`, `ILIKE`; `& | ^`; `* / %`; `+ - ||`; `IS`; `IN`;
+  `NOT`, `~`; the JSON operators. So `a = 1 AND b > 0` is `(a = (1 AND b)) > 0`,
+  `a + b * c` is `(a + b) * c` and `a BETWEEN 1 AND 2` is `a BETWEEN (1 AND 2)`.
+* **`NOT LIKE`, `NOT ILIKE` and `NOT IN` take `NOT`'s level**, which is high
+  and right-associative: `a + b NOT LIKE c` is `a + (b NOT LIKE c)`.
+* **Unary minus has binary minus's level**: `-a * b` is `(-a) * b`.
+* **`NOT BETWEEN` is one token only with exactly one space**; `NOT  BETWEEN` is
+  a syntax error.
+* **String literals are unescaped** (`\n`, `\r`, `\t`, `\\`, `\'`, `\"`); any
+  other escape keeps its backslash. Bracketed identifiers (`[a b]`) are kept
+  verbatim, a leading backslash of a name is dropped, and `:` may appear inside
+  names (`x::int`). Numbers stay strings.
+* **A NUL byte ends the statement**, as it ends a C string. Error messages
+  quote the statement up to the NUL but report its full length.
+* **Syntax errors name the last token scanned** and quote the input after it,
+  so an error on the statement's last byte reads as `Syntax error, unexpected
+  EOF`. Token names come from cphalcon's table: `%%` for `%`, `NOT EQUALS` for
+  `!=` / `<>`, `HEXAINTEGER` for hexadecimal numbers, `UNKNOWN` for `TRUE`,
+  `FALSE`, `{placeholders}`, `NOT BETWEEN`, `~` and `^`. The value is shown for
+  value tokens only, even when empty or `0`.
+* **Scanning errors** quote what follows the offending byte, cut to 16 bytes
+  when more than 16 bytes followed the previous token. When the very first
+  token fails, a long message is cut short exactly as cphalcon's fixed-size
+  buffer cuts it.
+* **A lone `:` is reported as `Scanner: Unknown opcode 58`.**
+* **Nesting fails closed** with "Parsing failed, the PHQL statement is nested
+  too deeply" exactly where cphalcon's 100-entry Lemon stack overflows (for
+  example 96 nested parentheses in a column, 31 nested subqueries, or 33
+  chained `NOT LIKE`).
+* **A statement without tokens returns `null`**, not an error.
+
+## Development
+
+```bash
+composer install
+composer test-unit     # PHPUnit
+composer analyze       # PHPStan, level max
+composer cs            # PHP_CodeSniffer
+composer cs-fixer-fix  # php-cs-fixer
+```
+
+### Verifying against the C parser
+
+The parity corpus in `tests/fixtures/parity/*.php` lists statements: those of
+cphalcon's, phalcon's and this package's test suites
+(`tests/harvest-corpus.php` collects them) plus hand-written groups for the
+grammar's shape, operator precedence, scanner edge cases, errors, nesting depth
+and empty statements.
+
+* `tests/generate-golden.php` runs every statement through ext-phalcon - with
+  literals enabled and disabled - and records the outcome in
+  `tests/fixtures/parity/*.json`. It refuses to run without the extension, so
+  the goldens always come from cphalcon.
+* `tests/unit/Parser/FixturesTest.php` replays the goldens against this package.
+* `tests/differential.php` compares both implementations live, over the corpus
+  and over `--fuzz=N` generated statements (`--seed=S` makes them
+  reproducible). It exits quietly without the extension.
+
+```bash
+php -d extension=phalcon tests/generate-golden.php   # after changing the corpus
+php -d extension=phalcon tests/differential.php --fuzz=50000 --seed=1
+```
+
+Deliberate differences would be listed, with the expected outcome and a reason,
+in `tests/fixtures/parity/allow-list.php`; there are none.
+
+## Why a hand-written parser
+
+The re2c DFA and the Lemon tables are build artefacts: thousands of lines that
+nobody can review, that hide the grammar, and that had drifted from cphalcon
+(error messages, scanning errors, string escapes, empty statements, the nesting
+limit). Their PHP transliteration was also quadratic on long lists: a
+40,000-item `IN` list took a minute. The grammar itself is small, and every
+shift/reduce decision Lemon makes comes from a 12-level precedence table. A
+precedence-climbing parser over that table, plus a counter that mirrors Lemon's
+stack, reproduces the generated parser exactly while staying readable: each
+parser method quotes the grammar rules it implements.
+
 <!-- Badges -->
 [packagist-version-badge]:   https://img.shields.io/packagist/v/phalcon/phql?include_prereleases&style=flat-square
 [packagist-version-link]:    https://packagist.org/packages/phalcon/phql