Skip to content

Accept a lone carriage return as a newline - #14

Open
danslo wants to merge 1 commit into
rectorphp:mainfrom
danslo:fix-lone-cr
Open

danslo wants to merge 1 commit into
rectorphp:mainfrom
danslo:fix-lone-cr

Conversation

@danslo

@danslo danslo commented Nov 14, 2026

Copy link
Copy Markdown

PHP treats \r, \n and \r\n all as newlines, but the lexer rejects a lone \r between tokens (Unexpected character in input: '' (ASCII=13)), and a \r at the very end of the input panics with index out of range.

  • The newline rule gave '\r\n' a higher entering priority than '\r', so Ragel dropped the transition that ends a newline at a lone \r. Removed the priorities.
  • The new_line action read lex.data[lex.p+1] past the end of the input.
  • internal/scanner/scanner.rl still imported github.com/TomasVotruba/reco/pkg/token, so the regenerated scanner didn't build.

Both scanners are regenerated with ragel 6.10 (ragel -Z -G2, then the Makefile's sed and gofmt -s), so most of the diff is renumbered state tables. Added a test to each scanner.

The newline rule gave '\r\n' a higher entering priority than '\r', so
Ragel dropped the transition that ends a newline at a lone '\r', and
the lexer rejected it as an unexpected character. PHP treats '\r', '\n'
and '\r\n' all as newlines. A '\r' at the end of the input also read past
the data in the new_line action.

The older scanner's .rl still imported the pre-extraction module path,
so regenerating it did not build; it now imports pkg/token from this
module.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant