Lex a minus directly after an operand as an operator (issue370)

[adarshsm]

Fixes #370.

The bug

The lexer let any number start with a -, so in a subtraction written without spaces the minus was swallowed by the right-hand number and the operator disappeared:

>>> from sqlparse import lexer
>>> list(lexer.tokenize('1-1'))
[(Token.Literal.Number.Integer, '1'), (Token.Literal.Number.Integer, '-1')]
>>> list(lexer.tokenize('a-1'))
[(Token.Name, 'a'), (Token.Literal.Number.Integer, '-1')]

So select a-1 from foo was grouped as Identifier 'a' followed by a loose -1 token instead of an Operation. a - 1 with spaces was fine. Hex (x-0x1F) and float (a-1.5, a-.5, a-1e3) literals behaved the same way. This still reproduces on current master.

The fix

The leading minus in the hex, float and integer regexes is now only taken when it doesn't directly follow an operand. That means a word character (name, number, keyword), ), ], or a closing quote (', ", `). The lexer calls the compiled patterns with match(text, pos), so the lookbehind sees the preceding character, as the existing (?<!\w)[$:?]\w+ and (?<![\w\])])(\[...\]) rules already rely on.

>>> list(lexer.tokenize('a-1'))
[(Token.Name, 'a'), (Token.Operator, '-'), (Token.Literal.Number.Integer, '1')]
>>> sqlparse.parse('select a-1 from foo')[0].tokens[2]
<Operation 'a-1' at 0x...>

Unary minus is lexed exactly as before wherever it follows an operator, punctuation or whitespace: x = -1, values(-1), in (1,-2), a*-1, select -1, and the ambiguous a -1 all still give an Integer '-1'. The E-? exponent sign is untouched.

Notes

• With - now a separate token, format(..., reindent=True) aligns the next column after a-1 correctly. select a-1, b from t used to indent b one column too far.
• One behavior change to be aware of: a keyword directly followed by a minus with no space (select-1, then-1) now lexes as keyword, -, 1 instead of keyword, -1, since a lookbehind can't tell keywords from names. Formatting output for those statements is unchanged. This is consistent with how select - 1 is already lexed.

Tests

• test_tokenize_minus_after_operand (parametrized over integer, float, exponent, hex, function call, array subscript and the three quoting styles). The 1-1 case is the test from the issue.
• test_operation_minus_without_whitespace: select a-1 from foo groups a-1 as an Operation.
• All 12 new cases fail on master and pass with this change. Full suite: 518 passed, 2 xfailed, 1 xpassed (the xfail/xpass set is the same as on master). ruff check sqlparse/ is clean.

---

• ran the tests (pytest)
• all style issues addressed (ruff)
• your changes are covered by tests
• your changes are documented, if needed (no docs change needed. I left the CHANGELOG entry to you, as with the other entries)

🤖 Generated with Claude Code