Skip to content

fix: strip whole CSI control sequences, not just the ESC - #23

Open
cycanic wants to merge 1 commit into
moderation-api:mainfrom
cycanic:fix/escape-sequence-remnant
Open

cycanic wants to merge 1 commit into
moderation-api:mainfrom
cycanic:fix/escape-sequence-remnant

Conversation

@cycanic

@cycanic cycanic commented Nov 14, 2026

Copy link
Copy Markdown

What & why

ESC is an illegal code point, but the rest of an ANSI colour code isn't, so only the
ESC was removed:

analyze('ok \x1b[31mred\x1b[0m done')
// before: normalized "ok [31mred[0m done", words ["\x1b", "\x1b"]
// after:  normalized "ok red done",        words ["\x1b[31m", "\x1b[0m"]

The leftover bytes also stuck to the next word (31mred became one token), so a
lookalike word right after a colour code could hide behind them.

At each ESC we now match a well-formed ECMA-48 control sequence (ESC [, parameter
bytes 0x30–0x3F, intermediate bytes 0x20–0x2F, final byte 0x40–0x7E) and report and
remove it as one illegal finding. The sequence is blanked before tokenizing, so its
bytes don't join the neighbouring word and offsets stay correct. An ESC without a
complete CSI (unterminated, OSC, other escapes) is removed on its own, as before.

Tests: whole-sequence removal, \x1b[1mНОТ deal … still finding НОТ,
unterminated sequences, and a must-not-flag case: plain text see note [31m] and [0m
is left alone.

Checklist

  • Tests added/updated (including false-positive cases for multilingual text)
  • pnpm lint && pnpm check-types && pnpm test && pnpm build pass locally
  • Added a changeset (pnpm changeset) if this is a user-facing change (patch)

Found while maintaining an extended version at https://github.com/cycanics/unicode-spoofing.

🤖 Generated with Claude Code

Only ESC is an illegal code point, so "ok \x1b[31mred\x1b[0m done"
normalized to "ok [31mred[0m done", and "31mred" tokenized as one word.

Match a well-formed ECMA-48 control sequence (ESC "[", parameter bytes
0x30-0x3F, intermediates 0x20-0x2F, final byte 0x40-0x7E) at each ESC and
report and remove it as one `illegal` finding. The sequence is blanked
before tokenizing so its bytes don't join the neighbouring word. An ESC
without a complete CSI (incl. OSC and other escapes) is removed on its
own, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cycanic
cycanic requested a review from chrisdengso as a code owner November 14, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant