Skip to content

fix: recognise the variation selector supplement (VS17–VS256) - #25

Open
cycanic wants to merge 1 commit into
moderation-api:mainfrom
cycanic:fix/variation-selector-supplement
Open

cycanic wants to merge 1 commit into
moderation-api:mainfrom
cycanic:fix/variation-selector-supplement

Conversation

@cycanic

@cycanic cycanic commented Nov 14, 2026 •

Copy link
Copy Markdown

What & why

isVariationSelector covered only U+FE00–FE0F and the Mongolian selectors, so the
supplement (U+E0100–E01EF) was never judged. That allows one hidden byte per letter:

analyze('h\u{E0151}e\u{E0152}l\u{E0153}lo')
// before: spoofed false, normalized unchanged
// after:  spoofed true (invisible), normalized "hello"

And because the selectors are category Mn, a valid ideographic variation sequence
followed by extra selectors was called zalgo, and the valid selector was stripped
along with the rest:

analyze('辻\u{E0100}\u{E0101}\u{E0102}')
// before: zalgo,     normalized "辻"
// after:  invisible, normalized "辻\u{E0100}"

The fix:

  • A supplement selector is valid only directly after a Han or kana letter. After
    another letter, a mark, a digit, an apostrophe, or at the start of a word, it's stray.
  • A base takes one selector, so a selector right after another is stray.
  • VS1–VS16 after a combining mark are stray. At the start of a word they're left
    alone, as before, because there they end the emoji or symbol in front of it
    (❤️you).
  • Valid selectors don't count toward a zalgo stack and aren't stripped as zalgo marks.
    Stray ones still count.

Judging each selector against the character right before it also removes a quadratic
walk back over selector runs: 'x' + '︁'.repeat(40_000) took ~3.1 s on main
and takes ~0.1 s now.

Tests: supplement on Latin, the per-letter byte channel, extra selectors on an
ideograph, other carriers, symbol selectors at word start (must not flag), stray
selectors in zalgo stacks, IVS kept when zalgo is stripped, legitimate IVS in running
Japanese text, and a perf test for the long selector run.

Heads-up: this touches the same lines in analyzeToken as #26.
Whichever lands second needs a small rebase. I'll do it.

Checklist

  • Tests added/updated (including false-positive cases for multilingual text)
  • pnpm lint && pnpm check-types && pnpm test && pnpm build pass locally
  • Added a changeset (pnpm changeset) if this is a user-facing change (minor)

Found while maintaining an extended version at https://github.com/cycanics/unicode-spoofing.

🤖 Generated with Claude Code

isVariationSelector covered only U+FE00..FE0F and the Mongolian
selectors, so a supplement selector (U+E0100..E01EF) on a Latin letter
was never judged: one hidden byte per letter passed. Being category Mn,
the selectors also counted as stacked marks, so a valid ideographic
variation sequence followed by more selectors was reported as zalgo and
the valid selector was stripped with the rest.

Add the supplement range. A supplement selector is valid only directly
after a Han or kana letter; after a mark, digit, apostrophe or at the
word start it is stray. A base takes one selector, so a selector that
follows another is stray. VS1-VS16 after a mark are stray too; at the
word start they are left alone, as before, because there they end the
emoji or symbol in front of the word ("❤️you", "≨︀"). Valid selectors
no longer count toward zalgo stacks and are not stripped as zalgo
marks; stray ones still count.

Judging each selector against the character right before it also
removes a quadratic walk back over selector runs (x + 40 000 VS2 took
about 3 s); a perf test covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant