Repository navigation
Conversation
isVariationSelector covered only U+FE00..FE0F and the Mongolian
selectors, so a supplement selector (U+E0100..E01EF) on a Latin letter
was never judged: one hidden byte per letter passed. Being category Mn,
the selectors also counted as stacked marks, so a valid ideographic
variation sequence followed by more selectors was reported as zalgo and
the valid selector was stripped with the rest.
Add the supplement range. A supplement selector is valid only directly
after a Han or kana letter; after a mark, digit, apostrophe or at the
word start it is stray. A base takes one selector, so a selector that
follows another is stray. VS1-VS16 after a mark are stray too; at the
word start they are left alone, as before, because there they end the
emoji or symbol in front of the word ("❤️you", "≨︀"). Valid selectors
no longer count toward zalgo stacks and are not stripped as zalgo
marks; stray ones still count.
Judging each selector against the character right before it also
removes a quadratic walk back over selector runs (x + 40 000 VS2 took
about 3 s); a perf test covers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
isVariationSelectorcovered only U+FE00–FE0F and the Mongolian selectors, so thesupplement (U+E0100–E01EF) was never judged. That allows one hidden byte per letter:
And because the selectors are category Mn, a valid ideographic variation sequence
followed by extra selectors was called zalgo, and the valid selector was stripped
along with the rest:
The fix:
another letter, a mark, a digit, an apostrophe, or at the start of a word, it's stray.
alone, as before, because there they end the emoji or symbol in front of it
(
❤️you).Stray ones still count.
Judging each selector against the character right before it also removes a quadratic
walk back over selector runs:
'x' + '︁'.repeat(40_000)took ~3.1 s onmainand takes ~0.1 s now.
Tests: supplement on Latin, the per-letter byte channel, extra selectors on an
ideograph, other carriers, symbol selectors at word start (must not flag), stray
selectors in zalgo stacks, IVS kept when zalgo is stripped, legitimate IVS in running
Japanese text, and a perf test for the long selector run.
Heads-up: this touches the same lines in
analyzeTokenas #26.Whichever lands second needs a small rebase. I'll do it.
Checklist
pnpm lint && pnpm check-types && pnpm test && pnpm buildpass locallypnpm changeset) if this is a user-facing change (minor)Found while maintaining an extended version at https://github.com/cycanics/unicode-spoofing.
🤖 Generated with Claude Code