[cycanic]

What & why

One letter of a FORMAT_CHAR_SCRIPTS script exempted its whole word from invisible
and zalgo. So anything hidden inside a Hebrew, Arabic, Persian or Indic word passed
and stayed in normalized:

analyze('של‮ום')                    // RLO inside a Hebrew word
// before: spoofed false, normalized unchanged
// after:  spoofed true (invisible), normalized "שלום"

analyze('می​​​​​خواهم')  // 5×ZWSP inside a Persian word
// before: spoofed false
// after:  spoofed true (invisible), normalized "میخواهم"

The same went for tag characters inside Arabic or Devanagari words.

Now each hidden character and mark is judged by the nearest letters around it
(script-neutral letters such as the tatweel are skipped). Still exempt:

• a single ZWNJ/ZWJ/ZWSP (or the ZWJ ZWNJ and ZWJ ZWNJ ZWJ idioms) in a word of these
  scripts, with no letter of another script on either side (Persian ZWNJ, also next
  to numerals; Indic conjuncts; Malayalam chillu; Thai/Khmer/Myanmar word breaks)
• a single implicit bidi mark or isolate next to an RTL letter, or next to a number
  in an RTL word
• CGJ between the marks of such a letter
• a format character of a neighbouring letter's script (Mongolian vowel separator,
  Syriac abbreviation mark)
• mark stacks on these letters

Everything else is judged as it would be in a Latin word, and only the offending
characters are stripped. Myanmar joins VARIATION_BASE_SCRIPTS so its standardized
dotted forms (က + VS1) don't start being reported now that its words are no longer
exempt as a whole. README wording for FORMAT_CHAR_SCRIPTS is updated to match.

Tests: 8 must-flag cases (overrides, tags, ZWSP runs, mixed-script joiners, Hangul
filler, bidi mark/isolate runs), zalgo on a Latin letter next to Hebrew points, and 23
must-not-flag cases across Persian, Hindi, Malayalam, Thai, Khmer, Myanmar, Hebrew
(incl. Biblical point order), Arabic, Syriac, Mongolian, Tibetan and Kurdish.

Heads-up: this touches the same lines in analyzeToken as #25.
Whichever lands second needs a small rebase. I have it ready.

Checklist

• Tests added/updated (including false-positive cases for multilingual text)
• pnpm lint && pnpm check-types && pnpm test && pnpm build pass locally
• Added a changeset (pnpm changeset) if this is a user-facing change (minor)

Found while maintaining an extended version at https://github.com/cycanics/unicode-spoofing.

🤖 Generated with Claude Code