feat(search): support searchNormalize across non-Latin characters and intra-word punctuation

[gnbm]

---

Issue number: resolves #279

---

What is the current behavior?

• The normalizeString utility used the regex /[^\w]/g to strip non-word characters after NFD decomposition.
• The \w character class in JavaScript only matches [a-zA-Z0-9_], so every non-Latin script (Greek, Cyrillic, Vietnamese, Chinese, Japanese, Korean, Arabic, Thai, …) was treated as non-word and stripped entirely during normalization. This made searchNormalize: true completely broken for non-Latin scripts — labels became empty strings, so nothing could be matched.

What is the new behavior?

normalizeString now performs a Unicode-aware two-pass strip after NFD decomposition:

1. /\p{M}/gu — strips Unicode combining marks (category M). Removes diacritics across all scripts while preserving the underlying letters/ideographs.
2. /[^\p{L}\p{N}_]/gu — strips characters that are not Letters, Numbers, or underscore. Restores the punctuation- and whitespace-insensitive behavior the original /[^\w]/g provided for ASCII content (e.g. co-op still matches coop), in a script-aware way.

Both regexes are defined at module scope so they are compiled once instead of per call.

Language coverage

searchNormalize: true now works correctly for a single dropdown containing options across many writing systems:

Script	Example	Search input	Status
Latin (French/Spanish)	Crème brûlée, Niño	creme, nino	✅ matches
German (ä, ö, ü)	München, Mädchen, Köln	Munchen, Madchen, Koln	✅ matches
German ß	Größe	Grosse	⚠️ does not match — ß is an atomic letter (no NFD decomposition)
Norwegian å	Ålesund	Alesund	✅ matches
Norwegian ø, æ	Bjørn, Tromsø	Bjorn, Tromso	⚠️ does not match — ø and æ are atomic letters
Swedish (å, ä, ö)	Göteborg, Malmö	Goteborg, Malmo	✅ matches
Finnish (ä, ö)	Jyväskylä, Hämeenlinna	Jyvaskyla, Hameenlinna	✅ matches
Greek	Ένα	Ενα	✅ matches
Cyrillic	Ёжик, Йогурт	Ежик, Иогурт	✅ matches
Vietnamese	Việt Nam, Hà Nội	Viet Nam, Ha Noi	✅ matches
Arabic (tashkeel)	مُرَحَّباً	مرحبا	✅ matches
Korean (Hangul)	서울, 한국어	exact text	✅ matches (NFD-symmetric on both sides)
Chinese	北京, 你好	exact text	✅ matches (no combining marks; previously broken)
Japanese kanji & katakana	東京, カタカナ	exact text	✅ matches (previously broken)
Intra-word punctuation	co-op, e-mail	coop, email	✅ matches

Scripts that rely on combining marks (Thai vowel signs, Devanagari matras, Japanese hiragana voicing marks like dakuten/handakuten) are NOT fully preserved because every Unicode combining mark is stripped (e.g. สวัสดี → สวสด, が → か). This enables fuzzy matching but loses some semantic precision. Use searchNormalize: false if exact-match behavior is required for those scripts.

Atomic letters (ø, æ, ß, etc.) are not decomposable under NFD and are therefore preserved literally — typing the ASCII fallback (Bjorn, Grosse) will not match the original (Bjørn, Größe). This is a Unicode-level limitation, not a regex limitation.

Punctuation/whitespace are now stripped from normalized values (matching the original ASCII behavior of /[^\w]/g). Search remains symmetric — both labels and the search query go through the same pipeline — so multi-word labels like Việt Nam still match Viet Nam, and labels with intra-word punctuation like co-op now correctly match coop again.

Performance

• Both COMBINING_MARKS_REGEX and NON_WORD_CHARS_REGEX are defined at module scope instead of being re-created inside normalizeString() on every call.
• Build toolchains (e.g. Babel targeting ES5/ES2015) transpile /\p{M}/gu into a ~2 KB character-class regex. Re-compiling that pattern on each keystroke during search was unnecessarily expensive.
• Hoisting to module scope means each regex is compiled once. Benchmarked against 10,000 calls with the actual transpiled pattern: ~29% faster vs. the original in-function regex.
• The added second .replace() pass is O(n) on already-short strings and adds no measurable overhead in real workloads.

Documentation and examples

• Added a unified Multi-language search normalize section that demonstrates a single dropdown spanning Latin (French/Spanish/German/Norwegian/Swedish/Finnish), Greek, Cyrillic, Vietnamese, Chinese, Japanese, Korean, Arabic, and Thai — both with searchNormalize: true and searchNormalize: false variants for direct comparison.
• Added tags variant (showValueAsTags) and popup variant (popupDropboxBreakpoint) sub-sections under the same multi-language data set, each with both searchNormalize: true and false dropdowns.
• Each language entry in the live demo dropdown now has 5–10 representative examples for manual testing.
• Added intra-word punctuation entries (co-op, e-mail) so the regression behavior is visible in the live demo.
• Updated docs/examples.md, docs/assets/script.js, and the table of contents.
• Expanded JSDoc on Utils.normalizeString to call out the limitation that combining-mark-dependent scripts (Thai, Devanagari, hiragana voicing) are NOT fully preserved by the normalization pipeline.

Tests

Added Cypress describe blocks against the unified multi-language dropdowns:

• Multi-language search with searchNormalize: true (~25 specs) covers all listed scripts including positive cases (e.g. Munchen → München, Goteborg → Göteborg, Jyvaskyla → Jyväskylä, Ежик → Ёжик, Viet Nam → Việt Nam, مرحبا → مُرَحَّباً) and the documented atomic-letter limitations (Grosse does NOT match Größe; Bjorn does NOT match Bjørn).
• Multi-language search with searchNormalize: false verifies exact matches succeed (Greek/Cyrillic/Chinese/Japanese/Korean exact text) and that accent-stripped queries correctly find no options across all scripts.
• Intra-word punctuation regression coverage: positive specs verify coop → co-op and email → e-mail under searchNormalize: true; a negative spec verifies coop finds nothing under searchNormalize: false. These guard the second .replace() pass against silent regressions.
• Multi-language tags variant with searchNormalize: true / false — exercise multi-select with showValueAsTags, including diacritic-insensitive search, tag rendering, and tag removal.
• Multi-language popup variant with searchNormalize: true / false — exercise popup mode (popupDropboxBreakpoint) with the same multi-language data.
• Latin diacritics regression suite (brulee → brûlée, cafe → café, nino → niño) preserved.

Does this introduce a breaking change?

• Yes
• No

Behavior for ASCII inputs is preserved vs. the original /[^\w]/g implementation (punctuation- and whitespace-insensitive search). The fix expands correctness to non-Latin scripts; it does not narrow any previously working case.

Validations

Ran regression scenarios in the documentation using the branch - ✅
Run automated tests - ✅

[Copilot]

Pull request overview

This PR updates the search normalization logic to support Greek and Cyrillic text when searchNormalize: true, and adds docs/examples + Cypress coverage to validate the behavior.

Changes:

• Update normalizeString() to strip Unicode combining marks after NFD normalization (instead of stripping non-ASCII “non-word” chars).
• Add documentation examples for Greek/Cyrillic normalization and wire them into the docs demo script.
• Add Cypress E2E coverage for Greek/Cyrillic normalization and Latin-diacritics regression checks.

Reviewed changes

Copilot reviewed 8 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/utils/utils.js	Updates the normalization regex used by searchNormalize.
package.json	Updates devDependencies (build/test tooling) and adds ts-api-utils.
docs/examples.md	Adds a new Greek/Cyrillic “searchNormalize” example section.
docs/assets/virtual-select.js	Updates built docs asset to reflect new normalization logic.
docs/assets/script.js	Initializes new Greek/Cyrillic example selects in the docs demo page.
dist/virtual-select.js	Updates distributed (unminified) build with new normalization logic.
dist/virtual-select.min.js	Updates distributed minified build with new normalization logic.
cypress/e2e/examples.cy.ts	Adds E2E tests for Greek/Cyrillic searchNormalize and Latin regression.
.github/PULL_REQUEST_TEMPLATE.md	Adds a PR template for future contributions.
.claude/settings.local.json	Adds Claude tooling permissions config.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 12 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 12 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 12 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 13 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 13 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 13 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.