Repository navigation
NaturalStringComparer sorts file10 before file9 when the digits are outside the BMP (surrogate-pair Nd digits compared as text) #58
Description
What's wrong
The class remarks in Sorting/NaturalStringComparer.cs promise:
A run of Unicode decimal digits (category
Nd) is a number, whatever script it is written in, and is compared by the value it spells rather than by its code points.
That holds only for digits in the Basic Multilingual Plane. Many Nd digits live above U+FFFF (Osmanya U+104A0–U+104A9, Brahmi, Adlam, Mathematical Bold/Monospace digits U+1D7CE–U+1D7FF, …), and in UTF-16 each is a surrogate pair. Neither the chunk regex (\d+)|(\D+) (line 21) nor char.IsDigit(xMatch.Value[0]) (line 66) recognises a surrogate code unit as a digit, so the run lands in a \D+ text chunk and is compared ordinally, code unit by code unit — never by value.
Repro
A console app compiling NaturalStringComparer.cs:
Compare("file𝟏𝟎", "file𝟗") = -8 // 10 vs 9, expected > 0
Compare("file𐒡𐒠", "file𐒩") = -8 // Osmanya 10 vs 9, expected > 0
Compare("file𝟗", "file10") = 2 // 9 vs 10, expected < 0
Compare("file𝟏𝟎", "file10") = 4 // same value 10 on both sides
CharUnicodeInfo.GetUnicodeCategory("𝟗", 0) = DecimalDigitNumber, GetDecimalDigitValue = 9
Sorting [Osmanya 10, 9, 2] -> 10, 2, 9
Why it matters
Any list of names numbered in one of these scripts sorts in lexical rather than natural order, which is the exact failure the comparer exists to prevent, and it contradicts the documented contract. Mixing such names with ASCII-numbered ones also gives orderings unrelated to the numeric values.
This is distinct from #55 (BMP non-ASCII digits make the comparer intransitive); both stem from chunks being classified per UTF-16 unit, so they are probably best fixed together.
Suggested fix
Tokenize by code point rather than by char:
- Walk the string, using
CharUnicodeInfo.GetUnicodeCategory(string, int)/CharUnicodeInfo.GetDecimalDigitValue(string, int)(both available on netstandard2.0), advancing by 2 at a surrogate pair. - Build digit runs from those code points, and make
CompareNumericChunks,SkipLeadingZerosandDigitValuework on code-point indices.
Acceptance criteria
Compare("file𝟗", "file𝟏𝟎") < 0, and the same for Osmanya digits.- A test covering a supplementary-plane digit run alongside the existing non-ASCII digit tests.
Activity
matt-edmondson commentedon Sep 27, 2026
Triage
NaturalStringComparerchunking and numeric comparison. Owner: @matt-edmondson\d+|\D+regex with a code-point splitter that usesCharUnicodeInfo.GetUnicodeCategory(string, int)/GetDecimalDigitValue(string, int). Its description gives"file𝟗" < "file𝟏𝟎"as a result. This issue is not a duplicate of NaturalStringComparer is intransitive for non-ASCII digits (٥ < 10 < z < ٥), so sort output depends on input order #55, but Keep NaturalStringComparer transitive across digit scripts #57 probably resolves both.file𝟏𝟎vsfile10being equal in value). Add the supplementary-plane test there if it is missing, and addFixes #58to the PR.Generated by Claude Code
matt-edmondson commentedon Oct 5, 2026
Triage
This is already fixed on
main(da6906b). The PR that fixed #55 (#57) replaced the\d+|\D+regex with a code-point walk, but it didn't link this issue, so the issue stayed open.NaturalStringComparer.IsDigitAtclassifies withCharUnicodeInfo.GetUnicodeCategory(string, int).WidthAtadvances by 2 at a surrogate pair. Digit values come fromGetDecimalDigitValue(string, int).NaturalStringComparerTestsalready covers this issue's acceptance criteria with mathematical digits:Compare("file\U0001D7D7", "file\U0001D7CF\U0001D7CE") < 0"file𝟗"against"file10","file8", and"file9"Closing as completed. Reopen if the Osmanya repro still misbehaves.
Generated by Claude Code