Repository navigation
Correct and test some behaviour around utf-16 characters - #251
Conversation
Signed-off-by: Jonatan Waern <jonatan.waern@intel.com>
There was a problem hiding this comment.
🟡 Changes recommended
Token conversion can panic on newline-containing character literals, and repeated line-prefix scans cause quadratic analysis work on lines with many declarations.
2 open issues
What changed in this PR
This PR corrects UTF-16 column handling in the virtual file system’s text-range extraction and adds regression coverage for Unicode documentation strings.
Changes:
- Convert UTF-16 columns to byte offsets and safely reject invalid ranges.
- Add Unicode coverage for range extraction, lexing, parser positions, and documentation strings.
- Document the parser crash fix in the changelog.
| File | Description |
|---|---|
src/vfs/mod.rs |
Correct range extraction and test valid and invalid UTF-16 boundaries. |
src/analysis/structure/mod.rs |
Test Unicode documentation values and source spans. |
src/analysis/parsing/parser.rs |
Test token positions around a supplementary Unicode character. |
src/analysis/parsing/lexer.rs |
Test Unicode string token recognition and byte spans. |
CHANGELOG.md |
Record the Unicode documentation-string crash fix. |
Review effort: Balanced
Signed-off-by: Jonatan Waern <jonatan.waern@intel.com>
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
5 open issues
Using aHashMap<usize, ...>forunicode_offsetsadds hashing overhead even thoughrowis a… · New Using aHashMap<usize, ...>forunicode_offsetsadds hashing overhead even thoughrowis a… · New Using aHashMap<usize, ...>forunicode_offsetsadds hashing overhead even thoughrowis a… · New Thebytefield/variable name is ambiguous (it reads like a single byte value rather than a byte… · New Thebytefield/variable name is ambiguous (it reads like a single byte value rather than a byte… · New
2 resolved since last review
Review effort: Lite
Signed-off-by: Jonatan Waern <jonatan.waern@intel.com>
d4a8c73 to
4f01c85
Compare
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
read_tokenwill panic if called with aTextFilethat doesn’t match the token’s originating… · NewTextFile::load_rangenow has UTF-16 column semantics (per the new docstring). Ifload_rangeis… · Newload_rangenow performs UTF-16->byte conversion by scanning line text (byte_in_str_utf16) for… · Newbyte_rangeis documented as half-open, but stored as a(usize, usize)tuple. Using… · New
Review effort: Lite
Signed-off-by: Jonatan Waern <jonatan.waern@intel.com>
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
Usingexpect(...)here will panic/crash the process if a token is ever read against a different… · New This diff removes previouslypubAPIs (Vfs::load_spanandFile::load_range). If this crate is… · NewSPDX-License-Identifiervalues are expected to be valid SPDX license expressions. `Apache-2.0 and… · New Minor consistency/readability: consider capitalizingUTF-16here (it’s an acronym and referenced… · New
4 resolved since last review
read_tokenwill panic if called with aTextFilethat doesn’t match the token’s originating…load_rangenow performs UTF-16->byte conversion by scanning line text (byte_in_str_utf16) for…TextFile::load_rangenow has UTF-16 column semantics (per the new docstring). Ifload_rangeis…byte_rangeis documented as half-open, but stored as a(usize, usize)tuple. Using…
Review effort: Lite
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the Unicode-handling and character-literal changes.
No open issues 🎉
4 resolved since last review
Usingexpect(...)here will panic/crash the process if a token is ever read against a different…SPDX-License-Identifiervalues are expected to be valid SPDX license expressions. `Apache-2.0 and… This diff removes previouslypubAPIs (Vfs::load_spanandFile::load_range). If this crate is… Minor consistency/readability: consider capitalizingUTF-16here (it’s an acronym and referenced…
Review effort: Balanced



Signed-off-by: Jonatan Waern jonatan.waern@intel.com