Skip to content

Correct and test some behaviour around utf-16 characters - #251

Merged
JonatanWaern merged 4 commits into
mainfrom
more-byte-boundary-fixes
Nov 9, 2026
Merged

JonatanWaern merged 4 commits into
mainfrom
more-byte-boundary-fixes

Conversation

@JonatanWaern

Copy link
Copy Markdown
Contributor

Signed-off-by: Jonatan Waern jonatan.waern@intel.com

Signed-off-by: Jonatan Waern <jonatan.waern@intel.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Token conversion can panic on newline-containing character literals, and repeated line-prefix scans cause quadratic analysis work on lines with many declarations.

2 open issues
What changed in this PR

This PR corrects UTF-16 column handling in the virtual file system’s text-range extraction and adds regression coverage for Unicode documentation strings.

Changes:

  • Convert UTF-16 columns to byte offsets and safely reject invalid ranges.
  • Add Unicode coverage for range extraction, lexing, parser positions, and documentation strings.
  • Document the parser crash fix in the changelog.
File Description
src/​vfs/​mod.rs Correct range extraction and test valid and invalid UTF-16 boundaries.
src/​analysis/​structure/​mod.rs Test Unicode documentation values and source spans.
src/​analysis/​parsing/​parser.rs Test token positions around a supplementary Unicode character.
src/​analysis/​parsing/​lexer.rs Test Unicode string token recognition and byte spans.
CHANGELOG.md Record the Unicode documentation-string crash fix.

Review effort: Balanced

Comment thread src/vfs/mod.rs Resolved
Comment thread src/vfs/mod.rs Resolved
Signed-off-by: Jonatan Waern <jonatan.waern@intel.com>
@JonatanWaern
JonatanWaern requested a balanced review from Copilot November 5, 2026 13:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/vfs/mod.rs Resolved
Comment thread src/vfs/mod.rs Resolved
Comment thread src/vfs/mod.rs Resolved
Comment thread src/vfs/mod.rs Resolved
Comment thread src/vfs/mod.rs Resolved
@JonatanWaern
JonatanWaern requested a balanced review from Copilot November 5, 2026 16:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Signed-off-by: Jonatan Waern <jonatan.waern@intel.com>
@JonatanWaern
JonatanWaern force-pushed the more-byte-boundary-fixes branch from d4a8c73 to 4f01c85 Compare November 6, 2026 10:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues

Review effort: Lite

Comment thread src/analysis/parsing/parser.rs Resolved
Comment thread src/vfs/mod.rs Resolved
Comment thread src/vfs/mod.rs Resolved
Comment thread src/analysis/parsing/parser.rs Resolved
Signed-off-by: Jonatan Waern <jonatan.waern@intel.com>
@JonatanWaern
JonatanWaern requested a balanced review from Copilot November 6, 2026 12:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/analysis/parsing/parser.rs Resolved
Comment thread src/vfs/mod.rs Resolved
Comment thread tests/test_files/unicode_lookup.dml Resolved
Comment thread CHANGELOG.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JonatanWaern
JonatanWaern merged commit 57ff162 into main Nov 9, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants