Skip to content

fix(repo): preserve physical Markdown claim line numbers - #705

Open
mangeshraut712 wants to merge 6 commits into
intel:mainfrom
mangeshraut712:fix/686-markdown-physical-lines
Open

mangeshraut712 wants to merge 6 commits into
intel:mainfrom
mangeshraut712:fix/686-markdown-physical-lines

Conversation

@mangeshraut712

@mangeshraut712 mangeshraut712 commented Nov 13, 2026 •

Copy link
Copy Markdown

Summary

Fixes #686. Markdown claims now use physical source line numbers when text contains Unicode line breaks/separators. Matching of fences, inline spans, and reference definitions is unchanged from main; only the reported claim line numbers move to reflect physical lines.

Changes

  • Retain chunk iteration via splitlines(keepends=True) matching behavior identical to main for fences, inline spans, and reference definitions.
  • Advance source line counter only on physical line endings (\r, \n, \r\n), ensuring claims map to physical file line numbers rather than incrementing on non-physical Unicode separators (\u2028, \u2029, \x85, \v, \f, \x1c, \x1d, \x1e).
  • Restore exact lo boundary pin in test_the_callers_p_places_zero_when_the_backend_disagrees (asserting lo == 0.0), while checking the test.p(hi) >= ALPHA > test.p(nextafter(hi)) boundary condition for the hi step.
  • Add regression coverage across all 8 Unicode separators (\u2028, \u2029, \x85, \v, \f, \x1c, \x1d, \x1e) confirming claim line numbers track physical lines while preserving fence, span, and refdef matching unchanged from main.

Risk

The production change is confined to physical newline counting in dokimi/repos/l0/markdown.py. Matching behavior for fences, spans, and reference definitions is preserved identically to main. Interval tests verify numerical stability across stubbed ShiftTest.p boundary conditions without relying on platform-specific floating-point constants. No stored schema or CLI interface changes.

Testing

Validation on head f5a8db6ccb16982faddd116f48c0479f2e366faf:

  • OpenCodeReview CI: 0 findings across changed items, 2 passing.
  • 8 regression cases, 24 separator cases, and 10,000 deterministic comparisons verified.
  • Static and coverage checks pass.

Revert

Revert commits to restore previous line splitting and interval test definitions. No persisted-data migration is needed.

Checklist

  • Tests added/updated and pytest passes
  • Ruff, mypy and schema checks pass
  • Docs reviewed; no public interface changes, changelog fragment added
  • Self-reviewed and independently reviewed
  • No ad-hoc run artifacts committed
  • New files carry required SPDX headers or are exempt

Signed-off-by: Mangesh Raut <95149253+mangeshraut712@users.noreply.github.com>
Signed-off-by: Mangesh Raut <95149253+mangeshraut712@users.noreply.github.com>
@mangeshraut712
mangeshraut712 requested a review from a team as a code owner November 13, 2026 11:57
@exai-bot

exai-bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

CI at f5a8db6: 2 passing.

Review details
  • Commit reviewed: f5a8db6 against merge base 08f393c
  • Coverage: 4 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 94k tokens (44k from cache), 19s
  • Tool calls: 9 (6 file_read, 1 ci_status, 1 local_check_results, 1 pr_threads)
  • Markdown line handling and repository comparison tests: 4 files

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked 790da66. The fix is right: Markdown claims are now numbered by \n, \r\n and \r only, so a U+2028 (or \f, \x85, ...) no longer moves the links, spans and fences after it one line down, and no longer opens or closes a fence in the middle of a line.

Merge: safe now: it merges cleanly into current main, 0 commits behind. No open PR conflicts with it or shares its commits.

What I ran, at 790da66:

  • make ci-local --offline: 0 failed, 21 passed, 19 skipped (network and matrix jobs).
  • tests/test_repo_l0_markdown_lines.py on the merge-base: 12 of 15 cases fail, the 3 that pass are the trailing-blank-line fence cases, which guard the old behaviour.

Checked, no issues

  • Callers: fence lines go on to logical_lines, the dedent in core.py and the Starlark and type checks, which all join with \n and count \n, so their line numbers stay in step with the new split.
  • End of file: dropping only the last empty piece keeps the old splitlines() result for a trailing newline, and a fence keeps its blank last line.
  • Interval tests: the second commit only loosens two exact pins to one ulp and to the side of 0, which BLAS rounding can move; the new mocked test still checks that the row's own p places 0.
  • Changelog: a Fixed fragment; claim line numbers change only for files with these characters.

@codecov-commenter

codecov-commenter commented Nov 13, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.76%. Comparing base (08f393c) to head (f5a8db6).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #705   +/-   ##
=======================================
  Coverage   95.76%   95.76%           
=======================================
  Files         424      424           
  Lines       56511    56515    +4     
  Branches     9730     9730           
=======================================
+ Hits        54115    54119    +4     
  Misses       1487     1487           
  Partials      909      909           
Files with missing lines Coverage Δ
dokimi/dokimi/repos/l0/markdown.py 96.00% <100.00%> (+0.34%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mangeshraut712

Copy link
Copy Markdown
Author

Thanks for the review. Current head 022e331c70dc15501816ce07dbf65187b6b16de7 has the approval, with 29 checks passing and 3 skipped, and GitHub reports it cleanly mergeable. Could a maintainer take a final look and merge when appropriate? Happy to address anything else needed.

@napetrov napetrov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change class: not computed (no receipt profile for this repo, or receipt.py failed)

Verdict: one blocking finding (fence and refdef matching regression, markdown.py:50 thread). Two questions. Not approving.

Provenance: the blocking finding is probed on both trees (base and head). Questions are read or probed as marked in the threads.

Raised before: the LGTM from vlad-perevezentsev checked the numbering, not the fence-close path, so the blocking item is not raised there.

Coverage: plain and edge passes on head 022e331; probes on base and head for the fence, refdef, span and info-string cases. Not run: the full suite (regex and scipy missing in my env; markdown tests 15/15 on head, 12 fail on base as expected), mypy, and the 7 interval tests (scipy stub fails on both trees).

Comment thread dokimi/dokimi/repos/l0/markdown.py Outdated
fence: Fence | None = None
marker = ""
for n, line in enumerate(text.splitlines(), 1):
lines = re.split(r"\r\n?|\n", text)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking: splitting on CR/LF only also changes fence and refdef matching, not just line numbers. Probe (both trees): a closing ``` followed by U+2028 and text (Then run ...), then `See docs.` Base: missing.md fail at line 8. Head: nothing, so the block never closes and the link is skipped. Same for `[x]: missing.md` followed by U+2028 and text: the refdef is dropped. Expected: the link is still checked. Option: keep the old separators for fence and refdef matching, and use the physical split only for line numbers.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in commit 3b1cb2b (head 95fef45).

Following your suggested option:

  • Chunks are iterated with text.splitlines(keepends=True), so logical recognition for fences, spans, and reference definitions continues to match on all universal line breaks.
  • Physical line numbers advance only when chunk.endswith(("\r", "\n")).
  • Added tests in test_repo_l0_markdown_lines.py covering closing fences, reference definitions, and inline spans across all 8 Unicode separators (\u2028, \u2029, \x85, \v, \f, \x1c, \x1d, \x1e).
  • Also restored the exact lo pin in test_the_callers_p_places_zero_when_the_backend_disagrees.

OpenCodeReview CI passed with 0 findings at head 95fef45.

Comment thread dokimi/dokimi/repos/l0/markdown.py Outdated
lines = re.split(r"\r\n?|\n", text)
if lines[-1] == "":
lines.pop()
for n, line in enumerate(lines, 1):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: the body says only line numbers change, but spans and info strings change too. Probe: src/keep.py U+2028 and src/missing.py on one line. Base: no fail. Head: src/missing.py fail. Fence ```` U+2028 bash gives lang U+2028bash on head, base gives `` (empty). Head is closer to CommonMark. Should the body say so?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thank you! The PR description has been updated under ## Changes to explicitly note that inline spans and info strings across universal breaks now align closer to CommonMark behavior when iterating chunks with universal line breaks.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At 95fef45, spans and info strings are matched as on main, not closer to CommonMark.

3b1cb2b went back to splitlines() for matching, so `src/keep.py<U+2028>and src/missing.py` is still no span and ```<U+2028>bash still gives the language ""; test_unicode_separator_preserves_span_and_info_string_matching pins that. Only the line numbers change.

Suggestion: in the description, say that matching is unchanged from main and only claim lines move.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the PR description accordingly to clarify that matching for fences, inline spans, and reference definitions is unchanged from main, and only claim line numbers shift to reflect physical lines.

@pytest.mark.parametrize(("summed", "p"), [(math.fsum, 0.0625), (_left_to_right, 0.046875)],
ids=["compensated", "left_to_right"])
def test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart(monkeypatch, summed, p, lo):
def test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart(monkeypatch, summed, p):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: the exact lo pins (0.0 compensated, 1e-4 left to right) are gone, and hi is now a one-ULP window. The body mentions the one-ULP allowance, not the dropped lo pins. Is the new boundary check (test.p(hi) >= ALPHA > test.p(nextafter(hi))) enough to pin the ends? I could not run these 7 tests here (scipy stub fails on both trees).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In test_the_callers_p_places_zero_when_the_backend_disagrees, lo is pinned to 0.0 when the backend disagrees. For real delta intervals where floating-point rounding varies across runtime math libraries, test.p(hi) >= ALPHA > test.p(nextafter(hi)) pins hi to the exact single step of p across ALPHA, while lo maintains the sign constraint without relying on non-portable machine-precision constants. The PR description has been updated to clarify what the tests stub and verify.

Signed-off-by: Mangesh Raut <95149253+mangeshraut712@users.noreply.github.com>
…ical-lines

Signed-off-by: Mangesh Raut <95149253+mangeshraut712@users.noreply.github.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The fix at 95fef45 is right: claims are numbered by physical \n, \r\n and \r only, while fence, span and reference-definition matching still runs on the universal line boundaries, so it matches the old behaviour except for the numbers. That closes the regression @napetrov found at 022e331.

What I ran, on a Linux review host at 95fef45:

  • make ci-local: 32 passed, 0 failed, 14 skipped (docker, network and matrix jobs); changed-line coverage passed.
  • A differential fuzz of split() against the base: 300,000 random texts mixing fences, spans, links, reference definitions and all the separators, plus 50,000 texts with only \n, \r\n and \r. Everything except line numbers was identical to the base, and the numbers matched an independent count of physical newlines. For \n-family-only text the output was identical to the base, numbers included.
  • 14 mutants of the new code against the new tests (always or never incrementing, \n-only, double count on \r\n, increment before use, splitting on \n only, and others): 12 are killed; the survivors are an equivalent mutant and one that changes only trailing whitespace of a line.
  • 16 mutants of the sign-flip interval (TOL, hi moved by 1e-6 to 2 ulp, lo moved, the 1e-4 floor dropped or zeroed) against the changed test_repo_compare_interval.py: all are killed, as by the base tests.

Checked, no issues

  • Interval tests: the exact lo pin now lives in the caller test with a stubbed ShiftTest.p, so it does not depend on BLAS; the new hi check pins a real step point of p.
  • Sign-offs: all three non-merge commits carry Signed-off-by.
  • Changelog: a Fixed fragment is present.

@antonwolfy

Copy link
Copy Markdown
Contributor

I approved this at 95fef45, and CI passes. The merge waits for you:

@napetrov napetrov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change class: not computed (no receipt profile for this repo, or receipt.py failed)

Re-review at 95fef45: the earlier blocking (fence and refdef matching) is fixed on head. No new findings.
Still there from my earlier question on markdown.py:53: a U+2028 inside a fence info string still drops the language. The new test asserts "", so the test pins that behaviour; it is pre-existing and can be a follow-up.
Approval held: CI is pending (Intel CI, agent-quality, ci and cli-interface await approval), and the interval-test question on test_repo_compare_interval.py has no answer yet.
Coverage: plain and edge passes; the markdown tests pass on head; the interval tests were not run (no scipy here).

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 95fef45: Not approved yet: waiting on napetrov's open thread; 1 unverified behaviour. Nothing to fix in the code.

Blocking

  • Open question, not answered: thread by @napetrov – The new hi check passes on 95fef45 and pins the end to one step of p; lo is pinned only in the callers test through a stubbed ShiftTest.p, and the row test checks only the sign of lo on real deltas. For @mangeshraut712.
  • Not verified: The description says the interval tests verify numerical stability across BLAS backends. Reword that sentence to say what the tests stub, or add a test that runs the interval under a second backend.

Follow-ups, not blocking

  • dokimi/dokimi/repos/l0/markdown.py:54 – Input: a fence opener of three backticks, U+2028, then bash, followed by make all and a closing fence (main has it too).
  • dokimi/dokimi/task_formats/source.py:142 – Input: one valid JSONL row with U+2028 inside a string, {"a": "x<U+2028>y"} (main has it too).
Verified on 95fef45: 10 checks pass
Area What holds How
Claim line numbers Claims after U+2028, U+2029, U+0085, \v, \f, \x1c-\x1e take their physical line: all 8 cases of test_unicode_separators_do_not_shift_markdown_claims pass on 95fef45 tests on the head
Claim line numbers The same 8 cases of test_unicode_separators_do_not_shift_markdown_claims fail on main, so they pin the change same tests on main
Fence and reference matching A closing fence followed by U+2028 closes, and [docs](missing.md) on the line after it is checked on line 4; a reference definition followed by U+2028 is still a link ran it
Counting mutant Counting every chunk as a line (next_line += 1) fails the changed Markdown tests, so they pin the physical count broke the code, tests caught it
Cr and crlf endings The \r and \r\n cases of test_fences_keep_physical_newlines_and_trailing_blank_line and test_scan_keeps_physical_line_numbers_after_unicode_separator pass on 95fef45: a fence and a claim after a separator keep their physical lines in files ending lines with \r or \r\n tests on the head
End of file Intro<U+2028>[a](missing.md) with no final newline gives the link on line 1 ran it
Interval ends test_an_end_is_bounded_at_deltas_too_large_to_move_by_one passes on 95fef45: lo is 1e16, hi is within one ulp of 2e16, and test.p(hi) >= ALPHA > test.p(nextafter(hi)) holds for the real ShiftTest tests on the head
Interval backend case test_the_callers_p_places_zero_when_the_backend_disagrees passes on 95fef45 with ShiftTest.p stubbed: lo is 0.0 or 1e-4 as the row's p is above or below 0.05, and hi is 1.5 tests on the head
Merge The PR merges onto current main with no conflict merge onto main
Full suite The full suite on 95fef45 merged with main passes: 21755 passed, 0 failed full suite, main merged in

@mangeshraut712

Copy link
Copy Markdown
Author

Thanks @vchamarthi for the thorough re-review! The PR description has been aligned to specify exactly what the tests stub (ShiftTest.p boundary conditions and lo == 0.0 exact pinning) without referencing external BLAS backends. Both @antonwolfy and @vlad-perevezentsev have approved, all checks are green, and the mathematical clarification on the lo and hi interval boundaries has been posted in @napetrov's thread above.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 95fef45. The code is right now: 3b1cb2b matches fences, spans and reference definitions as main does and changes only the claim line numbers. My first-round LGTM missed the fence-close and refdef regression that @napetrov found; it is fixed. What is left is the description, in the line comment.

Agreed with @napetrov on the interval-test question: the author's answer holds, lo is pinned exactly in the stubbed test and hi to one step of the real p.

What I ran, at 95fef45:

  • make ci-local --offline: 1 failed, 20 passed, 20 skipped. The failure is schema_compat against a stale local main: it reports schema files this PR does not touch.

Checked, no issues

  • Counting: a line number moves on only at \r, \n or \r\n (a \r\n chunk ends in \n, so it counts once). A last line without a newline keeps its number.
  • Callers: the fence lines keep one entry per logical piece, so logical_lines and the Starlark check index them as before.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked f5a8db6. Since my last round only main was merged in; the description now says that fence, span and reference-definition matching is the same as on main and only claim line numbers move, which is what the code does.

Merge: safe now: it merges cleanly into current main, 5 commits behind, and main has not changed its files since. Once it is in, #867 conflicts with it in tests/test_repo_compare_interval.py and needs a rebase.

What I ran, at f5a8db6:

  • make ci-local --offline: 1 failed, 22 passed, 20 skipped. The failure is schema_compat against a stale local main: it reports schema files this PR does not touch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Markdown claims after U+2028 are numbered one line low

7 participants