Repository navigation
fix(repo): preserve physical Markdown claim line numbers - #705
mangeshraut712 wants to merge 6 commits into
Conversation
Signed-off-by: Mangesh Raut <95149253+mangeshraut712@users.noreply.github.com>
Signed-off-by: Mangesh Raut <95149253+mangeshraut712@users.noreply.github.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s). CI at Review details
|
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked 790da66. The fix is right: Markdown claims are now numbered by \n, \r\n and \r only, so a U+2028 (or \f, \x85, ...) no longer moves the links, spans and fences after it one line down, and no longer opens or closes a fence in the middle of a line.
Merge: safe now: it merges cleanly into current main, 0 commits behind. No open PR conflicts with it or shares its commits.
What I ran, at 790da66:
make ci-local --offline: 0 failed, 21 passed, 19 skipped (network and matrix jobs).tests/test_repo_l0_markdown_lines.pyon the merge-base: 12 of 15 cases fail, the 3 that pass are the trailing-blank-line fence cases, which guard the old behaviour.
Checked, no issues
- Callers: fence lines go on to
logical_lines, the dedent incore.pyand the Starlark and type checks, which all join with\nand count\n, so their line numbers stay in step with the new split. - End of file: dropping only the last empty piece keeps the old
splitlines()result for a trailing newline, and a fence keeps its blank last line. - Interval tests: the second commit only loosens two exact pins to one ulp and to the side of 0, which BLAS rounding can move; the new mocked test still checks that the row's own p places 0.
- Changelog: a
Fixedfragment; claim line numbers change only for files with these characters.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #705 +/- ##
=======================================
Coverage 95.76% 95.76%
=======================================
Files 424 424
Lines 56511 56515 +4
Branches 9730 9730
=======================================
+ Hits 54115 54119 +4
Misses 1487 1487
Partials 909 909
🚀 New features to boost your workflow:
|
|
Thanks for the review. Current head |
napetrov
left a comment
There was a problem hiding this comment.
Change class: not computed (no receipt profile for this repo, or receipt.py failed)
Verdict: one blocking finding (fence and refdef matching regression, markdown.py:50 thread). Two questions. Not approving.
Provenance: the blocking finding is probed on both trees (base and head). Questions are read or probed as marked in the threads.
Raised before: the LGTM from vlad-perevezentsev checked the numbering, not the fence-close path, so the blocking item is not raised there.
Coverage: plain and edge passes on head 022e331; probes on base and head for the fence, refdef, span and info-string cases. Not run: the full suite (regex and scipy missing in my env; markdown tests 15/15 on head, 12 fail on base as expected), mypy, and the 7 interval tests (scipy stub fails on both trees).
| fence: Fence | None = None | ||
| marker = "" | ||
| for n, line in enumerate(text.splitlines(), 1): | ||
| lines = re.split(r"\r\n?|\n", text) |
There was a problem hiding this comment.
blocking: splitting on CR/LF only also changes fence and refdef matching, not just line numbers. Probe (both trees): a closing ``` followed by U+2028 and text (Then run ...), then `See docs.` Base: missing.md fail at line 8. Head: nothing, so the block never closes and the link is skipped. Same for `[x]: missing.md` followed by U+2028 and text: the refdef is dropped. Expected: the link is still checked. Option: keep the old separators for fence and refdef matching, and use the physical split only for line numbers.
There was a problem hiding this comment.
Addressed in commit 3b1cb2b (head 95fef45).
Following your suggested option:
- Chunks are iterated with
text.splitlines(keepends=True), so logical recognition for fences, spans, and reference definitions continues to match on all universal line breaks. - Physical line numbers advance only when
chunk.endswith(("\r", "\n")). - Added tests in
test_repo_l0_markdown_lines.pycovering closing fences, reference definitions, and inline spans across all 8 Unicode separators (\u2028,\u2029,\x85,\v,\f,\x1c,\x1d,\x1e). - Also restored the exact
lopin intest_the_callers_p_places_zero_when_the_backend_disagrees.
OpenCodeReview CI passed with 0 findings at head 95fef45.
| lines = re.split(r"\r\n?|\n", text) | ||
| if lines[-1] == "": | ||
| lines.pop() | ||
| for n, line in enumerate(lines, 1): |
There was a problem hiding this comment.
question: the body says only line numbers change, but spans and info strings change too. Probe: src/keep.py U+2028 and src/missing.py on one line. Base: no fail. Head: src/missing.py fail. Fence ```` U+2028 bash gives lang U+2028bash on head, base gives `` (empty). Head is closer to CommonMark. Should the body say so?
There was a problem hiding this comment.
Yes, thank you! The PR description has been updated under ## Changes to explicitly note that inline spans and info strings across universal breaks now align closer to CommonMark behavior when iterating chunks with universal line breaks.
There was a problem hiding this comment.
At 95fef45, spans and info strings are matched as on main, not closer to CommonMark.
3b1cb2b went back to splitlines() for matching, so `src/keep.py<U+2028>and src/missing.py` is still no span and ```<U+2028>bash still gives the language ""; test_unicode_separator_preserves_span_and_info_string_matching pins that. Only the line numbers change.
Suggestion: in the description, say that matching is unchanged from main and only claim lines move.
There was a problem hiding this comment.
Updated the PR description accordingly to clarify that matching for fences, inline spans, and reference definitions is unchanged from main, and only claim line numbers shift to reflect physical lines.
| @pytest.mark.parametrize(("summed", "p"), [(math.fsum, 0.0625), (_left_to_right, 0.046875)], | ||
| ids=["compensated", "left_to_right"]) | ||
| def test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart(monkeypatch, summed, p, lo): | ||
| def test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart(monkeypatch, summed, p): |
There was a problem hiding this comment.
question: the exact lo pins (0.0 compensated, 1e-4 left to right) are gone, and hi is now a one-ULP window. The body mentions the one-ULP allowance, not the dropped lo pins. Is the new boundary check (test.p(hi) >= ALPHA > test.p(nextafter(hi))) enough to pin the ends? I could not run these 7 tests here (scipy stub fails on both trees).
There was a problem hiding this comment.
In test_the_callers_p_places_zero_when_the_backend_disagrees, lo is pinned to 0.0 when the backend disagrees. For real delta intervals where floating-point rounding varies across runtime math libraries, test.p(hi) >= ALPHA > test.p(nextafter(hi)) pins hi to the exact single step of p across ALPHA, while lo maintains the sign constraint without relying on non-portable machine-precision constants. The PR description has been updated to clarify what the tests stub and verify.
Signed-off-by: Mangesh Raut <95149253+mangeshraut712@users.noreply.github.com>
…ical-lines Signed-off-by: Mangesh Raut <95149253+mangeshraut712@users.noreply.github.com>
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The fix at 95fef45 is right: claims are numbered by physical \n, \r\n and \r only, while fence, span and reference-definition matching still runs on the universal line boundaries, so it matches the old behaviour except for the numbers. That closes the regression @napetrov found at 022e331.
What I ran, on a Linux review host at 95fef45:
make ci-local: 32 passed, 0 failed, 14 skipped (docker, network and matrix jobs); changed-line coverage passed.- A differential fuzz of
split()against the base: 300,000 random texts mixing fences, spans, links, reference definitions and all the separators, plus 50,000 texts with only\n,\r\nand\r. Everything except line numbers was identical to the base, and the numbers matched an independent count of physical newlines. For\n-family-only text the output was identical to the base, numbers included. - 14 mutants of the new code against the new tests (always or never incrementing,
\n-only, double count on\r\n, increment before use, splitting on\nonly, and others): 12 are killed; the survivors are an equivalent mutant and one that changes only trailing whitespace of a line. - 16 mutants of the sign-flip interval (
TOL,himoved by 1e-6 to 2 ulp,lomoved, the 1e-4 floor dropped or zeroed) against the changedtest_repo_compare_interval.py: all are killed, as by the base tests.
Checked, no issues
- Interval tests: the exact
lopin now lives in the caller test with a stubbedShiftTest.p, so it does not depend on BLAS; the newhicheck pins a real step point ofp. - Sign-offs: all three non-merge commits carry
Signed-off-by. - Changelog: a
Fixedfragment is present.
|
I approved this at
|
napetrov
left a comment
There was a problem hiding this comment.
Change class: not computed (no receipt profile for this repo, or receipt.py failed)
Re-review at 95fef45: the earlier blocking (fence and refdef matching) is fixed on head. No new findings.
Still there from my earlier question on markdown.py:53: a U+2028 inside a fence info string still drops the language. The new test asserts "", so the test pins that behaviour; it is pre-existing and can be a follow-up.
Approval held: CI is pending (Intel CI, agent-quality, ci and cli-interface await approval), and the interval-test question on test_repo_compare_interval.py has no answer yet.
Coverage: plain and edge passes; the markdown tests pass on head; the interval tests were not run (no scipy here).
vchamarthi
left a comment
There was a problem hiding this comment.
Reviewed 95fef45: Not approved yet: waiting on napetrov's open thread; 1 unverified behaviour. Nothing to fix in the code.
Blocking
- Open question, not answered: thread by @napetrov – The new
hicheck passes on95fef45and pins the end to one step ofp;lois pinned only in the callers test through a stubbedShiftTest.p, and the row test checks only the sign ofloon real deltas. For @mangeshraut712. - Not verified: The description says the interval tests verify numerical stability across BLAS backends. Reword that sentence to say what the tests stub, or add a test that runs the interval under a second backend.
Follow-ups, not blocking
dokimi/dokimi/repos/l0/markdown.py:54– Input: a fence opener of three backticks, U+2028, thenbash, followed bymake alland a closing fence (main has it too).dokimi/dokimi/task_formats/source.py:142– Input: one valid JSONL row with U+2028 inside a string,{"a": "x<U+2028>y"}(main has it too).
Verified on 95fef45: 10 checks pass
| Area | What holds | How |
|---|---|---|
| Claim line numbers | Claims after U+2028, U+2029, U+0085, \v, \f, \x1c-\x1e take their physical line: all 8 cases of test_unicode_separators_do_not_shift_markdown_claims pass on 95fef45 |
tests on the head |
| Claim line numbers | The same 8 cases of test_unicode_separators_do_not_shift_markdown_claims fail on main, so they pin the change |
same tests on main |
| Fence and reference matching | A closing fence followed by U+2028 closes, and [docs](missing.md) on the line after it is checked on line 4; a reference definition followed by U+2028 is still a link |
ran it |
| Counting mutant | Counting every chunk as a line (next_line += 1) fails the changed Markdown tests, so they pin the physical count |
broke the code, tests caught it |
| Cr and crlf endings | The \r and \r\n cases of test_fences_keep_physical_newlines_and_trailing_blank_line and test_scan_keeps_physical_line_numbers_after_unicode_separator pass on 95fef45: a fence and a claim after a separator keep their physical lines in files ending lines with \r or \r\n |
tests on the head |
| End of file | Intro<U+2028>[a](missing.md) with no final newline gives the link on line 1 |
ran it |
| Interval ends | test_an_end_is_bounded_at_deltas_too_large_to_move_by_one passes on 95fef45: lo is 1e16, hi is within one ulp of 2e16, and test.p(hi) >= ALPHA > test.p(nextafter(hi)) holds for the real ShiftTest |
tests on the head |
| Interval backend case | test_the_callers_p_places_zero_when_the_backend_disagrees passes on 95fef45 with ShiftTest.p stubbed: lo is 0.0 or 1e-4 as the row's p is above or below 0.05, and hi is 1.5 |
tests on the head |
| Merge | The PR merges onto current main with no conflict | merge onto main |
| Full suite | The full suite on 95fef45 merged with main passes: 21755 passed, 0 failed |
full suite, main merged in |
|
Thanks @vchamarthi for the thorough re-review! The PR description has been aligned to specify exactly what the tests stub ( |
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 95fef45. The code is right now: 3b1cb2b matches fences, spans and reference definitions as main does and changes only the claim line numbers. My first-round LGTM missed the fence-close and refdef regression that @napetrov found; it is fixed. What is left is the description, in the line comment.
Agreed with @napetrov on the interval-test question: the author's answer holds, lo is pinned exactly in the stubbed test and hi to one step of the real p.
What I ran, at 95fef45:
make ci-local --offline: 1 failed, 20 passed, 20 skipped. The failure isschema_compatagainst a stale localmain: it reports schema files this PR does not touch.
Checked, no issues
- Counting: a line number moves on only at
\r,\nor\r\n(a\r\nchunk ends in\n, so it counts once). A last line without a newline keeps its number. - Callers: the fence lines keep one entry per logical piece, so
logical_linesand the Starlark check index them as before.
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked f5a8db6. Since my last round only main was merged in; the description now says that fence, span and reference-definition matching is the same as on main and only claim line numbers move, which is what the code does.
Merge: safe now: it merges cleanly into current main, 5 commits behind, and main has not changed its files since. Once it is in, #867 conflicts with it in tests/test_repo_compare_interval.py and needs a rebase.
What I ran, at f5a8db6:
make ci-local --offline: 1 failed, 22 passed, 20 skipped. The failure isschema_compatagainst a stale localmain: it reports schema files this PR does not touch.
Summary
Fixes #686. Markdown claims now use physical source line numbers when text contains Unicode line breaks/separators. Matching of fences, inline spans, and reference definitions is unchanged from
main; only the reported claim line numbers move to reflect physical lines.Changes
splitlines(keepends=True)matching behavior identical tomainfor fences, inline spans, and reference definitions.\r,\n,\r\n), ensuring claims map to physical file line numbers rather than incrementing on non-physical Unicode separators (\u2028,\u2029,\x85,\v,\f,\x1c,\x1d,\x1e).loboundary pin intest_the_callers_p_places_zero_when_the_backend_disagrees(assertinglo == 0.0), while checking thetest.p(hi) >= ALPHA > test.p(nextafter(hi))boundary condition for thehistep.\u2028,\u2029,\x85,\v,\f,\x1c,\x1d,\x1e) confirming claim line numbers track physical lines while preserving fence, span, and refdef matching unchanged frommain.Risk
The production change is confined to physical newline counting in
dokimi/repos/l0/markdown.py. Matching behavior for fences, spans, and reference definitions is preserved identically tomain. Interval tests verify numerical stability across stubbedShiftTest.pboundary conditions without relying on platform-specific floating-point constants. No stored schema or CLI interface changes.Testing
Validation on head
f5a8db6ccb16982faddd116f48c0479f2e366faf:Revert
Revert commits to restore previous line splitting and interval test definitions. No persisted-data migration is needed.
Checklist