Repository navigation
fix(llm): the JSON extractors scan a reply of unclosed openers in linear time; the code-span test counts CPU growth - #890
antonwolfy wants to merge 2 commits into
Conversation
…clock time (intel#882) test_code_span_stripping_is_linear_on_backtick_runs bounded each call by 5 s of wall-clock time, so a stalled runner failed it although _strip_code_spans is linear (about 15 ms on its largest input). It now takes the CPU-time growth from n to 4n (tests/growth_timing.py, the fix of intel#505) on each of the three input shapes and asserts it stays under GROWTH_LIMIT. The three equality asserts and the CodeQL py/redos reason are unchanged. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…ear time (intel#883) extract_json_array("[" * 100000) and extract_json_object("{" * 100000) took about 10 s each: the last fallback, re.search(r"\[[\s\S]*\]", text), starts a greedy scan to the end at every opener and backtracks, so a run of openers with no closer is quadratic (CPU time ratio from n to 4n about 16). With the 60 s test timeout a slow runner failed test_a_bare_deep_reply_is_a_parse_error_not_a_crash. The block is now text[first opener : last closer + 1], which is the span the greedy regex matched, through a helper shared by both functions. The fence regex was quadratic too on "```json" followed by n spaces and no closing fence (ratio about 15.6): \s* gave back one space at a time and rescanned the lazy body each time. The body is stripped by the caller, so the \s* is dropped and the result is unchanged. New tests pin the growth of both extractors on a run of "[", of "{" and of an unclosed fence plus spaces, and that the fallback still finds the same blocks. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #890 +/- ##
=======================================
Coverage 95.79% 95.79%
=======================================
Files 429 429
Lines 56842 56846 +4
Branches 9787 9787
=======================================
+ Hits 54452 54458 +6
+ Misses 1487 1486 -1
+ Partials 903 902 -1
🚀 New features to boost your workflow:
|
vchamarthi
left a comment
There was a problem hiding this comment.
🟢 Ready to approve · reviewed 23feb9b
Next: nothing required from @antonwolfy. 1 nit optional.
Verified on 23feb9b: ✅ changed tests: 162 passed, fail on main · ✅ full suite with main merged in: 22839 passed · ✅ merges onto main · ✅ 3 broken variants of the changed code: 3 caught by the tests · 2 mutations tried · 6 edge inputs tried
11 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Extractor results | Both extractors return or refuse what the old regex code did on 21 fixed and 3000 seeded inputs, messages included, which is what the changelog fragment says | ran it, edge input (text) |
| Unclosed replies | A reply of 100000 [ or {, or a fence and 100000 spaces, is refused by both extractors in under two seconds each; the largest takes about 5 ms of CPU |
ran it, edge input (text) |
| Plain values | {}, [] and {"a": null, "extra": [1]} come back unchanged; truncated or wrong-shape replies ({"a": 1, [1] for an object, 7, null) are refused with ValueError |
ran it, edge input (structured) |
| Line endings and whitespace | CRLF and CR fences, U+2028 and ideographic-space padding, ß, and a lone surrogate before an array are parsed; empty, blank and None replies are refused as empty |
ran it, edge input (text) |
| Reversed delimiters | A reply with a ] and no [, or ] x [, is refused with the No JSON array found message; the object extractor does the same for } x { and x } |
ran it, edge input (delimiter_order) |
| Last-block end | Changing j + 1 to j + 2 in the return makes the changed tests fail (2 cases), so the end of the block is pinned |
broke the code, tests caught it |
| Changed tests | The changed test files pass on the head, 162 cases: the growth cases of both extractors and of _strip_code_spans, the outermost-block cases, and the existing deep-reply cases |
tests on the head |
| Growth cases on main | On main 4 of the 6 linear-growth cases fail (brackets with the array extractor, braces with the object extractor, and both fence cases). The two that cannot reach the old regex, brackets with the object extractor and braces with the array extractor, pass on main |
same tests on main |
| Code-span test | The code-span growth test passes on main, so the change to it is test-only, as the description says | same tests on main |
| Whole suite | The whole test suite passes with main merged in: 22839 passed, 0 failed | full suite, main merged in |
| Merge | The head merges onto main with no conflict | merge onto main |
| ], | ||
| ) | ||
| def test_the_last_fallback_still_takes_the_outermost_block(name: str, text: str, want: str) -> None: | ||
| assert _outcome(name, text) == want |
There was a problem hiding this comment.
🟡 Nit: Input a] (no [ in the reply): a guard of i != -1 or j > i changes the message to a decode error and all 162 changed tests still pass.
Input a], a reply with a ] and no [. Head: ValueError: No JSON array found in LLM response: 'a]'. With i != -1 and j > i changed to i != -1 or j > i, the same input gives Expecting value: line 1 column 1 (char 0), and the changed tests still pass, because _outcome keeps only the exception type. The description calls this mutant equivalent; it is equivalent in the returned value and the exception type only. Expected: the head message, which the changelog's returns or rejects is unchanged promises. Pin it with pytest.raises(ValueError, match="No JSON array found") on a].
Also at: dokimi/dokimi/llm_json.py:18.
Fix, run on 23feb9b: the tests now catch the broken variant; tests and lint are no worse.
| assert _outcome(name, text) == want | |
| assert _outcome(name, text) == want | |
| def test_a_reply_with_a_closer_and_no_opener_keeps_the_head_message() -> None: | |
| from dokimi.llm_json import extract_json_array | |
| with pytest.raises(ValueError, match="No JSON array found"): | |
| extract_json_array("a]") |
Summary
Two flaky CI tests, with different causes.
maintest_code_span_stripping_is_linear_on_backtick_runsbounded each call by 5 s of wall-clock time and failed on a stalled runner (552.37 - 545.31 < 5)._strip_code_spansitself is linear: about 15 ms on the test's largest input, CPU time about 4.7x from n to 4n. The code is fine; the test measured the runner.extract_json_array("[" * 100000)and the same text in a fence take about 10.2 s each, andextract_json_object("{" * 100000)grows the same way. The last fallback,re.search(r"\[[\s\S]*\]", text)(and the{twin), starts a greedy scan to the end of the text at every opener and backtracks: CPU time from n=4000 to 4n grows 15.9x. With the 60 s pytest timeout and several such cases a slow runner times the test out. This is a code defect: a model reply of that shape stalls the caller.The fence regex had the same defect on a fence followed by spaces:
"```json" + " " * n + "x"grows 15.6x from n to 4n, because\s*gives back one space at a time and rescans the lazy body each time. The caller strips the body, so the\s*is dropped.For #882 the test now takes the CPU-time growth from n to 4n (
tests/growth_timing.py, the fix of #505, used the same way bytests/test_repo_environment.py) for each of the three input shapes. For #883 the block is taken withstr.findandstr.rfind, which is the span the greedy regex matched (the leftmost opener to the last closer after it), so every result stays the same.Closes #882
Closes #883
Changes
dokimi/tests/test_repo_budget.py:test_code_span_stripping_is_linear_on_backtick_runsassertsgrowth(...) < GROWTH_LIMITon each of its three inputs (n=1000) instead oftime.monotonic() - started < 5. The three equality asserts and the CodeQL py/redos reason are unchanged. No package change for Flaky on CI: tests/test_repo_budget.py::test_code_span_stripping_is_linear_on_backtick_runs #882.dokimi/dokimi/llm_json.py:_outer_block(text, opener, closer)returnstext[first opener : last closer + 1], and both extractors use it for their last fallback. The fence regex loses its\s*.dokimi/tests/test_fuzz_targets.py:test_extractors_scan_a_reply_without_a_closer_in_linear_time(growth of both extractors on"[" * n,"{" * nand an unclosed fence plus spaces, n=4000) andtest_the_last_fallback_still_takes_the_outermost_block(same blocks as before:x [1, [2]] y,a ] [1] b,x [1]z,[{"a": 1}]for the object extractor, and{"a": 1} then }, which stays aValueError).test_a_bare_deep_reply_is_a_parse_error_not_a_crashandtest_deep_nesting_is_a_parse_error_not_a_crashare unchanged and now run in milliseconds.Other sites of the two classes are tracked in #887 (the arithmetic-expansion regex in the context prefilter), #888 (other quadratic regexes over untrusted text) and #889 (tests that bound the CPU work of a scan by wall-clock time).
Risk
None for the results:
extract_json_arrayandextract_json_objectreturn or reject the same text as before, with the same messages and the sameRecursionErrortoValueErrormapping. A reply of unclosed openers is now rejected in milliseconds instead of seconds. The test change only affects the test.Testing
On Linux, at
23feb9b(onmaina98a0e7):make ci-local: 36 passed at23feb9b, 0 failed, 14 skipped, the gitleaks scan included. An earlier run, before the last test case (x [1]z) was added, gave 22731 tests passed, 19 skipped, changed-line coverage 100% (12 lines). Skipped: the other Python versions, Docker, semgrep, micromamba, the steps that only run on a pushed PR.ci-localleft two survivors on_outer_block's return line:i != -1 or j > i(equivalent: with no opener the slice is at most the last character, which parses to no array or object) andj + 1toj + 2, which the casex [1]znow kills.main's code, the new growth test:4 failed, 80 passed, first onassert 15.92 < 8(extract_json_arrayon"[" * 4000). The ratio is 15.6 to 16.1 for the four cases that carry the defect.main's code: the new test passes (the code is linear), so the mutant pins it:re.sub(r"(+)([\s\S]*?)\1", "", text)in place of the scan fails it onassert 52.7 < 8`, in 2 s.[regex alone (assert 15.9 < 8,[brackets-extract_json_array]), the old{regex alone (assert 16.0 < 8,[braces-extract_json_object]), the fence\s*back (assert 16.2 < 8and15.9 < 8, bothunclosed-fence-spacescases), the closer taken withfind(4 equality cases fail), the opener taken withrfind(2 fail),j + 2(2 fail).Open questions
None.
Revert
A plain revert restores the old regexes and the old wall-clock bound. Nothing is stored.
Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)