Skip to content

fix(llm): the JSON extractors scan a reply of unclosed openers in linear time; the code-span test counts CPU growth - #890

Open
antonwolfy wants to merge 2 commits into
intel:mainfrom
antonwolfy:fix/flaky-timing
Open

antonwolfy wants to merge 2 commits into
intel:mainfrom
antonwolfy:fix/flaky-timing

Conversation

@antonwolfy

Copy link
Copy Markdown
Contributor

Summary

Two flaky CI tests, with different causes.

Issue What goes wrong on main
#882 test_code_span_stripping_is_linear_on_backtick_runs bounded each call by 5 s of wall-clock time and failed on a stalled runner (552.37 - 545.31 < 5). _strip_code_spans itself is linear: about 15 ms on the test's largest input, CPU time about 4.7x from n to 4n. The code is fine; the test measured the runner.
#883 extract_json_array("[" * 100000) and the same text in a fence take about 10.2 s each, and extract_json_object("{" * 100000) grows the same way. The last fallback, re.search(r"\[[\s\S]*\]", text) (and the { twin), starts a greedy scan to the end of the text at every opener and backtracks: CPU time from n=4000 to 4n grows 15.9x. With the 60 s pytest timeout and several such cases a slow runner times the test out. This is a code defect: a model reply of that shape stalls the caller.

The fence regex had the same defect on a fence followed by spaces: "```json" + " " * n + "x" grows 15.6x from n to 4n, because \s* gives back one space at a time and rescans the lazy body each time. The caller strips the body, so the \s* is dropped.

For #882 the test now takes the CPU-time growth from n to 4n (tests/growth_timing.py, the fix of #505, used the same way by tests/test_repo_environment.py) for each of the three input shapes. For #883 the block is taken with str.find and str.rfind, which is the span the greedy regex matched (the leftmost opener to the last closer after it), so every result stays the same.

Closes #882
Closes #883

Changes

Other sites of the two classes are tracked in #887 (the arithmetic-expansion regex in the context prefilter), #888 (other quadratic regexes over untrusted text) and #889 (tests that bound the CPU work of a scan by wall-clock time).

Risk

None for the results: extract_json_array and extract_json_object return or reject the same text as before, with the same messages and the same RecursionError to ValueError mapping. A reply of unclosed openers is now rejected in milliseconds instead of seconds. The test change only affects the test.

Testing

On Linux, at 23feb9b (on main a98a0e7):

  • make ci-local: 36 passed at 23feb9b, 0 failed, 14 skipped, the gitleaks scan included. An earlier run, before the last test case (x [1]z) was added, gave 22731 tests passed, 19 skipped, changed-line coverage 100% (12 lines). Skipped: the other Python versions, Docker, semgrep, micromamba, the steps that only run on a pushed PR.
  • The changed-line mutation step of ci-local left two survivors on _outer_block's return line: i != -1 or j > i (equivalent: with no opener the slice is at most the last character, which parses to no array or object) and j + 1 to j + 2, which the case x [1]z now kills.
  • Flaky on CI: tests/test_fuzz_targets.py::test_a_bare_deep_reply_is_a_parse_error_not_a_crash #883 on main's code, the new growth test: 4 failed, 80 passed, first on assert 15.92 < 8 (extract_json_array on "[" * 4000). The ratio is 15.6 to 16.1 for the four cases that carry the defect.
  • Flaky on CI: tests/test_repo_budget.py::test_code_span_stripping_is_linear_on_backtick_runs #882 on main's code: the new test passes (the code is linear), so the mutant pins it: re.sub(r"(+)([\s\S]*?)\1", "", text)in place of the scan fails it onassert 52.7 < 8`, in 2 s.
  • Mutants of Flaky on CI: tests/test_fuzz_targets.py::test_a_bare_deep_reply_is_a_parse_error_not_a_crash #883, each failing its test: the old [ regex alone (assert 15.9 < 8, [brackets-extract_json_array]), the old { regex alone (assert 16.0 < 8, [braces-extract_json_object]), the fence \s* back (assert 16.2 < 8 and 15.9 < 8, both unclosed-fence-spaces cases), the closer taken with find (4 equality cases fail), the opener taken with rfind (2 fail), j + 2 (2 fail).
  • Not run: Python 3.11, 3.12 and 3.14.

Open questions

None.

Revert

A plain revert restores the old regexes and the old wall-clock bound. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…clock time (intel#882)

test_code_span_stripping_is_linear_on_backtick_runs bounded each call by 5 s of wall-clock time, so a stalled runner failed it although _strip_code_spans is linear (about 15 ms on its largest input). It now takes the CPU-time growth from n to 4n (tests/growth_timing.py, the fix of intel#505) on each of the three input shapes and asserts it stays under GROWTH_LIMIT. The three equality asserts and the CodeQL py/redos reason are unchanged.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…ear time (intel#883)

extract_json_array("[" * 100000) and extract_json_object("{" * 100000) took about 10 s each: the last fallback, re.search(r"\[[\s\S]*\]", text), starts a greedy scan to the end at every opener and backtracks, so a run of openers with no closer is quadratic (CPU time ratio from n to 4n about 16). With the 60 s test timeout a slow runner failed test_a_bare_deep_reply_is_a_parse_error_not_a_crash. The block is now text[first opener : last closer + 1], which is the span the greedy regex matched, through a helper shared by both functions.

The fence regex was quadratic too on "```json" followed by n spaces and no closing fence (ratio about 15.6): \s* gave back one space at a time and rescanned the lazy body each time. The body is stripped by the caller, so the \s* is dropped and the result is unchanged.

New tests pin the growth of both extractors on a run of "[", of "{" and of an unclosed fence plus spaces, and that the fallback still finds the same blocks.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

CI at 23feb9b: 27 passing, 3 skipped.

Review details
  • Commit reviewed: 23feb9b against merge base a98a0e7
  • Coverage: 4 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 60k tokens (36k from cache), 15s
  • Tool calls: 7 (5 file_read, 1 ci_status, 1 local_check_results)
  • Linear-time JSON extraction, regression tests, and budget updates: 4 files

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (a98a0e7) to head (23feb9b).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #890   +/-   ##
=======================================
  Coverage   95.79%   95.79%           
=======================================
  Files         429      429           
  Lines       56842    56846    +4     
  Branches     9787     9787           
=======================================
+ Hits        54452    54458    +6     
+ Misses       1487     1486    -1     
+ Partials      903      902    -1     
Files with missing lines Coverage Δ
dokimi/dokimi/llm_json.py 91.80% <100.00%> (+0.57%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Ready to approve · reviewed 23feb9b

Next: nothing required from @antonwolfy. 1 nit optional.

Verified on 23feb9b: ✅ changed tests: 162 passed, fail on main · ✅ full suite with main merged in: 22839 passed · ✅ merges onto main · ✅ 3 broken variants of the changed code: 3 caught by the tests · 2 mutations tried · 6 edge inputs tried

11 checks behind it, each with its evidence
Area What holds How
Extractor results Both extractors return or refuse what the old regex code did on 21 fixed and 3000 seeded inputs, messages included, which is what the changelog fragment says ran it, edge input (text)
Unclosed replies A reply of 100000 [ or {, or a fence and 100000 spaces, is refused by both extractors in under two seconds each; the largest takes about 5 ms of CPU ran it, edge input (text)
Plain values {}, [] and {"a": null, "extra": [1]} come back unchanged; truncated or wrong-shape replies ({"a": 1, [1] for an object, 7, null) are refused with ValueError ran it, edge input (structured)
Line endings and whitespace CRLF and CR fences, U+2028 and ideographic-space padding, ß, and a lone surrogate before an array are parsed; empty, blank and None replies are refused as empty ran it, edge input (text)
Reversed delimiters A reply with a ] and no [, or ] x [, is refused with the No JSON array found message; the object extractor does the same for } x { and x } ran it, edge input (delimiter_order)
Last-block end Changing j + 1 to j + 2 in the return makes the changed tests fail (2 cases), so the end of the block is pinned broke the code, tests caught it
Changed tests The changed test files pass on the head, 162 cases: the growth cases of both extractors and of _strip_code_spans, the outermost-block cases, and the existing deep-reply cases tests on the head
Growth cases on main On main 4 of the 6 linear-growth cases fail (brackets with the array extractor, braces with the object extractor, and both fence cases). The two that cannot reach the old regex, brackets with the object extractor and braces with the array extractor, pass on main same tests on main
Code-span test The code-span growth test passes on main, so the change to it is test-only, as the description says same tests on main
Whole suite The whole test suite passes with main merged in: 22839 passed, 0 failed full suite, main merged in
Merge The head merges onto main with no conflict merge onto main

],
)
def test_the_last_fallback_still_takes_the_outermost_block(name: str, text: str, want: str) -> None:
assert _outcome(name, text) == want

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Input a] (no [ in the reply): a guard of i != -1 or j > i changes the message to a decode error and all 162 changed tests still pass.

Input a], a reply with a ] and no [. Head: ValueError: No JSON array found in LLM response: 'a]'. With i != -1 and j > i changed to i != -1 or j > i, the same input gives Expecting value: line 1 column 1 (char 0), and the changed tests still pass, because _outcome keeps only the exception type. The description calls this mutant equivalent; it is equivalent in the returned value and the exception type only. Expected: the head message, which the changelog's returns or rejects is unchanged promises. Pin it with pytest.raises(ValueError, match="No JSON array found") on a].

Also at: dokimi/dokimi/llm_json.py:18.

Fix, run on 23feb9b: the tests now catch the broken variant; tests and lint are no worse.

Suggested change
assert _outcome(name, text) == want
assert _outcome(name, text) == want
def test_a_reply_with_a_closer_and_no_opener_keeps_the_head_message() -> None:
from dokimi.llm_json import extract_json_array
with pytest.raises(ValueError, match="No JSON array found"):
extract_json_array("a]")

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants