Repository navigation
fix(analyzer): bound token-gap compaction on mixed-script content - #1077
Varshith-Kali wants to merge 2 commits into
Conversation
…IDIA#1075) The static stage stalled on mixed-script content (CJK plus Sinhala ZWJ): every security window paid a per-character Python walk in _token_bridging_gap_spans even when it held no gap character at all, because the whole-string skip only excluded printable ASCII text, and the 1,515-entry confusable regex cost about a second per CJK window even when nothing matched. The same windows were then re-derived once per analyzer with no reuse. Bound the path: an exact gap-character class derived exhaustively from _is_token_gap_character gates the walk, the gap predicate is one category lookup instead of a regex fullmatch plus two category calls, gap spans are memoized per text across analyzers, and the confusable check is a str.translate length comparison instead of the regex search. Behavior is unchanged: the prefilter class, predicate, confusable check, and yielded spans were verified equivalent to the old code over the full code-point range and the reported skill files. Add regression tests that fail pre-fix: a prefilter/predicate equivalence test, spy tests proving gap-free CJK never enters the walk and repeated scans reuse the first walk, a confusable guard test, and an end-to-end timing bound on 2 MB of gap-free CJK. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Request changes at 0d90cf3003235a1eab29d5bf2455825a60e9ac86.
Value and readiness: The exact gap prefilter and translate-based confusable predicate directly target the redundant work in #1075, and deterministic spy tests support the performance intent. One availability issue in the new memoization needs a bound before this head is ready.
Material findings:
- [P2, required]
src/skillspector/artifacts.py:1844-1850now retains complete Python tuples of gap spans in a process-global 64-entry LRU. Previously these spans streamed. The 256,000-character caller window cap does not adequately bound the retained representation: alternatingaand U+200D yields about 128,000 separate contextual spans per window, and both require_word_boundaries modes are cached separately. Thirty-two distinct such windows can retain about 8.2 million pair tuples plus 16.4 million integer objects (roughly 1 GB on typical 64-bit CPython), from about 16 MB of UTF-8 source, until unrelated entries evict them. This is a static representation estimate, not a measured RSS result. Dense input can therefore exchange the targeted CPU stall for memory pressure or process termination; per-span runtime checks do not bound this retained memory. Bound cache admission by span count/retained bytes, use a compact bounded representation, or avoid caching dense results while preserving the gap-free/sparse fast path. Add a deterministic dense-input cache-bound regression. - [P2, optional]
tests/nodes/analyzers/test_token_gap_fast_paths.py:201-216adds a strict 4-second wall-clock assertion despite the PR reporting a 6.49-second failure under concurrent load. The deterministic no-walk/cache-reuse tests already establish the regression contract. Move the timing assertion to a benchmark or use an explicit performance-test runner to avoid load-dependent unit CI failures.
PIC tradeoffs: The release/product owner should preserve the mixed-script speedup while requiring a memory admission bound for dense attacker-controlled text. Removing in-walk deadline checks trades prompt interruption for a bounded full-window miss; the production static-runner paths inspected do bound normal windows to 256,000 characters, and the caller checks around views remain. The reporter's Windows CPU-flatline symptom remains unexplained, as the author states.
Verification and gaps: Reviewed the complete three-file diff, exact-head predicate/span/view logic and tests, exact-head static-runner window callers, #1075 and discussion, and cutoff PR history (no reviews/comments/threads). Hosted CI run 42632706962 is attributed to this exact head; test-unit reports 20,989 passed, 24 skipped, 258 deselected and 4 xfailed, including both changed test modules. All six reported checks succeeded. No contributor code, tests, imports, builds, regex probes or setup were executed locally; timing/RSS and the Windows hang were not reproduced. Author benchmark/equivalence claims are not independently established by this review.
…IDIA#1077) The 64-entry span cache admitted every result, so alternating word and gap characters could retain about 128,000 span tuples per 256K window, roughly 1 GB across 32 distinct windows. Admit only results with at most 8,192 spans (under 60 MB worst case across the cache); dense results are recomputed per call, which is the pre-cache behavior, while the gap-free and sparse fast paths stay cached. Drop the load-dependent 4-second wall-clock assertion from the unit test; the deterministic no-walk and cache-reuse tests pin the regression contract. Add a deterministic dense-input cache-bound regression. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
|
Fixed both findings. Required bound: Optional timing assertion: dropped the strict 4-second wall-clock assertion from the unit test. The deterministic no-walk and cache-reuse tests pin the regression contract; the timing check was load-dependent. Added a deterministic dense-input cache-bound regression: alternating Speedup preserved: 2 MB of gap-free CJK through Evidence: tests/nodes/analyzers/test_token_gap_fast_paths.py and test_security_text_predicate_cache.py 353 passed; analyzer suite 11,961 passed, 4 skipped, 4 xfailed. New head 2323c51. |
Fixes #1075
v2.13.4 can stall in the static stage on mixed-script content (CJK plus Sinhala ZWJ). The reporter saw the scan never reach the LLM stage; the py-spy stack pointed at the token-gap compaction path in src/skillspector/artifacts.py.
Root cause: no literal infinite loop, but pathological super-linear cost. The gap prefilter matched at the first non-ASCII character, so every CJK/Sinhala window paid the full per-character Python walk even though the real files contain zero gap spans. The 1,515-entry confusable regex cost about 0.8 s per 284K-character CJK window returning None. The same windows were re-derived about 15 times, once per analyzer, with no reuse.
Fix (all in src/skillspector/artifacts.py):
Verification:
Caveats, stated plainly: