Skip to content

fix(analyzer): bound token-gap compaction on mixed-script content - #1077

Open
Varshith-Kali wants to merge 2 commits into
NVIDIA:mainfrom
Varshith-Kali:fix-1075-token-gap-compaction
Open

Varshith-Kali wants to merge 2 commits into
NVIDIA:mainfrom
Varshith-Kali:fix-1075-token-gap-compaction

Conversation

@Varshith-Kali

Copy link
Copy Markdown
Contributor

Fixes #1075

v2.13.4 can stall in the static stage on mixed-script content (CJK plus Sinhala ZWJ). The reporter saw the scan never reach the LLM stage; the py-spy stack pointed at the token-gap compaction path in src/skillspector/artifacts.py.

Root cause: no literal infinite loop, but pathological super-linear cost. The gap prefilter matched at the first non-ASCII character, so every CJK/Sinhala window paid the full per-character Python walk even though the real files contain zero gap spans. The 1,515-entry confusable regex cost about 0.8 s per 284K-character CJK window returning None. The same windows were re-derived about 15 times, once per analyzer, with no reuse.

Fix (all in src/skillspector/artifacts.py):

  • Exact gap-character class (32 ranges, derived exhaustively over U+0000..U+10FFFF) gates the walk.
  • Cheaper exact gap predicate (one category lookup).
  • Gap spans memoized per text (lru_cache, 64 entries) with the runtime deadline checked per yielded span.
  • Confusable check via str.translate length comparison instead of the regex.

Verification:

  • security_text_views on 2 MB gap-free CJK: 8.4-11.0 s to 1.38-1.48 s; confusable check 4.4 s to 0.28 s; system_prompt_leakage on the reported file: 3.75 s to 2.35 s CPU mean (-37%).
  • Equivalence old vs new: 0 mismatches at four levels (full-range prefilter, predicate, confusable on 108 texts, spans on 106 texts x 2 modes).
  • 8 new regression tests, red pre-fix (timing test 25.67 s), green post-fix. Full analyzer suite: 12,466 passed, 0 failed. Ruff and format clean, mypy 0 errors in artifacts.py.

Caveats, stated plainly:

  • The reporter's CPU flatline on Windows was not reproduced on Linux and is unexplained; this fix removes the pathological costs but does not claim to explain it.
  • A cache-miss walk runs without intermediate deadline checks, bounded by the 256K-character window (worst measured 0.65 s); the deadline is still enforced per yielded span and between views.
  • The new timing test is load-sensitive (it failed once at 6.49 s under concurrent suite load, passes at 1.4 s quiet); the spy tests carry the regression guarantee.

…IDIA#1075)

The static stage stalled on mixed-script content (CJK plus Sinhala ZWJ):
every security window paid a per-character Python walk in
_token_bridging_gap_spans even when it held no gap character at all,
because the whole-string skip only excluded printable ASCII text, and
the 1,515-entry confusable regex cost about a second per CJK window
even when nothing matched. The same windows were then re-derived once
per analyzer with no reuse.

Bound the path: an exact gap-character class derived exhaustively from
_is_token_gap_character gates the walk, the gap predicate is one
category lookup instead of a regex fullmatch plus two category calls,
gap spans are memoized per text across analyzers, and the confusable
check is a str.translate length comparison instead of the regex search.
Behavior is unchanged: the prefilter class, predicate, confusable
check, and yielded spans were verified equivalent to the old code over
the full code-point range and the reported skill files.

Add regression tests that fail pre-fix: a prefilter/predicate
equivalence test, spy tests proving gap-free CJK never enters the walk
and repeated scans reuse the first walk, a confusable guard test, and
an end-to-end timing bound on 2 MB of gap-free CJK.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]
Request changes at 0d90cf3003235a1eab29d5bf2455825a60e9ac86.

Value and readiness: The exact gap prefilter and translate-based confusable predicate directly target the redundant work in #1075, and deterministic spy tests support the performance intent. One availability issue in the new memoization needs a bound before this head is ready.

Material findings:

  • [P2, required] src/skillspector/artifacts.py:1844-1850 now retains complete Python tuples of gap spans in a process-global 64-entry LRU. Previously these spans streamed. The 256,000-character caller window cap does not adequately bound the retained representation: alternating a and U+200D yields about 128,000 separate contextual spans per window, and both require_word_boundaries modes are cached separately. Thirty-two distinct such windows can retain about 8.2 million pair tuples plus 16.4 million integer objects (roughly 1 GB on typical 64-bit CPython), from about 16 MB of UTF-8 source, until unrelated entries evict them. This is a static representation estimate, not a measured RSS result. Dense input can therefore exchange the targeted CPU stall for memory pressure or process termination; per-span runtime checks do not bound this retained memory. Bound cache admission by span count/retained bytes, use a compact bounded representation, or avoid caching dense results while preserving the gap-free/sparse fast path. Add a deterministic dense-input cache-bound regression.
  • [P2, optional] tests/nodes/analyzers/test_token_gap_fast_paths.py:201-216 adds a strict 4-second wall-clock assertion despite the PR reporting a 6.49-second failure under concurrent load. The deterministic no-walk/cache-reuse tests already establish the regression contract. Move the timing assertion to a benchmark or use an explicit performance-test runner to avoid load-dependent unit CI failures.

PIC tradeoffs: The release/product owner should preserve the mixed-script speedup while requiring a memory admission bound for dense attacker-controlled text. Removing in-walk deadline checks trades prompt interruption for a bounded full-window miss; the production static-runner paths inspected do bound normal windows to 256,000 characters, and the caller checks around views remain. The reporter's Windows CPU-flatline symptom remains unexplained, as the author states.

Verification and gaps: Reviewed the complete three-file diff, exact-head predicate/span/view logic and tests, exact-head static-runner window callers, #1075 and discussion, and cutoff PR history (no reviews/comments/threads). Hosted CI run 42632706962 is attributed to this exact head; test-unit reports 20,989 passed, 24 skipped, 258 deselected and 4 xfailed, including both changed test modules. All six reported checks succeeded. No contributor code, tests, imports, builds, regex probes or setup were executed locally; timing/RSS and the Windows hang were not reproduced. Author benchmark/equivalence claims are not independently established by this review.

…IDIA#1077)

The 64-entry span cache admitted every result, so alternating word and
gap characters could retain about 128,000 span tuples per 256K window,
roughly 1 GB across 32 distinct windows. Admit only results with at most
8,192 spans (under 60 MB worst case across the cache); dense results are
recomputed per call, which is the pre-cache behavior, while the gap-free
and sparse fast paths stay cached.

Drop the load-dependent 4-second wall-clock assertion from the unit test;
the deterministic no-walk and cache-reuse tests pin the regression
contract. Add a deterministic dense-input cache-bound regression.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

Fixed both findings.

Required bound: _cached_token_bridging_gap_spans no longer uses an unconditional lru_cache. It is now a small admission-bounded cache (64 entries, same as before) that only retains results with at most 8,192 spans. A span costs about 112 bytes, so the cache retains under 60 MB worst case. Dense results are computed fresh on every call and never stored, which is the pre-cache behavior, while the gap-free and sparse fast paths stay cached. The 5,000-span cache-reuse test still passes; a 20,000-span dense window is computed correctly and leaves the cache empty (currsize 0).

Optional timing assertion: dropped the strict 4-second wall-clock assertion from the unit test. The deterministic no-walk and cache-reuse tests pin the regression contract; the timing check was load-dependent.

Added a deterministic dense-input cache-bound regression: alternating a and U+200D (20,000 spans) is computed correctly, leaves the cache empty, and a second scan re-walks instead of hitting the cache. The new test fails on the old code and passes on the new code.

Speedup preserved: 2 MB of gap-free CJK through security_text_views takes 1.99 s here (pre-fix 8.4 to 11 s), with the second run served from cache in 0.00 s.

Evidence: tests/nodes/analyzers/test_token_gap_fast_paths.py and test_security_text_predicate_cache.py 353 passed; analyzer suite 11,961 passed, 4 skipped, 4 xfailed. New head 2323c51.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants