Allow configurable per-request LLM token and prompt-byte limits for constrained deployments #1003

Summary

We run SkillSpector in an internal deployment using an OpenAI-compatible endpoint. Large-file LLM requests fail in our environment even when they fit within the configured model input budget.

We would like to configure a smaller per-request input-token or prompt-byte limit to accommodate deployment-specific constraints. Currently, this requires modifying SkillSpector’s code.

This request is for configurable request-size limits independent of model context metadata and output-token settings.

Observed behavior

In our downstream deployment:

• The configured model input budget was 196,500 tokens.
• A Python file of 227,195 bytes / 4,937 lines repeatedly failed across semantic analyzers and TP4.
• Failed prompts included a 138,173-byte TP4 chunk and larger whole-file requests.
• The failures appeared as APIConnectionError / ReadError.
• Applying a 20,000 estimated-token ceiling and a 90,000-byte prompt ceiling allowed all 203 submitted LLM calls to complete without retries.
• Model configuration, output-token cap and concurrency were retained.

We have not isolated the exact rejecting layer in our deployment. These observations do not establish a universal gateway limit or a model defect.

This is a reliability improvement rather than a demonstrated speed improvement: total scan time changed from approximately 334 seconds to 347 seconds, while previously failed analysis completed successfully.

The measurements come from a downstream build with local changes, not an unchanged-upstream reproduction.

Current implementation

1. In llm_analyzer_base.py, LLMAnalyzerBase.__init__ derives its input budget directly from the model:

   self._input_budget = get_max_input_tokens(model)
   

2. LLMAnalyzerBase.get_batches obtains a byte cap from _cli_max_input_bytes(). This supports CLI-provider limits, but does not expose a deployment-specific HTTP prompt-byte setting.

3. In mcp_tool_poisoning.py, TP4 uses its own planner and a fixed per-batch token ceiling:

   TP4_MAX_BATCH_INPUT_TOKENS = 32_000
   

Changing the model context registry is not an equivalent workaround because context metadata also influences output-token budgeting.

Requested configuration

Suggested names:

For example, our deployment would use:

SKILLSPECTOR_MAX_BATCH_INPUT_TOKENS=20000
SKILLSPECTOR_MAX_PROMPT_BYTES=90000
SKILLSPECTOR_TP4_MAX_BATCH_INPUT_TOKENS=20000

These values are examples from our environment, not proposed upstream defaults.

Suggested implementation

1. Common token budget — LLMAnalyzerBase.__init__

   Apply the optional request ceiling after resolving the model input budget:

   model_budget = get_max_input_tokens(model)
   configured_cap = optional_positive_int_env(
       "SKILLSPECTOR_MAX_BATCH_INPUT_TOKENS"
   )
   self._input_budget = (
   
   min(model_budget, configured_cap)
   
   if configured_cap is not None
   
   else model_budget
   
   )
   
   

   Leave model context metadata and output-token configuration unchanged.

2. Common byte budget — LLMAnalyzerBase.get_batches

   Combine the optional deployment cap with any existing provider cap:

   deployment_cap = optional_positive_int_env(
       "SKILLSPECTOR_MAX_PROMPT_BYTES"
   )
   provider_cap = _cli_max_input_bytes()
   caps = [
   
   cap for cap in (deployment_cap, provider_cap)
   
   if cap is not None
   
   ]
   
   max_input_bytes = min(caps) if caps else None
   
   

   Reuse the existing byte-aware chunking path. Account for fixed prompt content, line-number prefixes, findings, metadata and applicable structured-output instructions before determining the content budget.

   The setting should have a documented measurement boundary: prompt UTF-8 bytes are not necessarily the full serialized HTTP request size.

3. TP4 — _check_tp4 and _tp4_line_chunks

   Apply the effective token ceiling as the minimum of:

   • the model input budget;
   • TP4’s configured ceiling, or its existing default;
   • the common request ceiling, when configured.

   Extend TP4’s separate chunk planner to honor the configured byte ceiling after subtracting its fixed prompt overhead.

4. Preflight checks and incomplete analysis

   Recheck the completed input before submission in both synchronous and asynchronous paths.

   If a fixed prompt or indivisible line cannot fit, retain an explicit size-limit failure for that batch and continue unrelated batches. Do not silently truncate source or report omitted content as successfully inspected.

   Reuse the accounting and preflight mechanisms from PR fix(llm): bound complete serialized inputs before provider calls #980 where appropriate, rather than introducing competing token-counting logic.

5. Configuration validation

   Accept positive integers. Unset or blank values should preserve current upstream behavior. Invalid values should produce a clear warning without echoing untrusted environment contents and retain the existing behavior.

Acceptance criteria

• Unset configuration preserves existing batch planning and provider limits.
• Configured token and byte ceilings affect common semantic analyzers, meta-analysis and TP4.
• Smaller model input budgets and stricter provider limits still take precedence.
• Model context metadata and output-token settings remain unchanged.
• Tests cover multibyte UTF-8 text, line-number overhead, findings, metadata and schema overhead.
• Original source ranges, existing overlap and finding attribution remain intact.
• Oversized indivisible input is explicitly marked incomplete without aborting unrelated batches.
• Existing batch-count, concurrency, aggregate-byte and workflow-deadline limits remain enforced.
• README documentation explains defaults, precedence and measurement units.

Related work

• HTTP batches can exceed model input budgets through line rounding and prompt overhead #977 / fix(llm): bound complete serialized inputs before provider calls #980 address accurate input accounting against model budgets.
• LLM batches for large files exceed MAX_INPUT_BYTES on CLI providers, so the scan never reaches a verdict #853 / fix(llm): keep CLI prompts under MAX_INPUT_BYTES when batching large files #855 address CLI-provider byte limits.
• Make TP4 batch capacity configurable for skills with many executable examples #941 / feat: configure TP4 batches and default workflows to 18 minutes #945 address TP4 batch-count capacity.

This proposal adds deployment-specific configuration that can impose a smaller request limit even when a request otherwise fits the model’s supported context.