[coco-alen]

Closes #642

plan_cache_budget turned prefill overlap on whenever the slot cap allowed 2*num_experts slots, without checking the budget. When the budget fit num_experts slots plus the KV reserve but not 2*num_experts, max(lo, min(raw, hi)) clamped the plan up to the 2*num_experts floor, and the total <= budget_bytes assert fired after the full expert-bank load. The line meant to drop overlap for a tiny budget (overlap = overlap and moe_cache_size >= 2 * num_experts) could never fire, because the clamp had already raised moe_cache_size to that floor.

Overlap is now dropped, and the floor lowered to num_experts, before the clamp whenever the budget cannot fund 2*num_experts slots beside the KV reserve. The dead line is removed. The fallback shows in the existing --moe-cache-auto resolved ... (prefill_overlap=False) log line.

The change only affects plans that used to fail. The new branch runs only when raw < 2*num_experts, and every such plan on main hit the assert: 2*num_experts slots plus the KV reserve exceed the budget.

The issue links a fork commit (gdevenyi/FreeToken@27fe671) that applies the same fallback and also rewrites the "budget too small" message. This PR includes only the fallback.

Testing

RTX 6000 Ada (48 GB), driver 595.99.02, torch 2.14.1+cu130, Python 3.12. plan_cache_budget is pure arithmetic, so I tested it directly rather than through a server start; I have no 16 GB card to hit this budget on a real load.

• The issue's arguments, plan_cache_budget(2557061534, 2764800, 1683456, 512, 24576, <overlap>, 128, 24576):
  • main, overlap on: AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=128 pages) needs 3046637568 B > budget 2557061534 B
  • branch, overlap on: (846, 129, False), using 2556186624 of 2557061534 B. This is the same plan as overlap off on both main and the branch.
• New tests/engine/test_cache_budget.py::test_budget_below_the_overlap_floor_disables_prefill_overlap, next to the existing test for the slot-cap case. With the fix reverted, it fails with AssertionError: cache budget too small: minimum plan (moe=8 slots, kv=10 pages) needs 900 B > budget 700 B. With the fix, it passes.
• pytest tests/engine: 133 passed, 2 skipped. main: 132 passed, 2 skipped.