Route Lab

An offline model-routing simulator with UCB exploration, budget limits, and reproducible traces.

Built as a focused AI engineering lab: inspectable algorithms, deterministic examples, and explicit failure behavior. Runs offline with the Python standard library. No API keys, model downloads, or paid services.

Run it

git clone https://github.com/izzyfrm/route-lab.git
cd route-lab
python -m route_lab --rounds 200 --budget 10 --seed 7

Requires Python 3.11 or newer. Commands run from the repository root and work in PowerShell, bash, and zsh. An editable install with python -m pip install -e . also exposes the route-lab command; direct module execution needs no installation.

How it works

Affordable candidate models → UCB1 selection → simulated success feedback → cost penalized reward → updated arm estimates. Every untried affordable arm is explored before confidence bounds take over. Candidate filtering occurs before selection, so a costly model cannot overrun the simulated budget. The seed fixes feedback randomness for a repeatable run.

Where it fits

Explore the tradeoff between exploration, quality, and cost. Import Arm and simulate to define your own synthetic scenarios. Use --trace to inspect every decision. Real deployment would need delayed feedback, task features, provider failure handling, and persistent state.

Example output

See the checked demo output. It comes from the quickstart command and is a small fixture demonstration, not a performance benchmark.

Verify

python -m unittest discover -s tests -v

Tests cover core behavior and edge cases. GitHub Actions runs the suite on Python 3.11, 3.12, and 3.13. Local verification is recorded in VALIDATION.md.

Limitations and tradeoffs

All models and prices are fictional. This is a stationary multi armed bandit simulation, not a live LLM gateway, contextual router, or measured cost saving claim. UCB operates on cost penalized rewards and optimizes immediate reward, not guaranteed total successes under a budget. No network requests or real purchases occur.

Project layout

Path	Purpose
route_lab.py	Importable implementation and CLI
tests/	Behavioral and failure case tests
examples/	Reproducible demonstration artifacts
pyproject.toml	Package metadata and console entry point

Related labs

Evidence Search · Eval Forge · Prompt Vault · Drift Radar · Route Lab

License

MIT. See LICENSE.