Olla is a high-performance, low-overhead, low-latency proxy and load balancer for managing LLM infrastructure. It intelligently routes LLM requests across local and remote inference nodes with a wide variety of natively supported endpoints and extensible enough to support others. Olla provides model discovery and unified model catalogues within each provider, enabling seamless routing to available models on compatible endpoints.
Olla works alongside API gateways like LiteLLM or orchestration platforms like GPUStack, focusing on making your existing LLM infrastructure reliable through intelligent routing and failover. You can choose between two proxy engines: Sherpa for simplicity and maintainability or Olla for maximum performance with advanced features like circuit breakers and connection pooling.
Single CLI application and config file is all you need to go Olla!
For Large GPU deployments, Enterprise & Data-Centre use, see TensorFoundry FoundryOS.
- 🔄 Smart Load Balancing: Priority-based routing with automatic failover and connection retry
- 📌 Sticky Sessions: KV-cache-aware affinity routing that pins multi-turn conversations to the same backend
- 🔍 Smart Model Unification: Per-provider unification + OpenAI-compatible cross-provider routing
- ⚡ Dual Proxy Engines: Sherpa (simple) and Olla (high-performance)
- 🎯 Advanced Filtering: Profile and model filtering with glob patterns for precise control
- 💊 Health Monitoring: Continuous endpoint health checks with circuit breakers and automatic recovery
- 🔁 Intelligent Retry: Automatic retry on connection failures with immediate transparent endpoint failover
- 🔧 Self-Healing: Automatic model discovery refresh when endpoints recover
- 📊 Request Tracking: Detailed response headers and statistics
- ⚡🔄 Anthropic Messages API: Passthrough for backends with native support; automatic translation for others
- 🛡️ Production Ready: Rate limiting, request size limits, graceful shutdown
- ⚡ High Performance: Sub-millisecond endpoint selection with lock-free atomic stats
- 🎯 LLM-Optimised: Streaming-first design with optimised timeouts for long inference
- ⚙️ High Performance: Designed to be very lightweight & efficient, runs on less than 50Mb RAM.
Olla runs on multiple platforms and architectures:
| Platform | AMD64 | ARM64 | Notes |
|---|---|---|---|
| Linux | ✅ | ✅ | Full support including Raspberry Pi 4+ |
| macOS | ✅ | ✅ | Intel and Apple Silicon (M1/M2/M3/M4) |
| Windows | ✅ | ✅ | Windows 10/11 and Windows on ARM |
| Docker | ✅ | ✅ | Multi-architecture images (amd64/arm64) |
# Download latest release (auto-detects your platform)
bash <(curl -s https://raw.githubusercontent.com/thushan/olla/main/install.sh)# Docker (automatically pulls correct architecture)
docker run -t \
--name olla \
-p 40114:40114 \
ghcr.io/thushan/olla:latest
# Or explicitly specify platform (e.g., for ARM64)
docker run --platform linux/arm64 -t \
--name olla \
-p 40114:40114 \
ghcr.io/thushan/olla:latest# Install via Go
go install github.com/thushan/olla@latest# Build from source
git clone https://github.com/thushan/olla.git && cd olla && make build-release
# Run Olla
./bin/ollaWhen you have everything running, you can check it's all working with:
# Check health of Olla
curl http://localhost:40114/internal/health
# Check endpoints
curl http://localhost:40114/internal/status/endpoints
# Check models available
curl http://localhost:40114/internal/status/modelsFor detailed installation and deployment options, see Getting Started Guide.
Olla exposes multiple API paths depending on your use case:
| Path | Format | Use Case |
|---|---|---|
/olla/proxy/ |
OpenAI | Routes to any backend — universal endpoint |
/olla/openai/ |
OpenAI | Routes to any backend — universal endpoint |
/olla/anthropic/ |
Anthropic | Claude-compatible clients (passthrough or translated) |
/olla/{provider}/ |
OpenAI | Target a specific backend type (e.g. /olla/vllm/, /olla/ollama/) |
You can use /olla/openai or /olla/proxy
# Chat completion (routes to best available backend)
curl http://localhost:40114/olla/proxy/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'
# Streaming
curl http://localhost:40114/olla/proxy/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100, "stream": true}'
# List all models across backends
curl http://localhost:40114/olla/proxy/v1/models# Chat completion (passthrough for supported backends, translated for others)
curl http://localhost:40114/olla/anthropic/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: not-needed" \
-H "anthropic-version: 2023-06-01" \
-d '{"model": "llama3.2", "max_tokens": 100, "messages": [{"role": "user", "content": "Hello"}]}'
# Streaming
curl http://localhost:40114/olla/anthropic/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: not-needed" \
-H "anthropic-version: 2023-06-01" \
-d '{"model": "llama3.2", "max_tokens": 100, "messages": [{"role": "user", "content": "Hello"}], "stream": true}'# Target a specific backend type directly
curl http://localhost:40114/olla/ollama/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'
# Other providers: /olla/vllm/, /olla/vllm-mlx/, /olla/lm-studio/, /olla/llamacpp/, etc.We've also got ready-to-use Docker Compose setups for common scenarios:
- Home Lab: Olla → Multiple Ollama (or OpenAI Compatible - eg. vLLM) instances across your machines
- Hybrid Cloud: Olla → Local endpoints + LiteLLM → Cloud APIs (OpenAI, Anthropic, Bedrock, etc.)
- Enterprise: Olla → GPUStack cluster + vLLM servers + LiteLLM (cloud overflow)
- Development: Olla → Local + Shared team endpoints + LiteLLM (API access)
See integration patterns for detailed architectures.
For a robust enterprise setup, consider TensorFoundry FoundryOS.
Complete setup with OpenWebUI + Olla load balancing multiple Ollama instances or unify all OpenAI compatible models.
- See:
examples/ollama-openwebui/ - Services: OpenWebUI (web UI) + Olla (proxy/load balancer)
- Use Case: Web interface with intelligent load balancing across multiple Ollama servers with Olla
- Quick Start:
cd examples/ollama-openwebui # Edit olla.yaml to configure your Ollama endpoints docker compose up -d # Access OpenWebUI at http://localhost:3000
You can learn more about OpenWebUI Ollama with Olla or see OpenWebUI OpenAI with Olla.
Olla's Anthropic Messages API support (v0.0.20+) is enabled by default, allowing you to use CLI tools like Claude Code with local AI models on your machine via /olla/anthropic. It operates in two modes depending on your backend:
- ⚡ Passthrough: requests are forwarded as-is for backends with native Anthropic support (vLLM, llama.cpp, Ollama, LM Studio, Lemonade)
- 🔄 Translation: Anthropic ↔ OpenAI format conversion for backends that don't natively support the Anthropic Messages API
Still actively being improved -- please report any issues or feedback.
We have examples for:
Learn more about Anthropic API Translation.
Full documentation is available at https://thushan.github.io/olla/
- Getting Started - Getting Started with Olla
- Integrations - See which LLM backends are supported by Olla
- Comparisons - Compare with LiteLLM, GPUStack, LocalAI
- Olla Concepts - Understand Key Olla concepts
- Configuration - Extensive configuration documentation
- API Reference - Olla System API Reference
- Development - Contributing and development guide
We welcome contributions! Please open an issue first to discuss major changes.
This project has been built with the assistance of AI tools for documentation, test refinement, and code reviews.
We've utilised GitHub Copilot, Anthropic Claude, JetBrains Junie, Codex & TensorFoundry Kaizen for documentation, code reviews, test refinement and troubleshooting.
We also utilise CodeRabbit for AI-driven code reviews for PRs prior to human review.
- @pterm/pterm - Terminal UI framework
- @puzpuzpuz/xsync - High-performance concurrent maps
- @golangci/golangci-lint - Go linting
- @dkorunic/betteralign - Struct alignment optimisation
Licensed under the Apache License 2.0. See LICENSE for details.
Made with ❤️ for the LLM community
![@dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=64&v=4){kind=small}
