---
name: olla
description: Configure and manage Olla LLM proxy gateway — load balancing, model routing, API translation (Anthropic<>OpenAI), health checking, and multi-backend orchestration for local LLM inference. Use when setting up Olla proxy, configuring backends (Ollama, vLLM, LM Studio, llama.cpp, SGLang, LiteLLM), debugging routing issues, or connecting Claude Code to local models via Olla.
allowed-tools: Bash(*), Read(*), Write(*), Edit(*)
---

# Olla — LLM Proxy Gateway

Olla is a high-performance LLM proxy gateway written in Go that unifies multiple local LLM backends behind a single endpoint with intelligent routing, load balancing, health checking, and API translation.

**Repository**: https://github.com/thushan/olla
**Docs**: https://thushan.github.io/olla/
**Default Port**: 40114 (mnemonic: "4 OLLA")

## Installation

```bash
# Script install (Linux/macOS)
bash <(curl -s https://raw.githubusercontent.com/thushan/olla/main/install.sh)

# Go install
go install github.com/thushan/olla@latest

# Docker
docker pull ghcr.io/thushan/olla:latest
docker run -t --name olla -p 40114:40114 ghcr.io/thushan/olla:latest

# Build from source
git clone https://github.com/thushan/olla.git && cd olla && make build-release
# Binary: ./bin/olla
```

Verify: `olla --version` | Health: `curl http://localhost:40114/internal/health`

## Core Concepts

### Proxy Engines
- **Sherpa** (default): Simple, shared HTTP transport. Good for dev, <100 concurrent users, lower memory.
- **Olla**: Per-endpoint connection pools, advanced circuit breaker. Production, high throughput, streaming-heavy.

### Load Balancing Strategies
- **priority** (default): Routes to highest-priority healthy endpoint. Best for primary/backup hierarchies.
- **round-robin**: Even distribution across healthy endpoints. Best for homogeneous servers.
- **least-connections**: Routes to endpoint with fewest active connections. Best for variable request durations/streaming.

Health-aware weighting: healthy=1.0, busy=0.3, warming=0.1, unhealthy/unknown=0.

### Model Routing
- **strict** (default): Only route to endpoints known to have the model. 404 if not found.
- **optimistic**: Try any healthy endpoint if model not found. Prioritizes availability.
- **discovery**: Refresh model catalog before routing. Adds latency but ensures freshness.

Fallback behavior: `compatible_only` (reject if not found), `all` (any healthy), `none` (always reject).

### Model Aliases
Map a virtual model name to different actual names across backends:
```yaml
model_aliases:
  my-llama:
    - "llama3.1:8b"                    # Ollama
    - llama-3.1-8b-instruct            # LM Studio
    - Meta-Llama-3.1-8B-Instruct.gguf  # llama.cpp
```

### API Translation (Anthropic <> OpenAI)
Three-stage: Request translation -> Backend processing -> Response translation.
Supports streaming SSE, tool use, vision. Overhead: 1-5ms (translation), ~0ms (passthrough).

**Passthrough mode**: When backend natively supports Anthropic (vLLM v0.11.1+, llama.cpp b4847+, LM Studio v0.4.1+, Ollama v0.14.0+), Olla bypasses translation entirely. Header: `X-Olla-Mode: passthrough`.

### Health Checking & Circuit Breaker
- Continuous monitoring with configurable intervals (default: 5s)
- Exponential backoff on failures (2x, 4x, 8x... capped at 60s)
- Circuit breaker: Closed -> Open (3 failures) -> Half-Open (30s, test traffic) -> Closed
- Auto model discovery on recovery
- States: Healthy, Busy, Warming, Offline, Unhealthy

## Configuration

Config search order: `--config` flag > `OLLA_CONFIG_FILE` env > `config/config.local.yaml` > `config/config.yaml` > `config.yaml` > `default.yaml`

Best practice: Create `config/config.local.yaml` with only overrides.

### Minimal Config Example
```yaml
server:
  host: "localhost"
  port: 40114

proxy:
  engine: "sherpa"
  load_balancer: "priority"

discovery:
  type: "static"
  static:
    endpoints:
      - url: "http://localhost:11434"
        name: "local-ollama"
        type: "ollama"
        priority: 100
```

### Full Config Reference

#### Server
```yaml
server:
  host: "localhost"              # Bind address
  port: 40114                    # Listen port
  read_timeout: 30s
  write_timeout: 0s              # MUST be 0s for streaming
  shutdown_timeout: 10s
  idle_timeout: 0s
  request_logging: true
  request_limits:
    max_body_size: 104857600     # 100MB
    max_header_size: 1048576     # 1MB
  rate_limits:
    global_requests_per_minute: 1000
    per_ip_requests_per_minute: 100
    health_requests_per_minute: 1000
    burst_size: 50
    cleanup_interval: 5m
    trust_proxy_headers: false
    trusted_proxy_cidrs: ["127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
```

#### Proxy
```yaml
proxy:
  engine: "sherpa"               # sherpa | olla
  profile: "auto"                # auto | streaming | standard
  load_balancer: "priority"      # priority | round-robin | least-connections
  connection_timeout: 30s
  response_timeout: 600s         # 10 min default
  read_timeout: 120s
  stream_buffer_size: 8192       # 8KB (sherpa), 65536 (olla recommended)
  profile_filter:
    include: []                  # glob patterns
    exclude: []
```

#### Discovery & Endpoints
```yaml
discovery:
  type: "static"
  refresh_interval: 30s
  static:
    endpoints:
      - url: "http://localhost:11434"
        name: "local-ollama"
        type: "ollama"           # ollama|lm-studio|vllm|sglang|llamacpp|lemonade|litellm|openai|docker-model-runner
        priority: 100            # 0-100, higher = preferred
        preserve_path: false
        health_check_url: ""     # auto-detected per type
        model_url: ""            # auto-detected per type
        check_interval: 5s
        check_timeout: 2s
  model_discovery:
    enabled: true
    interval: 5m
    timeout: 30s
    concurrent_workers: 5
    retry_attempts: 3
    retry_backoff: 1s
```

#### Model Registry & Routing
```yaml
model_registry:
  type: "memory"
  enable_unifier: true
  routing_strategy:
    type: "strict"               # strict | optimistic | discovery
    options:
      fallback_behavior: "compatible_only"  # compatible_only | all | none
      discovery_timeout: 2s
      discovery_refresh_on_miss: false
  unification:
    enabled: true
    stale_threshold: 24h
    cleanup_interval: 5m
    cache_ttl: 10m

model_aliases:
  alias-name:
    - "backend1-model-name"
    - "backend2-model-name"
```

#### Translators
```yaml
translators:
  anthropic:
    enabled: true
    passthrough_enabled: true
    max_message_size: 10485760   # 10MB
    inspector:
      enabled: false
      output_dir: "logs/inspector/anthropic"
      session_header: "X-Session-ID"
```

#### Logging & Engineering
```yaml
logging:
  level: "info"                  # debug | info | warn | error
  format: "json"                 # json | text
  output: "stdout"               # stdout | file

engineering:
  show_nerdstats: false          # Memory/GC stats on shutdown
```

### Environment Variables
Pattern: `OLLA_<SECTION>_<KEY>` (uppercase, underscores)
```bash
OLLA_SERVER_PORT=8080
OLLA_PROXY_ENGINE=olla
OLLA_LOG_LEVEL=debug
OLLA_CONFIG_FILE=/path/to/config.yaml
```

## Claude Code Integration

Connect Claude Code to local models through Olla's Anthropic API translation:

```bash
export ANTHROPIC_BASE_URL="http://localhost:40114/olla/anthropic"
export ANTHROPIC_API_KEY="not-really-needed"
export ANTHROPIC_MODEL="openai/gpt-oss-120b"  # or your model name
export ANTHROPIC_SMALL_FAST_MODEL="${ANTHROPIC_MODEL}"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="${ANTHROPIC_MODEL}"
export ANTHROPIC_DEFAULT_SONNET_MODEL="${ANTHROPIC_MODEL}"
export ANTHROPIC_DEFAULT_OPUS_MODEL="${ANTHROPIC_MODEL}"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export API_TIMEOUT_MS=3000000
```

**Recommended models for Claude Code**: qwen2.5-coder:32b, deepseek-coder-v2, codellama:34b, llama3.3, qwen3:32b

### Docker Compose Quick Setup
```yaml
# compose.yaml
services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports: ["11434:11434"]
    volumes: ["ollama_data:/root/.ollama"]

  olla:
    image: ghcr.io/thushan/olla:latest
    container_name: olla
    ports: ["40114:40114"]
    volumes: ["./olla.yaml:/app/config/config.local.yaml"]
    depends_on: [ollama]

volumes:
  ollama_data:
```

For Docker: set `server.host: "0.0.0.0"` to bind all interfaces.

## API Endpoints

### System
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/internal/health` | GET | Health verification |
| `/internal/status` | GET | System metrics |
| `/internal/status/endpoints` | GET | Backend availability |
| `/internal/status/models` | GET | Model registry |
| `/internal/stats/models` | GET | Usage by model |
| `/internal/stats/translators` | GET | Translator performance |
| `/internal/process` | GET | Runtime info |

### Unified Models
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/olla/models` | GET | All models across all providers |

### Provider-Specific Routes
| Provider | Route Prefix |
|----------|-------------|
| Ollama | `/olla/ollama/*` |
| LM Studio | `/olla/lmstudio/*` |
| OpenAI | `/olla/openai/*` |
| vLLM | `/olla/vllm/*` |
| SGLang | `/olla/sglang/*` |
| LiteLLM | `/olla/litellm/*` |
| llama.cpp | `/olla/llamacpp/*` |
| Lemonade | `/olla/lemonade/*` |

### Anthropic Translation
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/olla/anthropic/v1/messages` | POST | Chat (streaming supported) |
| `/olla/anthropic/v1/models` | GET | List models |
| `/olla/anthropic/v1/messages/count_tokens` | POST | Token estimation |

### Response Headers
- `X-Olla-Request-ID` — Unique request ID
- `X-Olla-Endpoint` — Selected backend
- `X-Olla-Model` — Model identifier
- `X-Olla-Backend-Type` — Provider type
- `X-Olla-Response-Time` — Processing duration
- `X-Olla-Routing-Strategy` — Active strategy
- `X-Olla-Routing-Decision` — routed/fallback/rejected
- `X-Olla-Mode` — "passthrough" when native Anthropic format

## Troubleshooting

```bash
# Health check
curl http://localhost:40114/internal/health

# List available models
curl http://localhost:40114/olla/anthropic/v1/models | jq

# Check backend health
curl http://localhost:40114/internal/status/endpoints | jq

# Test streaming
curl -N -X POST http://localhost:40114/olla/anthropic/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"model":"llama4:latest","max_tokens":50,"messages":[{"role":"user","content":"Count to 5"}],"stream":true}'

# Translator stats
curl http://localhost:40114/internal/stats/translators | jq

# Debug logging
OLLA_LOG_LEVEL=debug olla --config config.yaml
```

### Common Issues
- **Port conflict**: Set `OLLA_SERVER_PORT` or change config
- **Streaming broken**: Ensure `server.write_timeout: 0s`
- **Docker can't connect**: Set `server.host: "0.0.0.0"`
- **Model not found**: Check routing strategy (try `optimistic`), verify with `/olla/models`
- **Slow responses**: Switch to `olla` engine, increase `stream_buffer_size`

## Supported Backends

| Backend | Type Key | Notes |
|---------|----------|-------|
| Ollama | `ollama` | Most common local setup |
| LM Studio | `lm-studio` | GUI-based, model unification |
| vLLM | `vllm` | High-perf, production grade |
| vLLM-MLX | `vllm-mlx` | Apple Silicon via MLX |
| SGLang | `sglang` | RadixAttention, vision |
| llama.cpp | `llamacpp` | GGUF, CPU-first, edge |
| Lemonade SDK | `lemonade` | AMD Ryzen AI |
| LiteLLM | `litellm` | 100+ cloud providers |
| Docker Model Runner | `docker-model-runner` | OCI model distribution |
| OpenAI-compatible | `openai` | Generic fallback |

## Development

```bash
git clone https://github.com/thushan/olla.git && cd olla
make deps      # Install dependencies
make dev       # Build with hot-reload
make test      # Run tests
make ready     # Pre-commit checks (fmt + lint + test)
make bench     # Benchmarks
```

Architecture: Hexagonal (ports & adapters) — `internal/core/` (domain), `internal/adapter/` (infra), `internal/app/` (HTTP handlers).
