Metadata-Version: 2.4
Name: llm-route
Version: 0.7.0
Summary: Intelligent LLM routing with weighted least-outstanding selection, 429 cooldown, and request class isolation
License-Expression: MIT
Keywords: azure,openai,anthropic,claude,routing,load-balancing,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai==2.30.0
Requires-Dist: pydantic==2.12.5
Requires-Dist: pydantic-settings==2.14.2
Provides-Extra: anthropic
Requires-Dist: anthropic<1.0.0,>=0.49.0; extra == "anthropic"
Provides-Extra: redis
Requires-Dist: redis==7.4.0; extra == "redis"
Provides-Extra: dev
Requires-Dist: pre-commit==4.5.1; extra == "dev"
Requires-Dist: pytest==9.0.3; extra == "dev"
Requires-Dist: pytest-asyncio==1.3.0; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"
Requires-Dist: respx==0.22.0; extra == "dev"
Requires-Dist: ruff==0.15.9; extra == "dev"
Requires-Dist: mypy==1.20.0; extra == "dev"
Dynamic: license-file

# llm-route

Intelligent LLM routing library for Python. Drop-in replacement for direct Azure OpenAI calls with automatic load balancing, 429 failover, and request class isolation.

Built as a minimal-dependency alternative to LiteLLM Router, focused on security and transparency.

## Features

- **Weighted least-outstanding routing** — routes to the deployment with the lowest load relative to its capacity
- **429-aware cooldown** — respects `Retry-After` headers, automatically fails over to the next backend
- **Request class isolation** — separate concurrency budgets for light/medium/heavy requests prevent expensive operations from starving fast ones
- **Token-aware capacity tracking** — tracks actual token usage per deployment per minute window
- **Request deadline** — enforces a total timeout across all retry attempts
- **Health reporting** — exposes deployment health, inflight counts, 429 rates, and TPM usage

## Install

```bash
pip install llm-route
```

Or with uv:

```bash
uv add llm-route
```

## Quick Start

```python
import asyncio
from llm_route import SmartRouter, RequestClass, RouterConfig
from llm_route.config import DeploymentConfig

config = RouterConfig(
    deployments=[
        DeploymentConfig(
            name="eastus-1",
            endpoint="https://my-resource.openai.azure.com/",
            api_key="your-key",
            deployment_name="gpt-4.1",
            tpm_quota=120_000,
        ),
    ],
)

router = SmartRouter(config=config)

async def main():
    response = await router.complete(
        messages=[{"role": "user", "content": "Hello"}],
        request_class=RequestClass.LIGHT,
    )
    print(response.choices[0].message.content)

asyncio.run(main())
```

## Configuration

### JSON config file

```json
{
  "deployments": [
    {
      "name": "eastus-1",
      "endpoint": "https://my-eastus-1.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "gpt-4.1",
      "tpm_quota": 120000
    },
    {
      "name": "eastus-2",
      "endpoint": "https://my-eastus-2.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "gpt-4.1",
      "tpm_quota": 60000
    }
  ],
  "default_timeout": 60.0,
  "max_retries": 3,
  "cooldown_seconds": 10.0,
  "concurrency": {
    "light": 20,
    "medium": 10,
    "heavy": 3
  }
}
```

Load it:

```python
config = RouterConfig.from_file("config.json")
router = SmartRouter(config=config)
```

### Environment variables

All settings can be set via env vars with `LLM_ROUTE_` prefix:

```bash
LLM_ROUTE_DEFAULT_TIMEOUT=60.0
LLM_ROUTE_MAX_RETRIES=3
LLM_ROUTE_COOLDOWN_SECONDS=10.0
```

## Model compatibility (2026-08)

Providers reject different parameters per model family. The router adapts requests
automatically and reports what it changed.

**Azure/OpenAI reasoning models** — deployment names starting with `o1`, `o3`, `o4`,
or `gpt-5` (case-insensitive) are auto-detected as reasoning models: `max_tokens` is
translated to `max_completion_tokens`, and `temperature` / `top_p` /
`presence_penalty` / `frequency_penalty` are stripped. `gpt-4o`, `gpt-4.1`, and the
chat-tuned `gpt-5-chat` / `gpt-5-chat-latest` are *not* reasoning models. Set
`reasoning_model: true` or `false` on the deployment to override the auto-detection —
needed when the Azure deployment name doesn't match the underlying model name (e.g. a
deployment called `prod-fast` serving `gpt-5-mini`).

**Claude 5-family sampling params** — `claude-opus-4-7`, `claude-opus-4-8`,
`claude-opus-5`, `claude-sonnet-5`, `claude-fable-5`, and `claude-mythos` models
return HTTP 400 for any explicit `temperature` / `top_p` / `top_k`, so the router
strips them. Set `supports_sampling: true` or `false` on the Anthropic deployment to
override.

**The two Anthropic flags are different axes** — on an Anthropic deployment,
`supports_sampling` governs REQUEST BEHAVIOUR (whether `temperature` / `top_p` /
`top_k` are sent, stripped, or raise), while `reasoning_model` is REPORTING ONLY (it
sets `is_reasoning`, which drives `reasoning_deployments` in the health report and the
startup family checks) and never changes what is sent to the API. Set both if a
deployment needs both.

**Seeing what was changed** — stripped parameters are reported on the router result
(`sanitized_params`) and logged. Set `strict_param_validation: true` in the router
config to raise `ReasoningModelParamError` instead of silently sanitizing: the
Anthropic provider raises for ANY parameter it would otherwise strip or drop —
sampling params on a rejecting model, params with no Messages API equivalent
(`response_format`, `seed`, `logprobs`, `top_logprobs`, `n`, `presence_penalty`,
`frequency_penalty`), and untranslated params (`tools`, `tool_choice`, `thinking`,
...). The Azure provider raises for reasoning-model sampling params.

**Claude refusals** — a safety-classifier refusal returns HTTP 200 with empty content
and `finish_reason == "content_filter"`. Callers that assume a successful response has
text should check `finish_reason` before using `choices[0].message.content`.

**`finish_reason == "length"` has two causes** — the `max_tokens` output cap
(`stop_reason: "max_tokens"`) and context-window exhaustion
(`stop_reason: "model_context_window_exceeded"`, logged as
`anthropic_context_window_exceeded`). Don't blindly retry with a larger `max_tokens`:
that fixes the first cause and makes the second worse. Shorten the input instead.

## Request Classes

Request classes provide concurrency isolation per deployment. Heavy requests (full document review) won't starve light ones (quick text search).

| Class | Default concurrency / deployment | Use case |
|-------|----------------------------------|----------|
| `LIGHT` | 20 | Short extraction, find-text, quick Q&A |
| `MEDIUM` | 10 | Clause analysis, section review |
| `HEAVY` | 3 | Full document review, redline, long synthesis |

```python
await router.complete(
    messages=[...],
    request_class=RequestClass.HEAVY,  # uses the heavy concurrency budget
)
```

## How Routing Works

1. **Filter** — remove disabled, cooled-down, and already-tried deployments
2. **Filter** — remove deployments with no available concurrency for the request class
3. **Score** — `inflight / weight` where `weight = tpm_quota / min_tpm`. Lower is better.
4. **Select** — pick lowest score; break ties by remaining TPM headroom
5. **Execute** — acquire semaphore, call Azure OpenAI with remaining deadline
6. **Failover** — on 429 or 5xx, mark cooldown, try next deployment
7. **Deadline** — if total timeout expires, raise `RouterExhaustedError`

## Health Monitoring

```python
health = router.health()
for dep in health.deployments:
    print(f"{dep.name}: healthy={dep.healthy}, inflight={dep.inflight}, "
          f"tpm={dep.tpm_used}/{dep.tpm_quota}, 429s={dep.total_429s}")
```

Expose as a FastAPI endpoint:

```python
@app.get("/health/llm")
async def llm_health():
    return router.health().model_dump()
```

## Dependencies

Minimal by design:

- `openai` — Azure OpenAI SDK
- `pydantic` — data validation
- `pydantic-settings` — configuration management

Optional:

- `redis` — reserved for planned shared-state support (`pip install llm-route[redis]`). **Not yet implemented**: cooldown/TPM/concurrency state is currently per-process (the `redis_url` config field is accepted but unused). Multi-replica or multi-app deployments sharing the same backends will each track 429s independently until this lands.

## Security

This library was built in response to the [LiteLLM supply chain attack](https://github.com/BerriAI/litellm/issues/24518) (March 2026). Design principles:

- **Minimal dependencies** — 3 required packages, all well-maintained
- **No build-time code execution** — pure Python, no compiled extensions
- **`uv.lock` committed** — exact dependency tree is auditable
- **OIDC publishing** — no stored PyPI tokens in CI

## License

MIT
