Metadata-Version: 2.4
Name: llm-route
Version: 0.6.2
Summary: Intelligent LLM routing with weighted least-outstanding selection, 429 cooldown, and request class isolation
License-Expression: MIT
Keywords: azure,openai,anthropic,claude,routing,load-balancing,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai==2.30.0
Requires-Dist: pydantic==2.12.5
Requires-Dist: pydantic-settings==2.13.1
Provides-Extra: anthropic
Requires-Dist: anthropic<1.0.0,>=0.49.0; extra == "anthropic"
Provides-Extra: redis
Requires-Dist: redis==7.4.0; extra == "redis"
Provides-Extra: dev
Requires-Dist: pre-commit==4.5.1; extra == "dev"
Requires-Dist: pytest==9.0.2; extra == "dev"
Requires-Dist: pytest-asyncio==1.3.0; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"
Requires-Dist: respx==0.22.0; extra == "dev"
Requires-Dist: ruff==0.15.9; extra == "dev"
Requires-Dist: mypy==1.20.0; extra == "dev"
Dynamic: license-file

# llm-route

Intelligent LLM routing library for Python. Drop-in replacement for direct Azure OpenAI calls with automatic load balancing, 429 failover, and request class isolation.

Built as a minimal-dependency alternative to LiteLLM Router, focused on security and transparency.

## Features

- **Weighted least-outstanding routing** — routes to the deployment with the lowest load relative to its capacity
- **429-aware cooldown** — respects `Retry-After` headers, automatically fails over to the next backend
- **Request class isolation** — separate concurrency budgets for light/medium/heavy requests prevent expensive operations from starving fast ones
- **Token-aware capacity tracking** — tracks actual token usage per deployment per minute window
- **Request deadline** — enforces a total timeout across all retry attempts
- **Health reporting** — exposes deployment health, inflight counts, 429 rates, and TPM usage

## Install

```bash
pip install llm-route
```

Or with uv:

```bash
uv add llm-route
```

## Quick Start

```python
import asyncio
from llm_route import SmartRouter, RequestClass, RouterConfig
from llm_route.config import DeploymentConfig

config = RouterConfig(
    deployments=[
        DeploymentConfig(
            name="eastus-1",
            endpoint="https://my-resource.openai.azure.com/",
            api_key="your-key",
            deployment_name="gpt-4o",
            tpm_quota=120_000,
        ),
    ],
)

router = SmartRouter(config=config)

async def main():
    response = await router.complete(
        messages=[{"role": "user", "content": "Hello"}],
        request_class=RequestClass.LIGHT,
    )
    print(response.choices[0].message.content)

asyncio.run(main())
```

## Configuration

### JSON config file

```json
{
  "deployments": [
    {
      "name": "eastus-1",
      "endpoint": "https://my-eastus-1.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "gpt-4o",
      "tpm_quota": 120000
    },
    {
      "name": "eastus-2",
      "endpoint": "https://my-eastus-2.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "gpt-4o",
      "tpm_quota": 60000
    }
  ],
  "default_timeout": 60.0,
  "max_retries": 3,
  "cooldown_seconds": 10.0,
  "concurrency": {
    "light": 20,
    "medium": 10,
    "heavy": 3
  }
}
```

Load it:

```python
config = RouterConfig.from_file("config.json")
router = SmartRouter(config=config)
```

### Environment variables

All settings can be set via env vars with `LLM_ROUTE_` prefix:

```bash
LLM_ROUTE_DEFAULT_TIMEOUT=60.0
LLM_ROUTE_MAX_RETRIES=3
LLM_ROUTE_COOLDOWN_SECONDS=10.0
```

## Request Classes

Request classes provide concurrency isolation per deployment. Heavy requests (full document review) won't starve light ones (quick text search).

| Class | Default concurrency / deployment | Use case |
|-------|----------------------------------|----------|
| `LIGHT` | 20 | Short extraction, find-text, quick Q&A |
| `MEDIUM` | 10 | Clause analysis, section review |
| `HEAVY` | 3 | Full document review, redline, long synthesis |

```python
await router.complete(
    messages=[...],
    request_class=RequestClass.HEAVY,  # uses the heavy concurrency budget
)
```

## How Routing Works

1. **Filter** — remove disabled, cooled-down, and already-tried deployments
2. **Filter** — remove deployments with no available concurrency for the request class
3. **Score** — `inflight / weight` where `weight = tpm_quota / min_tpm`. Lower is better.
4. **Select** — pick lowest score; break ties by remaining TPM headroom
5. **Execute** — acquire semaphore, call Azure OpenAI with remaining deadline
6. **Failover** — on 429 or 5xx, mark cooldown, try next deployment
7. **Deadline** — if total timeout expires, raise `RouterExhaustedError`

## Health Monitoring

```python
health = router.health()
for dep in health.deployments:
    print(f"{dep.name}: healthy={dep.healthy}, inflight={dep.inflight}, "
          f"tpm={dep.tpm_used}/{dep.tpm_quota}, 429s={dep.total_429s}")
```

Expose as a FastAPI endpoint:

```python
@app.get("/health/llm")
async def llm_health():
    return router.health().model_dump()
```

## Dependencies

Minimal by design:

- `openai` — Azure OpenAI SDK
- `pydantic` — data validation
- `pydantic-settings` — configuration management

Optional:

- `redis` — shared state for multi-replica deployments (`pip install llm-route[redis]`)

## Security

This library was built in response to the [LiteLLM supply chain attack](https://github.com/BerriAI/litellm/issues/24518) (March 2026). Design principles:

- **Minimal dependencies** — 3 required packages, all well-maintained
- **No build-time code execution** — pure Python, no compiled extensions
- **`uv.lock` committed** — exact dependency tree is auditable
- **OIDC publishing** — no stored PyPI tokens in CI

## License

MIT
