Metadata-Version: 2.4
Name: llm-fallback
Version: 0.1.0
Summary: Automatic failover between LLM providers. When OpenAI is down, seamlessly switch to Anthropic, Google, or any backup.
Author-email: Zach <zacharie@astera.org>
License: MIT
Project-URL: Homepage, https://github.com/zachbg/llm-fallback
Keywords: llm,failover,fallback,openai,anthropic,reliability,ai,resilience
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# llm-fallback

**Automatic failover between LLM providers.** When OpenAI is down, seamlessly switch to Anthropic, Google, or any backup. Circuit breaker, retry logic, and latency-based routing built in.

## The Pain

OpenAI goes down at 2 AM and your production chatbot returns 500s for 3 hours. Your users are furious. You could have fallen back to Claude or Gemini, but your code is hardwired to one provider.

## Install

```bash
pip install llm-fallback
```

## Quick Start

```python
from llm_fallback import FallbackChain, Provider

chain = FallbackChain([
    Provider("openai", model="gpt-4o", api_key="sk-..."),
    Provider("anthropic", model="claude-3-5-sonnet-20241022", api_key="sk-ant-..."),
    Provider("openai", model="gpt-3.5-turbo", api_key="sk-..."),  # cheaper backup
])

# Tries providers in order until one succeeds
response = chain.chat("What is the capital of France?")
print(response.content)     # "The capital of France is Paris."
print(response.provider)    # "openai" (or whichever succeeded)
print(response.model)       # "gpt-4o"
print(response.latency_ms)  # 450
```

## With Messages

```python
response = chain.chat(
    messages=[
        {"role": "system", "content": "You are helpful."},
        {"role": "user", "content": "Explain quantum computing"},
    ],
    temperature=0.7,
    max_tokens=500,
)
```

## Circuit Breaker

Automatically stops trying a provider that's failing:

```python
chain = FallbackChain(
    providers=[...],
    circuit_breaker=True,       # Enable circuit breaker
    failure_threshold=3,        # Open after 3 consecutive failures
    recovery_timeout=60,        # Try again after 60 seconds
    timeout=30,                 # Per-request timeout
)
```

## Latency-Based Routing

Route to the fastest provider instead of fixed order:

```python
chain = FallbackChain(
    providers=[...],
    strategy="latency",  # "ordered" (default) or "latency"
)
# Tracks response times and prefers the fastest healthy provider
```

## Provider Configuration

```python
Provider(
    name="openai",          # Provider identifier
    model="gpt-4o",         # Model name
    api_key="sk-...",       # API key
    base_url=None,          # Custom endpoint (for proxies/self-hosted)
    timeout=30,             # Request timeout in seconds
    weight=1.0,             # Priority weight for routing
)
```

### Supported Providers

| Provider | Name string | Notes |
|---|---|---|
| OpenAI | `"openai"` | GPT-4o, GPT-3.5, etc. |
| Anthropic | `"anthropic"` | Claude 3/3.5/4 |
| Google | `"google"` | Gemini (requires `google-generativeai`) |
| OpenAI-compatible | `"openai"` with `base_url` | Ollama, vLLM, Together, Groq, etc. |

## Callbacks

```python
def on_failover(from_provider, to_provider, error):
    print(f"Failing over from {from_provider} to {to_provider}: {error}")
    slack_alert(f"LLM failover: {from_provider} -> {to_provider}")

chain = FallbackChain(
    providers=[...],
    on_failover=on_failover,
    on_success=lambda r: metrics.record(r.provider, r.latency_ms),
)
```

## Features

- **Automatic failover** — tries next provider on any error
- **Circuit breaker** — stops hammering a dead provider
- **Latency routing** — prefer the fastest healthy provider
- **Retry with backoff** — configurable retry per provider
- **Timeout enforcement** — per-request timeouts
- **Unified API** — same interface regardless of provider
- **Callbacks** — hook into failover events for alerting
- **Zero required deps** — only needs the provider SDK you're already using

## License

MIT
