Metadata-Version: 2.4
Name: llm-arbiter
Version: 0.1.1
Summary: A/B testing for LLMs. Statistical proof, not vibes.
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: openai>=1.0

# Verdict

**A/B testing for LLMs. Statistical proof, not vibes.**

Verdict is a Python framework for systematically comparing large language models with statistical rigor. Instead of picking models based on gut feeling, verdict helps you make data-driven decisions using rigorous experimental design and statistical analysis.

## Features

- 🧪 **Parallel A/B Testing** — Run multiple model variants concurrently on the same queries
- 📊 **Statistical Analysis** — Built-in significance testing using scipy
- 💰 **Cost Tracking** — Monitor and compare inference costs across models
- 🎯 **LLM Judge Scoring** — Automatic evaluation using GPT-4o-mini with custom criteria
- ⚡ **Performance Metrics** — Track latency, tokens, and costs
- 📈 **Variant Comparison** — Side-by-side results with statistical significance

## Installation

```bash
pip install llm-arbiter
```

## Quick Start

```python
from verdict import Experiment, Variant, Judge

# Define model variants to test
variants = [
    Variant(
        name="gpt-4o",
        fn=call_model,
        config={"model": "gpt-4o"},
        model="gpt-4o"
    ),
    Variant(
        name="gpt-4o-mini",
        fn=call_model,
        config={"model": "gpt-4o-mini"},
        model="gpt-4o-mini"
    ),
]

# Create an experiment
experiment = Experiment(
    queries=["What is X?", "Explain Y"],
    variants=variants,
)

# Run the experiment
results = experiment.run()

# Analyze results with optional LLM judge
judge = Judge(criteria=[
    "Does it cite specific numbers?",
    "Is the analysis correct?",
])
analysis = experiment.analyze(judge=judge)
print(analysis.summary())
```

## Core Components

### Variant
Represents a single model configuration being tested.

```python
Variant(
    name="gpt-4o",
    fn=call_model,  # (config, query) → str or dict
    config={"model": "gpt-4o"},  # Passed to fn
    model="gpt-4o",  # For pricing lookup
    pricing={"input": 2.50/1_000_000, "output": 10.00/1_000_000}  # Optional override
)
```

### Experiment
Defines and runs the test across all variants.

- **queries**: List of queries to test
- **variants**: List of Variant objects
- **num_workers**: Parallel execution threads (default: auto)

### Judge
Scores model outputs using GPT-4o-mini based on custom criteria.

```python
judge = Judge(
    criteria=["Is it accurate?", "Does it cite sources?"],
    model="gpt-4o-mini"  # Can override
)
```

### Analysis
Statistical results after running the experiment.

- **RunResult**: Individual execution data (latency, tokens, cost, score)
- **summary()**: Display results with significance testing

## Supported Models & Pricing

Pre-configured pricing for OpenAI models:
- `gpt-4o`
- `gpt-4o-mini`
- `gpt-4.1`
- `gpt-4.1-mini`
- `gpt-4.1-nano`
- `o3-mini`

Custom pricing can be specified per variant.

## Example: Market Research Analysis

See `test.py` for a complete example comparing models on market research questions.

## Requirements

- Python 3.9+
- numpy >= 1.24
- scipy >= 1.10
- openai >= 1.0

## Environment Variables

Set your OpenAI API key:
```bash
export OPENAI_API_KEY="sk-..."
```

## License

MIT

## Contributing

Contributions welcome! This project uses statistical rigor—pull requests should include validation.
