Metadata-Version: 2.4
Name: promptbench-cli
Version: 0.1.0
Summary: A/B test prompts across LLM providers from your terminal
Author-email: Shehriyar Ali Rustam <shehriyar@example.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Shehriyar-Ali-Rustam/promptbench
Project-URL: Repository, https://github.com/Shehriyar-Ali-Rustam/promptbench
Project-URL: Issues, https://github.com/Shehriyar-Ali-Rustam/promptbench/issues
Keywords: llm,prompt,benchmark,openai,anthropic,gemini,cli,ai,testing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.30; extra == "anthropic"
Provides-Extra: google
Requires-Dist: google-genai>=1.0; extra == "google"
Provides-Extra: all
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: anthropic>=0.30; extra == "all"
Requires-Dist: google-genai>=1.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Dynamic: license-file

# ⚡ PromptBench

**A/B test prompts across LLM providers from your terminal.**

[![CI](https://github.com/Shehriyar-Ali-Rustam/promptbench/actions/workflows/ci.yml/badge.svg)](https://github.com/Shehriyar-Ali-Rustam/promptbench/actions/workflows/ci.yml)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![PyPI](https://img.shields.io/pypi/v/promptbench-cli.svg)](https://pypi.org/project/promptbench-cli/)

Compare how different LLMs respond to the same prompt — see latency, token usage, cost, and responses side by side in one command.

```
$ promptbench "Explain quantum computing in one sentence"

  ⚡ PROMPTBENCH RESULTS
  ────────────────────────────────────────────────────────────

  Prompt: Explain quantum computing in one sentence

  Model                               Latency     Tokens         Cost
  ─────────────────────────────────── ────────── ────────── ────────────
  claude-sonnet-4-20250514                 1.24s        142     $0.0006
  gemini-2.0-flash                    ⚡💰 312ms         98     $0.0000
  gpt-4o                                  845ms        127     $0.0010

  ┌─ claude-sonnet-4-20250514
  │ Quantum computing uses qubits that can exist in superpositions of
  │ 0 and 1 simultaneously, enabling parallel computation that can
  │ solve certain problems exponentially faster than classical computers.
  └───────────────────────────────────────────────────────────

  ┌─ gemini-2.0-flash
  │ Quantum computing harnesses quantum mechanical phenomena like
  │ superposition and entanglement to process information in ways
  │ impossible for traditional computers.
  └───────────────────────────────────────────────────────────

  ┌─ gpt-4o
  │ Quantum computing leverages the principles of quantum mechanics —
  │ superposition and entanglement — to perform computations that
  │ would be infeasible for classical computers.
  └───────────────────────────────────────────────────────────

  Total cost: $0.0016 · Avg latency: 799ms · 3 model(s)
```

## Why PromptBench?

- **One command, multiple models** — no switching between playgrounds
- **Side-by-side comparison** — latency, tokens, cost, and full responses
- **Supports major providers** — OpenAI, Anthropic, Google Gemini
- **Multiple output formats** — terminal table, JSON, CSV, Markdown
- **Fast parallel execution** — all models run concurrently by default
- **Batch mode** — test multiple prompts from a file
- **Zero required dependencies** — install only the providers you need
- **Pipe-friendly** — works with stdin for scripting workflows

## Install

```bash
pip install promptbench-cli
```

Install with provider SDKs:

```bash
# Individual providers
pip install "promptbench-cli[openai]"
pip install "promptbench-cli[anthropic]"
pip install "promptbench-cli[google]"

# All providers at once
pip install "promptbench-cli[all]"
```

## Setup

Export your API keys:

```bash
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="AI..."
```

You only need keys for the providers you want to use. PromptBench will warn you if a key is missing.

## Usage

### Basic comparison (default: GPT-4o, Claude Sonnet, Gemini Flash)

```bash
promptbench "Explain quantum computing in one sentence"
```

### Pick specific models

```bash
promptbench "Write a haiku about coding" -m gpt4o sonnet flash
```

### With a system prompt

```bash
promptbench "Summarize this text" -s "You are a concise technical writer" -m gpt4mini haiku
```

### Batch prompts from a file

```bash
# prompts.txt — one prompt per line
promptbench -f prompts.txt -m gpt4o sonnet
```

### JSON output

```bash
promptbench "What is Python?" -o json
```

### CSV output

```bash
promptbench "What is Python?" -o csv > results.csv
```

### Save results to file

```bash
promptbench "Compare REST vs GraphQL" --save results.json
```

### Pipe from stdin

```bash
echo "What is the meaning of life?" | promptbench -m gpt4o sonnet
```

### List all supported models

```bash
promptbench --list-models
```

## Supported Models

| Alias | Model | Provider |
|-------|-------|----------|
| `gpt4o` / `gpt4` | gpt-4o | OpenAI |
| `gpt4mini` | gpt-4o-mini | OpenAI |
| `gpt3.5` | gpt-3.5-turbo | OpenAI |
| `sonnet` / `claude-sonnet` | claude-sonnet-4-20250514 | Anthropic |
| `haiku` / `claude-haiku` | claude-haiku-4-5-20251001 | Anthropic |
| `flash` / `gemini-flash` | gemini-2.0-flash | Google |
| `gemini-pro` | gemini-1.5-pro | Google |

You can also use full model names directly (e.g., `gpt-4-turbo`, `gemini-1.5-flash`).

## Python Library Usage

```python
from promptbench.runner import run_bench
from promptbench.display import display_comparison

run = run_bench(
    prompt="Explain recursion simply",
    models=["gpt4o", "sonnet", "flash"],
    temperature=0.5,
)

print(display_comparison(run))

# Access individual results
for result in run.results:
    print(f"{result.model}: {result.latency_ms:.0f}ms, {result.cost_usd:.6f} USD")
```

## Configuration Flags

| Flag | Description | Default |
|------|-------------|---------|
| `-m, --models` | Models to test | `gpt4o sonnet flash` |
| `-s, --system` | System prompt | None |
| `-t, --temperature` | Sampling temperature | `0.7` |
| `--max-tokens` | Max output tokens | `1024` |
| `-f, --file` | Prompts file path | None |
| `-o, --output` | Output format: table, json, csv, markdown | `table` |
| `--full` | Show full responses (no truncation) | Off |
| `--no-parallel` | Run models sequentially | Off |
| `--save` | Save results to JSON file | None |
| `--list-models` | List supported models | — |
| `--version` | Show version | — |

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, how to add providers, and PR guidelines.

## License

MIT — see [LICENSE](LICENSE) for details.
