Metadata-Version: 2.4
Name: llm-regtest
Version: 0.1.0
Summary: Lightweight regression testing for LLM prompts
Author: Libin Samkutty
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.18; extra == "anthropic"
Provides-Extra: yaml
Requires-Dist: pyyaml>=6.0; extra == "yaml"
Provides-Extra: semantic
Requires-Dist: sentence-transformers>=2.2; extra == "semantic"
Requires-Dist: numpy>=1.24; extra == "semantic"
Provides-Extra: all
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: anthropic>=0.18; extra == "all"
Requires-Dist: pyyaml>=6.0; extra == "all"
Requires-Dist: sentence-transformers>=2.2; extra == "all"
Requires-Dist: numpy>=1.24; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"

# llm-regtest

A lightweight regression testing framework for LLM prompts. Catch semantic drift before it ships.

---

## What does this tool do?

When you use an AI model, the exact wording of your instructions — called a **prompt** — has a big effect on the quality of responses. Small wording changes can cause unexpected regressions that are impossible to spot without running tests.

This tool works like a test suite for prompts:

1. You define test cases with prompts and optional inputs
2. Run `update-baseline` — the AI's responses become your **golden dataset**
3. Later, after changing a prompt or upgrading a model, run `llm-regtest run`
4. The tool compares new responses to the baselines and flags anything that regressed

Each comparison produces a **score from 0.0 to 1.0** using one or more methods:

| Method | How it works | Best for |
|--------|-------------|----------|
| `exact` | Character-for-character match (0 or 1) | Classification labels, structured outputs |
| `fuzzy` | Levenshtein edit-distance ratio | General text where minor wording drift is acceptable |
| `semantic` | Cosine similarity of sentence embeddings (`all-MiniLM-L6-v2`) | Longer outputs where meaning matters more than wording |
| `llm_judge` | An LLM rates quality similarity 0.0–1.0 | Conversational outputs where both fuzzy and semantic are too strict |

Results are bucketed into **PASS / WARN / FAIL** based on configurable score thresholds.

---

## What you need before starting

- **Python 3.10 or newer** ([download here](https://www.python.org/downloads/))
- **pip** — comes with Python automatically
- **An API key** — for OpenAI ([get one here](https://platform.openai.com/api-keys)) or Anthropic ([get one here](https://console.anthropic.com/))

To check your Python version:

```bash
python --version
```

---

## Installation

```bash
# OpenAI support
pip install -e ".[openai]"

# Anthropic (Claude) support
pip install -e ".[anthropic]"

# Semantic similarity scoring (sentence-transformers)
pip install -e ".[semantic]"

# Everything
pip install -e ".[all]"
```

> The `[semantic]` extra installs `sentence-transformers` and `numpy`. The first time you use semantic scoring, the `all-MiniLM-L6-v2` model (~80 MB) is downloaded automatically.

---

## Quickstart

### 1. Initialise the project

```bash
llm-regtest init
```

Creates `.promptregtest/config.json` and a `prompt_cases.json` file.

### 2. Set your API key

```bash
# Mac / Linux
export OPENAI_API_KEY="sk-..."

# Windows PowerShell
$env:OPENAI_API_KEY="sk-..."
```

### 3. Define test cases

Edit `prompt_cases.json`:

```json
[
  {
    "id": "summarize-article",
    "prompt": "Summarize the following text in one sentence:",
    "input": "Scientists discovered that short 10-minute walks after meals reduce blood sugar spikes.",
    "baseline_output": "",
    "tags": ["summarization"]
  },
  {
    "id": "sentiment-label",
    "prompt": "Classify the sentiment. Reply with exactly one word: positive, negative, or neutral.",
    "input": "I absolutely loved the product!",
    "baseline_output": "",
    "tags": ["classification"]
  }
]
```

### 4. Generate baselines

```bash
llm-regtest update-baseline
```

This calls the AI for every case and saves the responses as baselines inside `prompt_cases.json`.

### 5. Run the regression tests

```bash
llm-regtest run
```

Sample output:

```
  [PASS] summarize-article (fuzzy: 0.94, semantic: 0.97, agg=0.96) [312ms, $0.00018]
  [WARN] email-rewrite (fuzzy: 0.71, semantic: 0.79, agg=0.77) [289ms, $0.00014]
  [FAIL] sentiment-label (exact: 0.00, fuzzy: 0.43, agg=0.21) [198ms, $0.00009]

  ------------------------------------------------
  Results (3 cases): 1 passed, 1 warned, 1 failed
  Cost: $0.00041  |  Latency: 266.3ms avg
  ------------------------------------------------
```

Each result line now shows **latency** (ms) and **cost** (USD) alongside the score. The summary line aggregates total cost and average latency across the run.

### 6. View a saved report

```bash
llm-regtest report
```

---

## How the Regression Check Works

```
prompt_cases.json            .promptregtest/baselines/
┌──────────────────┐         ┌───────────────────────────┐
│  id: "summarize" │         │  summarize.txt             │
│  prompt: "..."   │──run──▶ │  "Short 10-min walks..."   │◀── baseline
│  input:  "..."   │         └───────────────────────────┘
└──────────────────┘                      │
                                          │ compare
         New run output ──────────────────┘
         "Brief post-meal walks..."
                  │
                  ▼
         scorer.semantic_similarity(new, baseline)
              = cosine(embed(new), embed(baseline))
              = 0.91  →  PASS (threshold: 0.85)
```

On each run, the tool:

1. Loads every case from `prompt_cases.json`
2. Sends the prompt (+ optional input) to the configured model
3. Measures **latency** (wall-clock ms) and **token counts** for that call
4. Computes **USD cost** from the provider's pricing table
5. Runs each configured scorer against the stored baseline
6. Computes a **weighted aggregate score**
7. Compares it to the `pass` / `warn` thresholds
8. Saves a JSON report to `.promptregtest/reports/`

---

## Cost and Latency Tracking

Every run automatically captures:

| Metric | Where it appears |
|--------|-----------------|
| Per-request latency (ms) | Beside each result: `[312ms, $0.00018]` |
| Per-request USD cost | Beside each result: `[312ms, $0.00018]` |
| Total run cost | Summary line: `Cost: $0.00041` |
| Average latency | Summary line: `Latency: 266.3ms avg` |
| Input / output token counts | Saved in the JSON report |

Cost is calculated from built-in pricing tables for common OpenAI and Anthropic models. Unknown models report `$0.00000` (no crash). The JSON report stores `input_tokens`, `output_tokens`, `cost_usd`, and `latency_ms` per case for later analysis.

---

## Semantic Similarity Scoring

The `semantic` scorer encodes both the new output and the baseline as sentence embeddings, then computes their **cosine similarity**. Two responses that express the same idea in different words score close to 1.0; responses with entirely different meaning score near 0.0.

**Install:**

```bash
pip install -e ".[semantic]"
```

**Configure:**

```json
{
  "scoring": {
    "methods": ["fuzzy", "semantic"],
    "weights": { "fuzzy": 0.3, "semantic": 0.7 },
    "thresholds": { "pass": 0.85, "warn": 0.65 }
  }
}
```

The `all-MiniLM-L6-v2` model is downloaded once and cached in memory across cases in the same run. If `sentence-transformers` is not installed, the scorer is simply not registered — no crash on import. It will only fail if you explicitly request `"semantic"` in your config without the package installed.

---

## Comparing Two Versions of a Prompt

The most common workflow: lock in version A, change the prompt, compare.

**1. Generate the baseline (version A)**

```bash
llm-regtest update-baseline
```

**2. Modify your prompt in `prompt_cases.json`**

```json
"prompt": "Write a one-sentence summary focusing on the key finding:"
```

**3. Run the comparison**

```bash
llm-regtest run --verbose
```

`--verbose` also prints a line-by-line diff between the baseline and the new output for each case.

**4. Compare two saved reports side-by-side**

```bash
llm-regtest compare \
  --report-a .promptregtest/reports/report_before.json \
  --report-b .promptregtest/reports/report_after.json
```

Output:

```
  Case                score-a  score-b    delta  change
  ------------------------------------------------------
  email-reply           0.73     0.95    +0.22  IMPROVED
  summarize-article     0.91     0.84    -0.07  REGRESSED
  sentiment-label       1.00     1.00    +0.00  unchanged

  1 improved, 1 regressed, 1 unchanged
```

---

## End-to-End OpenAI Demo

A complete demo script is included that shows the full workflow with real OpenAI API calls — including baseline generation, clean regression run, drift simulation, and A/B comparison.

**Prerequisites:**

```bash
pip install "llm-regtest[openai,semantic]"
export OPENAI_API_KEY="sk-..."
```

**Run:**

```bash
python examples/demo_openai.py
```

The script runs four steps automatically:

| Step | What happens |
|------|-------------|
| 1/4 Generate baselines | Calls `gpt-4o-mini` for all 8 cases and saves responses |
| 2/4 Regression run | Reruns all cases — scores should be near 1.0 |
| 3/4 Simulate drift | Rewrites two prompt wordings to mimic a real prompt change |
| 4/4 Detect regression | Reruns with drifted prompts — expect WARN/FAIL on changed cases |
| Bonus A/B compare | Side-by-side delta table between the two runs |

The 8 demo cases cover summarization, tone rewriting, sentiment classification, and factual Q&A — a representative spread for testing how different task types respond to prompt drift.

---

## Configuration Reference

Default config lives at `.promptregtest/config.json`:

```json
{
  "model": {
    "provider": "openai",
    "model_name": "gpt-4o-mini",
    "temperature": 0.0,
    "max_tokens": 1024,
    "system_prompt": ""
  },
  "scoring": {
    "methods": ["fuzzy", "semantic"],
    "weights": { "fuzzy": 0.3, "semantic": 0.7 },
    "thresholds": {
      "pass": 0.85,
      "warn": 0.65
    },
    "llm_judge_model": null
  },
  "prompt_cases_path": "prompt_cases.json",
  "reports_dir": ".promptregtest/reports",
  "baselines_dir": ".promptregtest/baselines",
  "concurrency": 1
}
```

| Setting | What it does |
|---------|-------------|
| `provider` | `"openai"`, `"anthropic"`, or `"stub"` (no API key needed, for testing) |
| `model_name` | Model ID, e.g. `"gpt-4o-mini"`, `"claude-sonnet-4-6"` |
| `temperature` | Set to `0.0` for deterministic, repeatable outputs — strongly recommended for testing |
| `max_tokens` | Maximum response length |
| `system_prompt` | Global system-level instruction sent with every case |
| `methods` | Scorers to use: any combination of `"exact"`, `"fuzzy"`, `"semantic"`, `"llm_judge"` |
| `weights` | Per-method weights for the aggregate. Omit for equal weighting |
| `thresholds.pass` | Score at or above this → PASS (default: `0.8`) |
| `thresholds.warn` | Score at or above this → WARN (default: `0.5`); below → FAIL |
| `llm_judge_model` | Model config for the LLM-as-judge scorer (same shape as `model`) |
| `concurrency` | Cases to run in parallel (default: `1`) |

---

## Test Case Fields

```json
{
  "id": "my-test",
  "prompt": "Summarize in one sentence:",
  "prompt_file": "prompts/summarize.md",
  "system_prompt": "You are a concise assistant.",
  "system_prompt_file": "prompts/system/concise.md",
  "input": "Text to summarize goes here.",
  "inputs": ["Input A", "Input B", "Input C"],
  "inputs_file": "fixtures/reviews.json",
  "variables": { "name": "Alice", "role": "engineer" },
  "baseline_output": "",
  "tags": ["smoke", "summarization"]
}
```

| Field | Required | What it does |
|-------|----------|--------------|
| `id` | Yes | Unique identifier (no spaces) |
| `prompt` | Yes* | The instruction text |
| `prompt_file` | Yes* | Path to a `.txt` or `.md` file containing the prompt |
| `system_prompt` | No | Per-case system prompt (overrides global config) |
| `system_prompt_file` | No | Path to a file containing the system prompt |
| `input` | No | Extra text appended to the prompt |
| `inputs` | No | List of inputs — generates `id[0]`, `id[1]`, ... sub-cases |
| `inputs_file` | No | JSON file with a list of input strings |
| `variables` | No | Values for `{placeholder}` templates in the prompt |
| `baseline_output` | No | Auto-filled by `update-baseline` — leave blank |
| `tags` | No | Labels for filtering with `--tag` |

*`prompt` or `prompt_file` is required, not both.

---

## CLI Reference

| Command | What it does |
|---------|-------------|
| `llm-regtest init` | Create the project folder structure |
| `llm-regtest init --ci` | Also create a GitHub Actions workflow file |
| `llm-regtest update-baseline` | Run prompts and save responses as new baselines |
| `llm-regtest run` | Run prompts and compare to existing baselines |
| `llm-regtest report` | Display the most recent saved report |
| `llm-regtest compare` | Compare two saved reports side-by-side |

**Flags for `run` and `update-baseline`:**

| Flag | What it does |
|------|-------------|
| `--config PATH` | Use a non-default config file |
| `--case ID` | Run only this case ID (repeatable) |
| `--tag TAG` | Run only cases with this tag (repeatable, OR logic) |
| `--concurrency N` | Run N cases in parallel |

**Flags for `run` only:**

| Flag | What it does |
|------|-------------|
| `--verbose` / `-v` | Print a unified diff for each case |
| `--format console` | Default coloured output |
| `--format github` | GitHub Actions `::error::` / `::warning::` annotations |

---

## Advanced Features

### Semantic similarity

```json
"scoring": {
  "methods": ["fuzzy", "semantic"],
  "weights": { "fuzzy": 0.3, "semantic": 0.7 }
}
```

Requires `pip install -e ".[semantic]"`. Uses `all-MiniLM-L6-v2`.

### LLM-as-judge

```json
"scoring": {
  "methods": ["fuzzy", "llm_judge"],
  "weights": { "fuzzy": 0.4, "llm_judge": 0.6 },
  "llm_judge_model": {
    "provider": "openai",
    "model_name": "gpt-4o-mini"
  }
}
```

### Parallel execution

```bash
llm-regtest run --concurrency 10
```

### Tag filtering

```bash
llm-regtest run --tag smoke           # fast per-PR smoke suite
llm-regtest run --tag smoke --tag critical  # OR logic
llm-regtest update-baseline --tag customer-facing
```

### Parameterized inputs

```json
{
  "id": "classify-sentiment",
  "prompt": "Classify as positive, negative, or neutral:",
  "inputs": [
    "Love it!",
    "Broke after one day.",
    "It's fine."
  ]
}
```

Generates sub-cases `classify-sentiment[0]`, `classify-sentiment[1]`, `classify-sentiment[2]`.

### Prompt files

```json
{
  "id": "legal-analysis",
  "prompt_file": "prompts/analyze_contract.md",
  "system_prompt_file": "prompts/system/legal_analyst.md",
  "input": "..."
}
```

Paths are relative to the directory containing `prompt_cases.json`. Prompt files appear as clean diffs in pull requests.

### Custom scorers

```python
from llm_regtest.scorer import register_scorer

def keyword_overlap(output: str, baseline: str) -> float:
    out_words = set(output.lower().split())
    base_words = set(baseline.lower().split())
    if not base_words:
        return 1.0
    return len(out_words & base_words) / len(base_words)

register_scorer("keyword_overlap", keyword_overlap)
```

Then add `"keyword_overlap"` to `methods` in your config.

---

## CI / GitHub Actions

### Generate a workflow file

```bash
llm-regtest init --ci
```

Creates `.github/workflows/prompt-regression.yml` — runs on every PR that touches prompt files.

### Annotated PR output

```bash
llm-regtest run --format github
```

Emits `::error::` and `::warning::` lines that GitHub renders as inline annotations on the PR diff.

### Example workflow

```yaml
name: Prompt Regression Tests
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'prompt_cases.json'
jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -e ".[openai,semantic]"
      - run: llm-regtest run --format github
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

Add `OPENAI_API_KEY` as a repository secret under **Settings → Secrets → Actions**.

---

## Using Claude (Anthropic) as the Model

```json
{
  "model": {
    "provider": "anthropic",
    "model_name": "claude-haiku-4-5-20251001",
    "temperature": 0.0,
    "max_tokens": 1024
  }
}
```

```bash
pip install -e ".[anthropic]"
export ANTHROPIC_API_KEY="sk-ant-..."
```

Available model names: `claude-haiku-4-5-20251001`, `claude-sonnet-4-6`, `claude-opus-4-6`.

---

## Understanding the Results

| Status | Score range | Meaning |
|--------|------------|---------|
| **PASS** | ≥ `thresholds.pass` | Response is very similar to baseline |
| **WARN** | ≥ `thresholds.warn` | Response has drifted noticeably — worth reviewing |
| **FAIL** | < `thresholds.warn` | Significant regression detected |
| **SKIP** | — | No baseline exists for this case yet |

Default thresholds: `pass = 0.8`, `warn = 0.5`. Adjust in `config.json` under `scoring.thresholds`.

---

## Troubleshooting

**"No module named llm_regtest"**
Run `pip install -e .` from the project root directory.

**"OPENAI_API_KEY not set" or authentication errors**
Set the environment variable in the same terminal window you're running the tool from.

**All tests show "skip"**
No baselines yet. Run `llm-regtest update-baseline` first.

**Semantic scorer not available**
Install with `pip install -e ".[semantic]"`. The scorer is silently skipped if `sentence-transformers` is not installed.

**Scores are lower than expected after a small change**
Switch from `exact` to `fuzzy` or `semantic` scoring, which are more tolerant of minor wording differences.

**Runs are slow with many cases**
Use `--concurrency N` to run cases in parallel. Start with `5` and increase if your API rate limits allow.
