Metadata-Version: 2.4
Name: mizanbench
Version: 0.1.1
Summary: Open LLM cost simulator: model mix, cascade routing, and savings estimates from real workload numbers.
Author: LL TECHNOLOGIES L.L.C
License: Apache-2.0
Project-URL: Homepage, https://github.com/GlobalMarkGroup/MizanBench
Project-URL: Source, https://github.com/GlobalMarkGroup/MizanBench
Project-URL: Issues, https://github.com/GlobalMarkGroup/MizanBench/issues
Keywords: llm,cost,simulator,benchmark,routing,cascade
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# MizanBench — open LLM cost simulator

Estimate what your LLM workload actually costs — and what smarter routing
would save — before you spend a dollar. MizanBench models three strategies
over your real traffic numbers:

- **Single model** — everything to one model (your baseline)
- **Mix** — static traffic split across models
- **Cascade** — cheap model tries first, unresolved requests escalate

MizanBench's cascade accounting is deliberately **honest**: a request that
escalates past a cheap tier still *pays* for that tier's attempt, exactly
as real routers bill. The savings it reports are conservative, not
marketing numbers.

## Quickstart

```bash
pip install pyyaml
PYTHONPATH=src python -m mizanbench.cli simulate examples/workload.yaml
# (after `pip install -e .`: just `mizanbench simulate examples/workload.yaml`)
```

Output for the bundled example (a 20k-requests/day support assistant):

```
scenario                                 monthly  per-request    vs baseline
-----------------------------------------------------------------------------
all frontier (GPT-5.6 Sol)             $9,900.00        $0.02       baseline
all mid-tier (Claude Sonnet 5)         $3,540.00    $0.005900         -64.2%
70/30 mix (Flash + Sonnet 5)           $1,580.70    $0.002635         -84.0%
cascade nano -> small -> frontier      $1,285.35    $0.002142         -87.0%
```

List the pricing catalog:

```bash
PYTHONPATH=src python -m mizanbench.cli models
```

## Describe your own workload

Copy `examples/workload.yaml` and edit:

```yaml
workload:
  requests_per_day: 20000
  avg_input_tokens: 1200
  avg_output_tokens: 350
  days: 30

scenarios:
  - name: "baseline"
    type: single
    model: gpt-5.6-sol
    baseline: true

  - name: "cascade"
    type: cascade
    tiers:
      - { model: gpt-5.4-nano,     resolve_rate: 0.65 }  # fraction resolved here
      - { model: gemini-2.5-flash, resolve_rate: 0.80 }  # of what escalated
      - { model: claude-opus-4.8 }                       # final tier takes the rest
```

`resolve_rate` is the honest knob: it is the fraction of requests the tier
*successfully terminates*. Measure it from your own eval data — an
optimistic guess here produces optimistic savings.

## Pricing catalog

`src/mizanbench/data/pricing.yaml` ships with prices **as of 2026-08-13** (USD
per 1M tokens) for OpenAI, Anthropic, Google, DeepSeek, Groq, and Together
models. Prices change often:

- override with your own file: `mizanbench -p my-pricing.yaml simulate ...`
- or better, **send a PR updating the catalog** — it's the easiest first
  contribution to this project, and the `as_of` field keeps us honest.

## Use as a library

```python
from mizanbench import PricingCatalog, Workload, simulate_cascade

catalog = PricingCatalog.load()
w = Workload(requests_per_day=20000, avg_input_tokens=1200, avg_output_tokens=350)
result = simulate_cascade(catalog, w, [
    {"model": "gpt-5.4-nano", "resolve_rate": 0.65},
    {"model": "claude-sonnet-5"},
])
print(result.monthly_cost, result.per_request_cost)
```

## Tests

```bash
PYTHONPATH=src python -m unittest discover -s tests   # stdlib, no deps
# or: pytest
```

## Roadmap

- Benchmark harness: measured (not assumed) resolve rates per model/task.
- Latency modeling alongside cost.
- Web calculator UI.

## License

Apache-2.0 — see [LICENSE](LICENSE).
