Metadata-Version: 2.4
Name: prompt-cache-doctor
Version: 0.1.0
Summary: Diagnose LLM prompt-cache hit rates, find wasted tokens, and get prompt-reordering advice that cuts your API bill.
Author: mathCrazyy
License: MIT
Project-URL: Homepage, https://github.com/mathCrazyy/prompt-cache-doctor
Project-URL: Guide, https://github.com/mathCrazyy/prompt-cache-doctor#guide
Project-URL: Issues, https://github.com/mathCrazyy/prompt-cache-doctor/issues
Keywords: llm,prompt-caching,openai,anthropic,cost,token-optimization,developer-tools
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# prompt-cache-doctor

**Your LLM bill has a silent leak: cache misses. `pcd` finds them.**

Every major LLM provider serves cached prompt tokens at a steep discount —
OpenAI at 50%, Anthropic at 10%, DeepSeek at ~25% of the input price. But
caches are **prefix-matched**: from the first changed token on, everything
is recomputed at full price. A user question placed *before* your 12k-token
knowledge base quietly busts the cache on every single request.

`pcd` analyzes your API logs, measures your real cache hit rate, prices the
waste, and rewrites your prompt templates into **static-first** layouts.

```
$ pcd analyze traffic.jsonl --model gpt-4o
requests analyzed      : 40
input tokens           : 515,400
cached tokens          : 468,000
cache hit ratio        : 90.8%     ← healthy layout

$ pcd analyze bad-traffic.jsonl --model gpt-4o
cache hit ratio        : 3.8%      ← same traffic, question-first template
extra savings possible : $0.59     ← static-first prompt layout
```

## Install

```bash
pip install prompt-cache-doctor     # or: pipx install prompt-cache-doctor
```

Zero dependencies. Python ≥ 3.9. Works offline — your logs never leave
your machine.

## The one rule

> **LLM prompt caches are prefix-matching. Put static content first,
> dynamic content last.** One changed token invalidates every token
> after it.

```
❌  [user question] + [12k-token knowledge base]   → ~0% hits
✅  [12k-token knowledge base] + [user question]   → ~90% hits
```

## Usage

### `pcd analyze` — what is the cache leak costing you?

Feed it JSONL request logs (one record per line) in OpenAI, Anthropic, or
GLM/Zhipu usage shapes:

```bash
pcd analyze logs.jsonl --model gpt-4o          # any model in the rate table
pcd analyze logs.jsonl --model sonnet --json   # machine-readable
pcd analyze logs.jsonl --html report.html      # + visual report
```

You get: hit ratio, cached/computed token totals, actual cost vs
no-cache cost, dollar headroom from a static-first rewrite, and your worst
requests ranked.

### `pcd scan` — find cache-busting prompt templates

Point it at a template file or directory (`.txt .md .j2 .jinja .prompt …`):

```bash
pcd scan prompts/
```

```
prompts/bad_support_bot.prompt
  total tokens (approx) : 308
  first dynamic token   : #3
  static-after-dynamic  : 301 tokens
  volatile prefix vars  : {{timestamp}}  (changes every request → busts cache)
  suggested static-first layout:
    1. [static] knowledge base, tone rules, product docs …
    2. [dynamic] {{ticket_id}}, {{timestamp}}, {{question}}, …
```

It detects `{var}`, `{{ jinja }}`, `${var}`, `<%= erb %>` syntax, flags
static blocks stranded after the first variable, and warns about
per-request values (`timestamp`, `request_id`, …) sitting in your prefix.

### `pcd report` — one HTML file for your team

```bash
pcd report logs.jsonl --model gpt-4o --scan prompts/ -o report.html
```

Self-contained dark-mode report: summary cards, per-request hit-ratio
chart, cost comparison bars, worst requests, template findings.

### Accurate pricing

Built-in rates are indicative (per 1M tokens, USD). Override with your own
table:

```bash
pcd analyze logs.jsonl --model my-model --rates my-rates.json
```

```json
{"my-model": {"provider": "vllm", "input": 0.5, "cached_input": 0.1, "output": 1.5}}
```

## Provider cheat sheet

| Provider | Cached-input price | Min prefix | Notes |
|---|---|---|---|
| OpenAI | 50% of input | 1024 tok | automatic, ≥10-min idle eviction |
| Anthropic | 10% of input | 1024/2048 tok | explicit `cache_control` breakpoints; writes cost 125% |
| DeepSeek | ~25% of input | 64 tok | automatic |
| Zhipu GLM | discounted input | model-dependent | automatic |

## How it works

- `engine.py` — prefix-matching cache simulator (single-slot and
  multi-entry models) ported from a controlled experiment set; the
  "interleaved agents" scenario is reproduced in the tests
- `analyzer.py` — normalizes `usage` payloads across providers,
  aggregates hit stats, prices traffic with/without caching
- `scanner.py` — segments templates into static/dynamic spans and measures
  how many static tokens sit after the first dynamic one
- Everything is local, dependency-free, and covered by 27 unit tests

## Guide: *Prompt Caching Engineering*

A deeper companion guide — how each provider's cache actually behaves,
three controlled experiments (layout, prefix length, multi-agent
interleaving), and a prompt-design checklist:
see [`content/prompt-caching-engineering.md`](content/prompt-caching-engineering.md).

## FAQ

**My provider caches automatically. Why do I need this?**
Automatic caching is still prefix-matched. The provider caches *whatever
prefix you send* — if your template puts a timestamp or user text first,
"automatic" caching has nothing stable to cache. `pcd scan` catches exactly
that.

**Does it send my logs anywhere?**
No. It's offline, dependency-free, and MIT-licensed.

**Which loggers produce compatible JSONL?**
Anything that dumps one JSON object per request with a `usage` block —
Helicone exports, LiteLLM proxy logs, OpenAI SDK wrappers, homegrown
middleware. `examples/make_examples.py` shows the exact shapes.

## License

[MIT](LICENSE)
