Metadata-Version: 2.4
Name: proofshift
Version: 0.1.0
Summary: Prove a cost/quality change to your LLM system is safe before you ship it.
Author-email: Rohit Takkole <rohitakkole85@gmail.com>
Maintainer-email: Rohit Takkole <rohitakkole85@gmail.com>
License: Proprietary
Keywords: anthropic,cost,evaluation,llm,observability,openai,optimization
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Requires-Dist: cryptography>=42
Requires-Dist: pydantic-settings>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Description-Content-Type: text/markdown

<div align="center">

<img src="assets/logo.svg" alt="ProofShift — an AI agent that optimizes and verifies your LLM core" width="140" />

# ProofShift

**A local-first CI gate that refuses to ship an LLM cost change unless it's provably safe —
and understands your usage to find the savings.**

</div>

ProofShift is a local-first CLI for AI engineers. Point it at your existing LLM usage
traces and it will:

- `scan` / `advise` (free) — understand your usage and surface dollar-ranked cost
  loopholes (over-high reasoning effort, bloated prompts, missing prompt-caching,
  cheaper-but-capable model swaps). Every finding is a **hypothesis**.
- `proof` (paid) — build a representative eval set from your own traffic, run a
  calibrated judge with honest statistics, and return **SAFE / REGRESSION /
  INCONCLUSIVE** — never a guess. This is the gate.
- `ci` — run in CI and post the findings as a PR check (GitHub-flavored Markdown
  summary + exit code); pair with `proof` to fail a build on a proven regression.

**The honest contract:** findings are hypotheses; `proof` certifies; **dollars only when priced.**
**Privacy invariant (no egress):** ProofShift makes no ProofShift-owned network calls — no
telemetry, no cloud. The only network call it can make is to the LLM endpoint **you** configure,
to replay **your own** traffic. Audit it any time with `proofshift privacy`.
**Honesty invariant:** weak evidence never returns SAFE; an uncalibrated judge never certifies.

## Just run `proofshift` (interactive agent)

Don't want to learn flags? Run it with no arguments and talk to it:

```bash
proofshift                 # opens an interactive session
proofshift chat --continue # resume your last session
```

It runs on **any LLM you point it at** (OpenAI / Anthropic / Bedrock / your gateway — set once),
asks only for what it can't auto-detect, **remembers** your setup per org-profile, and drives
`scan`/`proof`/`pull` itself — you never type cost or config. The plain commands below still work
for CI/scripts. (The agent surfaces; the proof engine still decides — it cannot fabricate a verdict.)

## No observability platform? Capture in one line.

You don't need Langfuse/LangSmith. If you make raw API calls, wrap your client once and every
call is recorded to a local trace file (token-shape only; nothing leaves your machine):

```python
import proofshift
from openai import OpenAI                      # or: from anthropic import Anthropic

client = proofshift.wrap(OpenAI())             # one line — every call now recorded
client.chat.completions.create(model="gpt-4o", messages=[...])   # your normal code

# opt in to capturing prompt TEXT locally (enables live `proof` replay):
client = proofshift.wrap(OpenAI(), content="content.jsonl")
```

Then `proofshift scan proofshift_traces.jsonl`. The wrapper is provider-agnostic (OpenAI- or
Anthropic-style clients), has no SDK dependency, and is transparent — a capture error can never
break your actual call.

### Or change zero code — point a base URL at the proxy

For non-Python apps (JS/Go/anything) or when you can't touch the code, run the transparent
recording proxy and just change your client's base URL:

```bash
proofshift proxy --port 8787            # forwards to OpenAI, records locally
# then set your app's base_url = http://127.0.0.1:8787/v1   (your key passes through, never stored)
```

It's an OpenAI-compatible endpoint that forwards every call verbatim to the real provider and
records it on the way through (stdlib-only, no web framework). `--upstream` points it at a
different provider/gateway; `--content` opts into local prompt capture.

## Install

ProofShift installs as a normal command-line tool — `proofshift` ends up on your `PATH`
(like `gh` or `ruff`); you do **not** need `uv` to use it.

```bash
# Recommended: isolated global install
uv tool install proofshift        # from PyPI (once published)
pipx install proofshift           # or pipx
pip install proofshift            # or a plain pip/venv

# Internal rollout before publishing: install the built wheel directly
uv tool install ./dist/proofshift-0.1.0-py3-none-any.whl

proofshift --version
```

(`uv run proofshift …` in the examples below is only for running from a **source checkout**
during development — installed users just type `proofshift …`.)

## Quickstart (see everything work in 30 seconds, no keys)

The fastest tour is the built-in demo. It runs the WHOLE pipeline — scan → fit → advise →
seed content → **a real `optimize` verdict** → privacy — entirely OFFLINE on bundled
fixtures, under an isolated profile it sets and clears itself (idempotent, safe to re-run):

```bash
proofshift demo
```

No API key required: the optimize step uses a deterministic offline judge + completer, so it
reaches an actual SAFE verdict on a held-out test split instead of failing with "no captured
content". `examples/demo.sh` is the shell equivalent.

> **The demo's verdict is a pipeline demonstration, not a real quality test.** It injects a
> deterministic offline stub judge that accepts every pair — so it **always** reaches SAFE and
> can **never** show REGRESSION. It exercises the whole pipeline offline; a real run uses a
> calibrated LLM judge via your key.

### What needs captured content

Most commands work on the **metadata-only** traces you can pull with **no API key**. Only
`proof`/`optimize` need the actual **prompt text** to replay — and LangSmith/Langfuse pulls
carry only token metadata, so on pulled traces those two return INCONCLUSIVE
(`insufficient_captured_content`) until you capture text locally.

| Command | Needs captured prompt text? | Works on pulled metadata (no key)? |
|---|---|---|
| `scan` / `fit` / `advise` / `recommend` | no | ✅ yes |
| `privacy` / `demo` | no | ✅ yes |
| `proof` / `optimize` | **yes** | ❌ no — returns INCONCLUSIVE without it |

To capture prompt text locally, run `proofshift proxy` (point your client's base URL at it) or
`proofshift.wrap(client, content="content.jsonl")` (one line), then re-run `proof`/`optimize`.
Nothing leaves your machine until you replay against the LLM endpoint **you** configure.

Or step through it yourself on `examples/sample_traces.jsonl` (12 priced calls):

```bash
# tell ProofShift which models you can actually call (fit/advise reason over these)
proofshift models add gpt-4.1-mini --context 1000000 --input-price 0.4 --output-price 1.6
proofshift models add gpt-5        --context 400000  --input-price 1.25 --output-price 10

proofshift scan   examples/sample_traces.jsonl   # find + rank likely cost waste (free)
proofshift fit    examples/sample_traces.jsonl   # cheaper-but-capable swaps among YOUR models
proofshift advise examples/sample_traces.jsonl   # the one dollar-ranked "what to do next" feed
proofshift privacy                               # exactly what can/can't leave this machine

# which cheaper model still covers a model's strengths?
proofshift recommend claude-opus-4-8
```

Point `scan`/`advise` at your own exported traces — **OpenTelemetry GenAI spans, LiteLLM logs,
Langfuse observations, or generic JSONL** (auto-detected). Nothing leaves your machine.

## Use it as a CI gate

```bash
# Post dollar-ranked findings as a PR check (writes $GITHUB_STEP_SUMMARY + stdout):
proofshift ci traces.jsonl
proofshift ci traces.jsonl --fail-on-waste-usd 50   # fail the build if recoverable ≥ $50

# Gate an actual change: fail the build unless it is PROVEN safe.
proofshift proof traces.jsonl --group openai:gpt-5:classify \
  --kind reasoning_effort --from high --to medium --judge anthropic --content content.jsonl
```

`ci` always exits 0 (or 1 with `--fail-on-waste-usd` over budget; 3 if traces can't be read).
`proof` exits **0 SAFE / 2 REGRESSION / 3 INCONCLUSIVE** — the CI-gate contract. Drop-in workflow
templates ship with the repo: [`.github/workflows/proofshift.yml`](.github/workflows/proofshift.yml)
(surface findings on every PR) and [`examples/ci/proof-gate.yml`](examples/ci/proof-gate.yml)
("don't merge a cost change unless it's proven safe").

### Get your traffic in
- **LangChain / LangGraph:** `pip install 'proofshift[langchain]'`, then
  ```python
  from proofshift.integrations.langchain import make_langchain_handler
  graph.invoke(state, config={"callbacks": [make_langchain_handler("traces.jsonl")]})
  ```
- **Already on Langfuse?** `proofshift pull` fetches your traces automatically (set
  `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` / `LANGFUSE_HOST`) — **no manual export** —
  then `proofshift scan langfuse_traces.json`.
- **Raw OpenAI/Anthropic / custom agents:** wrap calls with `proofshift.capture.Recorder`.

## Command map
| Command | What it does |
|---|---|
| `scan` | find + rank likely cost/time waste from traces (free) |
| `fit` | profile usage + cheaper-but-capable swaps among your registered models |
| `advise` | the one dollar-ranked "what to do next" feed (merges scan + fit) |
| `models` | register/list/remove the models you can actually call (the intake) |
| `proof` | prove a change is SAFE / REGRESSION / INCONCLUSIVE (CI-gate exit codes 0/2/3) |
| `optimize` | design a safe, cheaper prompt rewrite and certify it (opt-in, local-content) |
| `privacy` | show exactly what can and cannot leave this machine (the no-egress promise) |
| `pull-langsmith` | pull llm runs directly from LangSmith (no manual export) |
| `pull` | fetch traces from Langfuse (no export) |
| `recommend` | cheaper, capable model alternatives (knowledge base) |
| `calibrate` | calibrate a judge vs human anchors (κ gate) before trusting it |
| `apply` | emit reviewable change instructions (ProofShift never edits your code) |
| `ci` | run in CI: GitHub-flavored Markdown summary + meaningful exit code |
| `chat` | the interactive agent — just run `proofshift` with no arguments |

Full plan: `docs/specs/2026-06-27-proofshift-dev-plan.md`.

## Privacy (no egress)
ProofShift runs entirely on your machine and makes **no ProofShift-owned network calls** — no
telemetry, no cloud. Metadata pulls (LangSmith/Langfuse) carry **only model ids + token counts**,
never prompt text. The single path that can carry your text is to the LLM endpoint **you**
configure for `proof`/`optimize`, replaying your own traffic. `proofshift privacy` prints the
full, auditable list of what can leave and whether it carries your text.

Reading raw prompt/response **text** is **opt-in** and local: prompt optimization needs
`--local-content` (or `PROOFSHIFT_LOCAL_CONTENT=1`). By default ProofShift reads only token
counts + model ids — never your prompt text. Without the opt-in, `optimize` explains itself and
exits cleanly instead of touching any text.

## LLM backends (for `proof` / `calibrate`)
Provider-agnostic by design — the judge and runner take any completion function, and routing
is a **registry of adapters** (`proofshift.providers.registry`), not an `if/elif`. Built-in:
- **OpenAI** and **any OpenAI-compatible endpoint** via `--base-url` — Ollama
  (`http://localhost:11434/v1`), vLLM, LiteLLM, Together, Groq, Azure, Fireworks, or your
  own service. Set `OPENAI_API_KEY` (use `OPENAI_BASE_URL` to point elsewhere).
- **Anthropic** (Claude) — set `ANTHROPIC_API_KEY`.
- **AWS Bedrock** — Bedrock-style ids are deliberately refused (never misrouted); reach
  Bedrock/Gemini today via an OpenAI-compatible gateway, or register a native adapter (below).

Keys are read from the environment, never pasted into code or committed.

### Adding / overriding a backend (no core changes)
Adapters are data. Register one to add a native family or change behaviour for your org:

```python
from proofshift.providers.registry import ProviderAdapter, with_adapters
registry = with_adapters(ProviderAdapter(name="google", provider=Provider.GOOGLE,
    matches=lambda m: m.startswith("gemini"), native_cache_output_preserving=True,
    default_judge_model="gemini-2.5-pro", make_judge=..., complete_model=...))
```

The registry is the single source of truth: the cache-preservation policy derives its trusted
providers from the adapters that declare `native_cache_output_preserving`, so adding a backend
automatically extends what `enable_cache` can certify.

### Multi-tenant (per-user / per-organisation)
Set `PROOFSHIFT_PROFILE=<org>` to namespace all local state — calibration, price overrides, db —
under `~/.proofshift/profiles/<org>/`. Different users and organisations stay isolated on one
machine with no collisions; everything else (pricing via `PROOFSHIFT_PRICES_PATH`, stores via
`--store`/`--content`) remains explicitly overridable per invocation.

### What `proof` certifies (v1 scope)
- **model_swap** — judged via a live replay (calibrated judge + captured prompts); may be
  cross-provider (e.g. baseline OpenAI vs variant Anthropic).
- **reasoning_effort** — judged via a live replay too: the change is applied through a
  provider-agnostic params channel (OpenAI's native `reasoning_effort`; on Claude it maps to
  an extended-thinking budget). Dollar savings for effort changes come from `scan`/`recommend`,
  not the proof verdict (the verdict certifies *safety*).
- **enable_cache** — SAFE by construction **on OpenAI / Anthropic**, where prompt caching is
  contractually output-preserving; for other backends reached via a gateway, `proof` returns
  INCONCLUSIVE (`cache_preservation_unverified`) until you confirm your backend preserves output.
  The reported figure
  is **estimated net savings**: the per-read discount minus the one-time cache-creation write,
  assuming a stable prefix and ignoring TTL eviction — so it can be small or negative for
  unstable/low-volume prefixes.

A SAFE verdict needs a calibrated judge **and** enough cases that the statistics can clear the
floor: at the default acceptance floor (90%), even a flawless record needs 35 judged cases
(the gate raises its minimum to match the floor so it is never silently unwinnable).

The LLM judge is **order-robust**: every pair is judged twice with baseline/variant positions
swapped, and the variant is accepted only if judged non-inferior regardless of position — so an
uncontrolled position bias can't leak into the verdict. The reported swap-consistency rate makes
any residual order-dependence visible.

Native AWS Bedrock / Gemini judges are planned; today, reach them via an OpenAI-compatible
gateway (e.g. a LiteLLM proxy or Bedrock Access Gateway) with `--base-url`.

## Develop

```bash
uv sync            # provisions Python 3.12 + installs deps
uv run proofshift --version
uv run pytest      # run the test suite
uv run ruff check  # lint
uv run mypy        # type-check (strict)
```
