Metadata-Version: 2.4
Name: tokenwatt
Version: 0.3.0
Summary: Know what your local LLM inference actually costs in electricity (Apple Silicon).
Author-email: Justin Stewart <justinstewart313+tokenwatt@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/mmmugh/tokenwatt
Project-URL: Repository, https://github.com/mmmugh/tokenwatt
Project-URL: Issues, https://github.com/mmmugh/tokenwatt/issues
Keywords: llm,inference,electricity,energy,cost,apple-silicon,mlx,proxy,openai
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: starlette<2,>=1.0
Requires-Dist: uvicorn>=0.30
Requires-Dist: httpx>=0.27
Requires-Dist: typer>=0.12
Requires-Dist: tiktoken>=0.7
Requires-Dist: zeus-apple-silicon>=1.0; sys_platform == "darwin" and platform_machine == "arm64"
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml>=6
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Dynamic: license-file

# TokenWatt

[![CI](https://github.com/mmmugh/tokenwatt/actions/workflows/ci.yml/badge.svg)](https://github.com/mmmugh/tokenwatt/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/tokenwatt)](https://pypi.org/project/tokenwatt/)
[![Python](https://img.shields.io/pypi/pyversions/tokenwatt)](https://pypi.org/project/tokenwatt/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/mmmugh/tokenwatt/blob/main/LICENSE)
![Platform](https://img.shields.io/badge/platform-macOS_Apple_Silicon-lightgrey)

**Know what your local LLM inference actually costs you in electricity — per model, per request, on Apple Silicon. No sudo.**

TokenWatt is a transparent, OpenAI-compatible proxy. It sits in front of your local inference server, forwards every request byte-for-byte, measures the real per-rail SoC energy via Apple's IOReport (no password, ever), prices it at your utility rate, and logs a per-request, per-model electricity ledger.

```text
TokenWatt — electricity cost of local inference
  (numbers are ESTIMATED until you calibrate against a wall meter)
  last 24h :    119 req   0.0745 kWh   $0.0231

  model           type   req      kWh        $    J/tok   $/Mtok   conf
  qwen3.6-27b     text   119   0.0745   $0.0231   5.019    0.432    est
```

*Real capture: an 84-minute agentic coding session (qwen3.6-27B building a Scheme interpreter,
119 streamed tool-calling turns) measured through TokenWatt on an M3 Ultra at $0.31/kWh.*

## The number people actually want

```text
$ tokenwatt compare
local electricity vs cloud LIST price for the SAME tokens (cloud snapshot 2026-06; list
price incl. their compute+margin, not just power; edit cloud.py):
  qwen3.6-27b   $0.0231 electricity — ~27.2× cheaper than gemini-2.5-flash-lite ($0.6286 vs $0.0231)
```

That session re-sent its growing context every turn — **6.07M input tokens for 53K output**, the
shape of high-context agentic coding. Locally, re-processing that context costs only electricity,
not a per-token API charge. A cloud API bills those re-sent tokens too — and even prompt caching
(which discounts re-sent context) only narrows it: local stays well ahead in this regime.

That leverage is specific to high-context agentic loops. For **balanced chat** (input ≈ output),
the cheapest cloud models are now *competitive* — gemini-2.5-flash-lite's $0.40/Mtok output is about
the same as local's $0.432, so there the win is privacy and control, not raw cost. TokenWatt shows
you which regime you're actually in — honestly, in both directions.

> Numbers are labeled **estimated (±15–30%)** until you calibrate against a wall meter, and the
> local figure is electricity only — it doesn't amortize the Mac. The point isn't one headline
> multiple; it's that you can see your real number, honestly, for *your* machine and *your* rate.

## Install

```bash
uv tool install tokenwatt        # or:  uvx tokenwatt  ·  pip install tokenwatt
```

## Use

```bash
tokenwatt init                         # scaffold a commented tokenwatt.yaml (routes + your $/kWh)
tokenwatt serve -c tokenwatt.yaml      # one port in front of your local backends; no API key, no sudo
# point your OpenAI client (Pi / OpenClaw / Claude Code / any SDK) at http://127.0.0.1:7000/v1

tokenwatt report                       # today/month $, per-model $/Mtok and J/token
tokenwatt compare                      # your electricity vs named cloud prices, for the same tokens
tokenwatt wrap                         # a shareable "my inference bill" card
tokenwatt doctor                       # health-check config, proxy, upstreams, routing, ledger, meter
tokenwatt calibrate probe|campaign|fit # replace the ±15–30% estimate with a plug-measured number (see Calibrate)
```

A single `--upstream` shortcut works too, with no config file:

```bash
tokenwatt serve --upstream http://127.0.0.1:8080 --rate 0.31
```

## Routing

`serve` forwards each request to the backend whose route matches the request's
`model` id — **exact > longest glob > `*` catch-all**:

```yaml
routes:
  - name: mlx
    type: text                       # text | vision | embeddings (request classification)
    upstream: http://127.0.0.1:8080
    match: ["qwen3.6-27b", "mlx-community/*"]
  - name: lmstudio
    type: text
    upstream: http://127.0.0.1:1234
    match: ["*"]                     # catch-all fallback
```

### Dynamic discovery

Maintaining `match` patterns by hand gets tedious when you swap models. Turn on
discovery and routing follows **what's actually loaded** — TokenWatt polls each
upstream and sends a request to wherever its model is currently served, so an
mlx-tui slot swap or an `lms load` needs no config edit and no restart:

```yaml
discovery:
  enabled: true                      # default false
```

`routes` remain the fallback (for a model not loaded anywhere) and supply the
upstream list + per-upstream `type`. One wrinkle it handles for you: backends
disagree on `/v1/models` — mlx-openai-server lists its one *loaded* model, but
LM Studio lists its whole on-disk *catalog*, so discovery reads LM Studio's
`/api/v0/models` load state instead. For a backend whose `/v1/models` is itself a
catalog (e.g. an mlx-vlm vision server), add `discover: false` to its route to
keep it on the static pattern. `tokenwatt doctor` flags any that need it.

## Doctor

`tokenwatt doctor` health-checks the whole setup in one shot — for a quick "is it
wired right?" and for bug reports.

```text
$ tokenwatt doctor
config:    ✓ load: 3 route(s); discovery on
proxy:     ✓ listening: reachable on 127.0.0.1:7000
ledger:    ✓ open: ~/.tokenwatt/ledger.sqlite (2517 request rows)
metering:  ✓ meter: energy accumulating (+19.48 J in 0.4s, no sudo)
upstream:  ✓ http://127.0.0.1:1234: LM Studio: (nothing loaded)
           ✓ http://127.0.0.1:8080: serving: qwen3.6-27b
healthy — 6 ok, 0 warn, 0 fail
```

It checks the **config** (parse + a warning if no rate is set), the **proxy** port,
**every upstream** (what each is actually serving — including LM Studio's loaded
state via `/api/v0/models` vs the full `/v1/models` catalog), **routing**
(model-on-two-upstreams collisions; catalog backends that should set
`discover: false`), the **ledger** (writable + schema + row count), and the
**sudoless energy meter** (confirms a live reading — TokenWatt's whole point).

- `--json` — machine-readable output, for pasting into an issue.
- `--fix` — apply only safe fixes (create the ledger directory, scaffold a missing
  config); never touches ledger data or restarts anything.
- Exits non-zero when something is broken, so it drops into CI / health probes.

## Calibrate

By default costs are **estimated (±15–30%)** — TokenWatt uses SoC rail energy as a stand-in for wall
energy. To replace the estimate with a *measured* number, calibrate against a metering smart plug (a
Shelly Gen2+ with an `aenergy.total` counter). Calibration is per machine **and** per model: a fit
measured on one model doesn't transfer cleanly to another (the token slope drifts for large,
memory-bound MoEs), so each served model gets its own profile, scoped to the exact quantization it was
measured on.

```bash
# point at your plug with --meter-host (or set calibration.meter_host in your config)
tokenwatt calibrate probe    --meter-host shellyplug.local        # sanity-check the plug (not a calibration)
tokenwatt calibrate campaign --meter-host shellyplug.local \
    --upstream http://127.0.0.1:8080 --model qwen3.6-27b          # run the Core-4 load battery, capture samples
tokenwatt calibrate fit ~/.tokenwatt/calibration/campaign-*.json  # → ~/.tokenwatt/profiles/<machine>__<model>.json
```

The fit prints a **measured** confidence band and an honest tier: `plug-calibrated (±2.7%)` when the
plug fit beats the estimated floor, or `uncertified (±X%) — no tighter than estimated` when it doesn't
(an uncertified fit is *not* applied — your costs stay honestly estimated). Pass several campaign files
taken at different `--cell-seconds` for a combined varying-duration fit that pins the per-time term.

Once a certified profile exists, `serve` loads it automatically: matching requests are priced from
measured wall energy, and `report` shows the plug band in its `conf` column. Recalibrate, then restart
`serve` to pick up a new profile.

## What it does

For each request it: forwards byte-exact to your local server (`mlx-openai-server`, `mlx-vlm`,
Ollama, LM Studio, llama.cpp…) so response bodies are byte-identical to hitting the backend
directly; brackets
the request with Apple IOReport per-rail SoC energy (sudoless); subtracts a rolling idle baseline;
books model-load (cold-start) energy to a separate row; classifies the request type (text / vision /
embedding) and counts tokens from the backend's own usage when available; prices it at your flat
utility rate; and logs a per-request row — with a `request_id` that ties each cost row to a
structured JSONL operations log.

It is a *meter*, not a gateway: no API key, no rewriting, no buffering. Streaming, tool calls,
sampling params, chat templates, and `response_format` all pass straight through. The one endpoint it
synthesizes rather than forwards is `GET /v1/models` — it returns the de-duplicated union of every
upstream's models (discovery, unmetered, never a ledger row).

## Honesty

- Costs read **estimated (±15–30%)** until you run the wall-meter calibration (`tokenwatt calibrate`).
  After a **certified** plug calibration for that machine+model, its rows are priced from measured wall
  energy and read the **plug-calibrated (±measured%)** tier; models you haven't calibrated stay
  estimated, and an uncertified fit (no tighter than estimated) is never applied.
- A model with no rate set shows `—`, never a fabricated `$0`; a real sub-cent cost shows `<$0.0001`,
  never a fake zero.
- `$/Mtok` for text uses completion tokens **including** reasoning/`<think>` tokens; the cloud
  comparison uses total input+output cost at dated list prices.

See the design spec and milestone plans under `docs/design/`.
