Metadata-Version: 2.4
Name: token-assay
Version: 0.1.0
Summary: Unit standardization for token price indices: the Standard Token Equivalent (STE)
Project-URL: Homepage, https://github.com/tpypan/assay
Project-URL: Repository, https://github.com/tpypan/assay
Project-URL: Issues, https://github.com/tpypan/assay/issues
Project-URL: Methodology, https://github.com/tpypan/assay/blob/main/docs/METHODOLOGY.md
Project-URL: Dataset, https://github.com/tpypan/assay/tree/main/results
Author: Tony Pan
License: MIT License
        
        Copyright (c) 2026 Tony Pan
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: benchmark,llm,normalization,price-index,pricing,tokenizer,tokens
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: anthropic>=0.40
Requires-Dist: httpx>=0.27
Requires-Dist: huggingface-hub>=0.26
Requires-Dist: platformdirs>=4.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: tiktoken>=0.8
Requires-Dist: tokenizers>=0.20
Requires-Dist: typer>=0.12
Description-Content-Type: text/markdown

# assay

**Unit standardization for token price indices.**

Inference is priced in dollars per million tokens, but a *token* is not a standard
unit — each provider's tokenizer segments identical text into a different number of
them. Two providers with the same headline `$/1M tokens` can bill materially
different amounts for the same work, and any index aggregating across providers is
summing incommensurable quantities.

`assay` measures how each tokenizer segments a fixed, versioned reference corpus and
emits normalization factors — with confidence intervals — that convert
provider-specific counts to a common basis: the **Standard Token Equivalent (STE)**.
That makes `$/1M tokens` comparable.

It measures the *denominator* of token pricing only, and needs no paid inference:
tokenizers run locally or via free counting endpoints. It does not evaluate model
quality — conflating the two produces bad measurements of both.

## Install

```sh
uvx --from token-assay assay factors   # zero-install: the published table, now
pip install token-assay                # or: uv add token-assay
```

The PyPI distribution is `token-assay`; the import and the command are `assay` — hence
the `--from`.

## Findings

[`docs/DRIFT-LEDGER.md`](docs/DRIFT-LEDGER.md) records every token-unit redefinition
found across seven model families, paired with the list price either side of it,
measured on the reference corpus with a 95% CI on every figure.

**Three redefinitions, in three of the largest families**, changing the cost of
identical work by up to 55% (OpenAI), 34% (Anthropic — 41% on conversational
English), and 40% (Meta), depending on content type.

**In two of them the price per token did not move at all**, so a dollars-per-token
index reports perfect stability across the entire change. That blindness is
algebraic, not statistical: the redefinition scales numerator and denominator
together, so no amount of transaction data recovers it.

**Four families were stable across every generation measured**, and the two redefining
families that span more than one boundary held their unit constant across the others —
which rules out the "tokenizers just drift" reading. Redefinition is a discrete,
datable event.

## Use

```sh
# The published table, from data shipped in the package — no keys, no network.
assay factors                        # STE vector: per-class + per-segment, with CIs
assay factors --weights mine.toml    # substitute your own workload mix
assay compare --price anthropic/claude-opus-5=5/25 --price openai/gpt-5.4=2.50/15

# Recompute every published figure offline from the committed artifact.
assay report --counts results/counts-corpus-v0.1.jsonl.gz
assay report --counts results/counts-corpus-v0.1.jsonl.gz --export results/

# Reproduce the corpus and raw counts (free credentials; see .env.example).
assay corpus fetch && assay measure --out counts.jsonl.gz

# Diagnostics. These re-measure live, so they need network — and, for Anthropic and
# Google, a key. Without one those rows report `unmeasured` rather than guessing;
# the offline path to the same figures is `assay report --counts` above.
assay ledger                # redefinitions and their cost effect
assay sweep --provider anthropic
assay canary --check        # hosted-tokenizer drift; exit 3 = drift
```

## Provenance is a first-class output

Every count records where it came from, and figures are not interchangeable across
those sources:

| Provenance | Meaning | Publishable |
| --- | --- | --- |
| `official/local` | Vendor vocabulary, evaluated offline | yes |
| `official/hub` | Vendor repository, pinned to a commit | yes |
| `official/api` | Vendor's own counting endpoint | yes |
| `mirror/approx` | Third-party redistribution | **no** |

Mirrors are never substituted silently — a gated repository is a recorded skip, and
counts from one are refused by the publication gate rather than by convention.

Local vocabularies are **pinned**: the exact bytes are re-fetchable and verified by
content digest. Hosted endpoints can only be **stamped** with a model id and a
timestamp, because the vendor does not publish the vocabulary and can revise it
under a stable name. For those, `assay canary` is the only integrity control there
is, and it runs weekly in CI against a committed baseline. Its probe set is four short
texts covering three scripts, so it is a lower bound on detection, not a guarantee —
`METHODOLOGY.md` §7.2 states what it would miss and why widening it is a versioned
migration rather than an edit.

## Scale

24 tokenizers · 7 families · `corpus-v0.1` = 1,740 documents, 16 languages — 1,500 of
them across the five published workload segments, 240 in a diagnostic segment that
carries no published figure · 39,120 counts, zero degradations.

The count is not a full matrix and does not claim to be: the 13 local tokenizers counted
all 1,740 documents, the 11 hosted ones only the 1,500 that figures are drawn from.

The corpus ships as manifest-plus-fetcher — pins, hashes and a retrieval script
rather than third-party text. Assembled locally it is CC BY-SA 4.0 (FLORES+ requires
share-alike); the measurement outputs are CC0; the code is MIT.

## Development

```sh
uv sync
cp .env.example .env        # every endpoint used here is free
uv run pytest               # offline; needs no credentials
uv run pytest -m live       # hits real endpoints; needs keys
uv run ruff check src tests && uv run mypy
```

`ANTHROPIC_API_KEY` is required for Claude counts — Anthropic publishes no
tokenizer, so `count_tokens` is the only admissible source. `GEMINI_API_KEY` and
`HF_TOKEN` widen coverage. OpenAI needs no key: `tiktoken` runs locally and is exact.

## License

MIT
