Metadata-Version: 2.4
Name: snaptokens
Version: 0.2.4
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
License-File: LICENSE
Summary: Fast BPE tokenization from local Hugging Face tokenizer.json files
Keywords: bpe,llm,tokenizer
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/coder-2011/snaptokens
Project-URL: Repository, https://github.com/coder-2011/snaptokens

<h1 align="center">snaptokens</h1>

<p align="center">
Fast BPE tokenization in Rust. <a href="https://github.com/coder-2011/snaptokens/blob/main/benchmarks/speed.md"><strong>The fastest in the current generic BPE matrix</strong></a>, in fact.
</p>

<p align="center">
  <img src="https://raw.githubusercontent.com/coder-2011/snaptokens/main/assets/benchmark-portable-overview.svg" alt="Snaptokens throughput advantage across 13 hosts">
</p>

snaptokens is an optimized fork of fastokens that is significantly quicker and simplified to work in [sinter](https://github.com/coder-2011/sinter). This repo will be archived once ported to sinter.

Give snaptokens a local Hugging Face `tokenizer.json` and get token IDs back. Opt in to a `.tkz` copy when faster future loads matter.

The current twelve-tokenizer, thirteen-host comparison is 1.89× faster than Gigatoken, 13.45× faster than upstream fastokens, and 41.41× faster than Hugging Face by geometric mean of paired medians. Snaptokens wins 778/780 Gigatoken cells; the [full report](https://github.com/coder-2011/snaptokens/blob/main/benchmarks/speed.md) keeps both losses and the Gemma sensitivity check visible.

## Optimizations

- Hand-written scanners replace general regex machinery for recognized production tokenizer patterns, with a table-driven ASCII path and exact Unicode fallback.
- Split, ByteLevel transformation, and BPE are fused so raw bytes map directly to initial token IDs and cache hits skip ByteLevel work entirely.
- BPE uses precomputed flat lookups: byte-to-token IDs, all 65,536 initial byte pairs, open-addressed merge ranks, and CSR merge adjacency.
- Pre-tokenized text stays in one contiguous buffer with byte ranges instead of allocating a string for every piece.
- Thread-local and shared caches are amortized across whole chunks, while a fixed Rayon pool keeps worker caches warm and balances uneven BPE work.
- Non-ByteLevel tokenizers split only at vocabulary-proven unbridgeable byte pairs; byte-fallback models stay on the normal exact merge path.
- Opt-in `.tkz` sidecars cache validated native construction data. Across twelve models and thirteen hosts, direct `.tkz` loads are 1.47× faster than Snaptokens JSON and artifacts are 24.9% smaller by geometric mean.

## Benchmarks

The 2026-07-31 suite uses DeepSeek R1, Gemma 3, GLM 4.7, GPT-2, GPT-OSS, Llama 3, MiniMax M2.1, Mistral Nemo, Nemotron 3, Phi-4 mini, Qwen 2.5, and Qwen 3. It runs five batch and input shapes on Apple M2 plus twelve four-core Modal placements across AWS, GCP, and OCI.

Every timed engine first passes Hugging Face token-ID parity. Gigatoken and Snaptokens cover all twelve models; upstream fastokens covers nine. Flat-ragged engines are compared only with the same flat-ragged output contract, nested engines only with nested output.

`tokenizer.json` and `.tkz` produce the same steady-state Snaptokens encoder, so the format comparison measures fresh load, first encode, disk size, and memory rather than drawing two redundant throughput bars. TKZ loads 1.47× faster than Snaptokens JSON, but is still 1.40× slower than Hugging Face JSON in this suite.

The [full benchmark report](https://github.com/coder-2011/snaptokens/blob/main/benchmarks/speed.md) contains the per-host, per-model, per-shape, load, footprint, variability, correctness, and limitation analysis. The complete [accepted evidence](https://github.com/coder-2011/snaptokens/tree/main/benchmarks/data/2026-07-31/portable) includes every raw round, hardware capture, hash, summary, and an empty failure ledger.

## Install

Install the Python package:

```bash
pip install snaptokens
```

Or add the Rust crate:

```toml
[dependencies]
snaptokens = "0.2"
```

## Use

Python:

```python
from snaptokens import Tokenizer

tokenizer = Tokenizer.from_file("/path/to/tokenizer.json")
ids = tokenizer.encode("Tokenization should not be the bottleneck.").ids
```

Pass `tkz_cache=True` to create and reuse the adjacent `.tkz` file. A `.tkz`
path loads directly without that flag.

To replace a Transformers v4 or v5 fast-tokenizer backend:

```python
import snaptokens

snaptokens.patch_transformers()
```

Call `snaptokens.unpatch_transformers()` to restore the original backend.

Rust:

```rust
use std::{error::Error, path::Path};

use snaptokens::Tokenizer;

/// Loads one local tokenizer and encodes a prompt.
fn main() -> Result<(), Box<dyn Error>> {
    let tokenizer = Tokenizer::from_file(Path::new("/path/to/tokenizer.json"))?;
    let ids = tokenizer.encode("Tokenization should not be the bottleneck.")?;

    println!("{ids:?}");
    Ok(())
}
```

Download the model yourself and pass the local JSON path. snaptokens never tries to infer or download a Hugging Face model.

### Optional `.tkz` cache

```rust
let tokenizer = Tokenizer::from_file_with_tkz_cache(
    Path::new("/path/to/tokenizer.json"),
    true,
)?;
```

The first enabled load atomically writes `tokenizer.tkz`; later loads validate and reuse it. Passing that `.tkz` path loads it directly without creating another copy. `Tokenizer::from_file` remains JSON-only and never writes a cache.

## Scope

Snaptokens is only built for inference on BPE tokenizers, and to do just that, well.

Real-tokenizer validation covers GPT-2, GPT-OSS, DeepSeek, MiniMax, Qwen and Qwen3.5, GLM, Nemotron, Mistral, Gemma 4, Phi-4, and Kimi K2.5 families, including added-token flags and byte fallback where present.

## Credits

Licensed under [Apache-2.0](https://github.com/coder-2011/snaptokens/blob/main/LICENSE).

