Metadata-Version: 2.4
Name: tokz
Version: 0.2.0
Summary: Deterministic context compression with adaptive budgets, provenance, preserve rules, batch calls, and agent integrations.
Project-URL: Homepage, https://tokz.dev
Project-URL: Documentation, https://docs.tokz.dev
Project-URL: Source, https://github.com/tokz-dev/platform
Author: tokz
License: MIT
Keywords: agents,compression,context,llm,prompt,rag,tokens
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: httpx<1,>=0.24
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Description-Content-Type: text/markdown

# tokz

Deterministic context compression with byte offsets, extractive provenance,
adaptive budgets, hard preserve rules, ordered batch calls, and agent adapters.

Hosted mode sends the payload to Tokz, where it is never stored or logged.
Structural responses contain byte ranges and semantic responses contain extractive
text with provenance. Nothing is paraphrased or generated.

```bash
pip install tokz
```

## Use the resilient call, not the raw one

```python
import os
from tokz import Tokz, CircuitBreaker, CompressionCache, resilient_compress

tokz = Tokz(api_key=os.environ["TOKZ_API_KEY"])

# One of each per process, not per call — both carry the state that is the point.
breaker = CircuitBreaker()
cache = CompressionCache()

text, compressed = resilient_compress(
    tokz,
    breaker,
    payload,
    cache=cache,
    on_error=lambda info: print("tokz fell back:", info["message"]),
    on_credits_exhausted=lambda cents: alert(f"tokz balance {cents}c"),
)
# `compressed` is False when this is your original payload coming back.
```

`Tokz.compress()` **raises** on failure — it is a plain client, so a fault surfaces
rather than hiding. `resilient_compress()` never raises and never lets a failure
reach your LLM call. Reach for the second one unless you have a specific reason not
to.

### What it does that a `try`/`except` does not

- **Bounds an outage.** A bare `except` pays the full timeout on *every* payload.
  The breaker opens after five consecutive failures and admits one trial per reset
  window, so a dead API costs one timeout instead of N.
- **Serves the cache first.** The cache is checked *before* the breaker, so a
  payload you have already compressed still returns its real result while the API
  is down — better than falling back to text you have an answer for.
- **Separates billing from outage.** HTTP 402 will not succeed on retry and does
  not mean the API is down, so it routes to `on_credits_exhausted` and leaves the
  breaker closed. Tripping it would skip compression for a whole reset window over
  an empty wallet.
- **Cannot be broken by your logger.** `on_error` is best-effort; a handler that
  raises will not turn a swallowed compression failure into a failed request.

## Async

```python
from tokz import AsyncTokz, CircuitBreaker, aresilient_compress

async with AsyncTokz(api_key=...) as tokz:
    text, compressed = await aresilient_compress(tokz, breaker, payload)
```

Same contract, same retry policy, same fallback.

## Getting dropped text back

Compression is not a one-way door. Every elision carries its position and length,
so a dropped run stays addressable — locally, with no second API call and no key.

```python
from tokz import expand

result = tokz.compress(payload, target_ratio=0.4)
dropped = expand(payload, result["spanMap"], elision=0)
```

The source hash is verified first, so a span map can never be applied to text it
was not built from.

## Offsets are UTF-8 bytes

Not string indices. The two agree only for pure ASCII, and this SDK handles the
conversion for you — but if you slice a span map yourself, encode first:

```python
src = payload.encode("utf-8")
kept = b"".join(src[s["s"]:s["e"]] for s in span_map["spans"]).decode("utf-8")
```

Doing it on the `str` corrupts any payload containing an accent, a CJK character or
an emoji, and does it quietly.

## When it is not worth using

Compression pays above roughly $0.18 per million input tokens on uncached payload.
On cheap models, or on context your prompt cache already holds, the fee exceeds the
saving. The estimator at <https://app.tokz.dev/savings> will tell you which side of
that line you are on, including when the answer is no.

## Links

- [tokz.dev](https://tokz.dev)
- [Documentation](https://docs.tokz.dev)
- [TypeScript SDK](https://www.npmjs.com/package/@tokz/sdk)

MIT
