Metadata-Version: 2.4
Name: tokentik
Version: 0.1.0
Summary: A simple utility to estimate token counts using tiktoken o200k_base
License: MIT
Project-URL: Homepage, https://github.com/speech2srt/tokentik
Project-URL: Repository, https://github.com/speech2srt/tokentik
Project-URL: Issues, https://github.com/speech2srt/tokentik/issues
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.7.0
Dynamic: license-file

# Tokentik

A lightweight Python utility for estimating token counts in text, specifically optimized for modern LLMs using the `o200k_base` encoding.

## Installation

```bash
pip install tokentik
```

## Usage

```python
from tokentik import count_tokens

text = "Hello, world!"
token_count = count_tokens(text)
print(f"Token count: {token_count}")
```

## Configuration

### Environment Variables

`tiktoken` needs to download and cache the BPE (Byte Pair Encoding) vocabulary files. By default, it uses a temporary directory. To specify a persistent location for these files, set the `TIKTOKEN_CACHE_DIR` environment variable:

```bash
export TIKTOKEN_CACHE_DIR="/path/to/your/models/tiktoken"
```

This is highly recommended for production environments or Cloud Run environments where the storage might be mounted (e.g., at `/mnt/models/tiktoken`).

## License

MIT
