Metadata-Version: 2.4
Name: ms3p-sushima-tokenizer
Version: 3.0.0
Summary: MS3P Sushima byte-level BPE tokenizer engine with multilingual synthetic corpus generation.
Home-page: https://github.com/smcgandco/ms3p_sushima_engine_tokenizer
Author: SUJIT MAITY CONSULTING GROUP & CORPORATION
Author-email: SUJIT MAITY CONSULTING GROUP & CORPORATION <smcgandco@gmail.com>
Maintainer: Sujit Shibaprasad Maity
Maintainer-email: Sujit Shibaprasad Maity <sujitmaity.in@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/smcgandco/ms3p_sushima_engine_tokenizer
Project-URL: Repository, https://github.com/smcgandco/ms3p_sushima_engine_tokenizer.git
Project-URL: Issues, https://github.com/smcgandco/ms3p_sushima_engine_tokenizer/issues
Project-URL: Documentation, https://github.com/smcgandco/ms3p_sushima_engine_tokenizer#readme
Keywords: tokenizer,bpe,byte-pair-encoding,byte-level,nlp,language-model,llm,tokenization,multilingual,dataset-generator,artificial-intelligence,machine-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Education
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: msgpack
Requires-Dist: msgpack>=1.0; extra == "msgpack"
Provides-Extra: cachetools
Requires-Dist: cachetools>=5.3; extra == "cachetools"
Provides-Extra: all
Requires-Dist: msgpack>=1.0; extra == "all"
Requires-Dist: cachetools>=5.3; extra == "all"
Provides-Extra: dev
Requires-Dist: black>=24.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.1; extra == "dev"
Dynamic: author
Dynamic: home-page
Dynamic: maintainer
Dynamic: requires-python

# MS3P Sushima Tokenizer

A byte-level Byte Pair Encoding (BPE) tokenizer engine with heap-based training and encoding, plus a synthetic multilingual dataset generator for training it from scratch.

- **Package**: `ms3p-sushima-tokenizer`
- **Import name**: `ms3p_sushima_tokenizer`
- **License**: MIT
- **Python**: 3.10+
- **Author**: SUJIT MAITY CONSULTING GROUP & CORPORATION
- **Maintainer**: Sujit Shibaprasad Maity

---

## Features

- **Byte-level BPE** — operates directly on UTF-8 bytes, so every input string is representable with zero out-of-vocabulary loss (worst case, a token falls back to raw bytes).
- **Heap-based training** — pair frequencies are tracked incrementally with a max-heap instead of rescanning the whole corpus on every merge, with periodic heap-rebuilds to clear out stale entries.
- **Heap-based encoding** — `O(n log n)` merge application per word via a linked-list + priority-heap walk over trained merge ranks, instead of a naive `O(n²)` rescan.
- **GPT-2-style pre-tokenizer** — regex splits on contractions (`'s`, `'t`, `'re`, ...; both straight and curly apostrophes), digit runs, word runs, punctuation runs, and whitespace.
- **NFKC normalization** applied before tokenization for consistent handling of visually-equivalent Unicode sequences.
- **LRU word cache** — repeated words skip BPE merging entirely. Uses `cachetools.LRUCache` if installed, otherwise falls back to a built-in `SimpleLRU`.
- **Deterministic, multiprocessing-capable corpus generator** covering 30+ languages plus numbers, code tokens, URLs, emails, symbols, and emojis.
- **Portable model files** — saved with `msgpack` (compact, binary-safe) or a JSON fallback (hex-encoded), written atomically via a temp-file + `os.replace` swap so a crash mid-save never corrupts the model on disk.
- **No hard dependencies** — `msgpack` and `cachetools` are both optional; the tokenizer degrades gracefully without them.

---

## Installation

```bash
# Core package only
pip install ms3p-sushima-tokenizer

# With compact msgpack model saving
pip install "ms3p-sushima-tokenizer[msgpack]"

# With faster LRU caching
pip install "ms3p-sushima-tokenizer[cachetools]"

# Everything
pip install "ms3p-sushima-tokenizer[all]"

# From source, for development
git clone https://github.com/smcgandco/ms3p_sushima_engine_tokenizer.git
cd ms3p_sushima_engine_tokenizer
pip install -e ".[dev]"
```

---

## Quick Start

### 1. Train a tokenizer from a text file

```python
from ms3p_sushima_tokenizer import MS3P_sushima_tokenizer_v3

tokenizer = MS3P_sushima_tokenizer_v3(vocab_size=32000)
tokenizer.train("corpus.txt", min_pair_freq=2)

tokenizer.save("tokenizer.ms3p")  # msgpack by default
```

### 2. Load and use it

```python
from ms3p_sushima_tokenizer import MS3P_sushima_tokenizer_v3

tokenizer = MS3P_sushima_tokenizer_v3(vocab_size=32000)
tokenizer.load("tokenizer.ms3p")

ids = tokenizer.encode("Byte-pair encoding is deceptively simple.")
text = tokenizer.decode(ids)

print(ids)
print(text)
print(tokenizer.vocabulary_coverage("some unfamiliar text"))
```

### 3. Batch encode/decode with padding

```python
batch = tokenizer.encode_batch(
    ["hello world", "a much longer sentence to pad against"],
    padding=True,
    truncation=False,
)

print(batch["input_ids"])
print(batch["attention_mask"])

decoded = tokenizer.decode_batch(batch["input_ids"])
```

### 4. Train from pre-counted word frequencies

Useful if you already have a word-frequency table (e.g. from a large corpus processed elsewhere) and want to skip re-reading the raw file:

```python
from collections import Counter

word_counts = Counter({"the": 50000, "quick": 1200, "fox": 300})
tokenizer.train_from_word_counts(word_counts, num_merges=5000)
```

---

## Generating a Training Corpus

The package ships a synthetic multilingual corpus generator, exposed both as a library function and a CLI entry point.

### CLI

```bash
ms3p-generate-corpus \
    --num-lines 100000 \
    --min-words 5 \
    --max-words 50 \
    --output corpus.txt \
    --seed 42 \
    --workers 4 \
    --stats-file corpus_stats.json
```

| Flag | Default | Description |
|---|---|---|
| `--num-lines` | `100` | Number of lines to generate |
| `--min-words` / `--max-words` | `5` / `50` | Words per line range |
| `--output` | `ms3p_sushima_tokenizer/ms3p_sushima_corpus_center/ms3p_corpus_raw.txt` | Output file path |
| `--seed` | `42` | Random seed (generation is fully deterministic for a given seed + config) |
| `--workers` | `1` | Worker processes (chunks are generated in parallel, written back in order) |
| `--chunk-size` | `5000` | Lines per multiprocessing chunk |
| `--buffer-size` | `10000` | Lines buffered before a disk write |
| `--stats-file` | `None` | Optional path to write a JSON generation report |
| `--languages-file` | `None` | Optional custom JSON language spec (see below) |
| `--no-overwrite` | off | Fail instead of overwriting an existing output file |
| `--log-level` | `INFO` | `DEBUG` / `INFO` / `WARNING` / `ERROR` |

### Library usage

```python
from ms3p_sushima_tokenizer.ms3p_sushima_data_center.ms3p_corpus_generate import (
    GeneratorConfig,
    generate_dataset,
)

config = GeneratorConfig(num_lines=50_000, seed=7, num_workers=4)
stats = generate_dataset(config)
print(stats["lines_written"], stats["lines_per_second"])
```

### Token composition

Each generated line samples from eight token categories according to a configurable distribution (defaults shown):

| Category | Default weight | Example |
|---|---|---|
| `multilingual` | 0.60 | words from 30+ languages, weighted toward English |
| `random_word` | 0.15 | random lowercase ASCII strings |
| `number` | 0.10 | integers 0–1,000,000 |
| `code` | 0.07 | `def`, `class`, `return`, `import`, ... |
| `url` | 0.04 | `https://www.example.com` style |
| `email` | 0.02 | `user123@example.com` style |
| `symbol` | 0.01 | punctuation/symbol clusters |
| `emoji` | 0.01 | 😀 🔥 🚀 💡 ✅ 📊 🎯 |

Languages covered out of the box: English, Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Punjabi, Malayalam, Kannada, Chinese, Japanese, Korean, Arabic, Hebrew, Persian, Spanish, French, German, Italian, Portuguese, Dutch, Russian, Ukrainian, Polish, Czech, Thai, Vietnamese, Indonesian, Malay, Swahili, Yoruba, Greek, and Turkish.

You can override the language mix entirely with `--languages-file path/to/languages.json`, where each entry is `"language": [["word1", "word2", ...], weight]`.

---

## How It Works

### Training

1. The corpus is pre-tokenized (regex split + NFKC normalization) into a word-frequency table.
2. Each word is exploded into its individual UTF-8 bytes.
3. Adjacent byte/token pairs are counted across the whole corpus, weighted by word frequency.
4. A max-heap repeatedly pops the most frequent pair, merges it into a new token, and incrementally updates only the pairs affected by that merge (not the whole corpus) — this is what keeps training subquadratic.
5. Merges continue until either the configured `vocab_size` is reached, the pair frequency drops below `min_pair_freq`, or no mergeable pairs remain.
6. Pairs that would exceed `max_token_length` are dropped and permanently excluded from further consideration, so they don't leak memory or skew future frequency deltas.

### Encoding

Applying merges to a new word uses the same heap-based approach as training, but over a fixed set of trained merge ranks:

1. Each byte becomes a node in a doubly linked list.
2. Every currently-adjacent pair with a known merge rank is pushed onto a min-heap.
3. The lowest-rank (highest-priority) pair is popped, validated as still-adjacent and unchanged, merged, and the list is spliced.
4. Newly-adjacent pairs created by the merge are pushed onto the heap.
5. This continues until the heap is empty, yielding `O(n log n)` merge application instead of a naive `O(n²)` full rescan per word.

Encoded words are cached (LRU), so repeated tokens in a corpus only pay the merge cost once.

### Special tokens

| Token | ID |
|---|---|
| `<PAD>` | 0 |
| `<UNK>` | 1 |
| `<BOS>` | 2 |
| `<EOS>` | 3 |
| `<MASK>` | 4 |
| `<CLS>` | 5 |
| `<SEP>` | 6 |

`vocab_size` is the **total** addressable id space (specials + 256 base byte tokens + learned merges), so a minimum of `263` is enforced at construction time, and training will never exceed the configured budget.

---

## API Reference

### `MS3P_sushima_tokenizer_v3(vocab_size=50000, max_token_length=64, cache_size=100000, word_cache_size=200000)`

| Method | Description |
|---|---|
| `.train(file_path, num_merges=None, min_pair_freq=2, progress_every=100)` | Train from a raw text file |
| `.train_from_word_counts(word_counter, num_merges=None, min_pair_freq=2, progress_every=100)` | Train from a pre-built `{word: frequency}` dict |
| `.encode(text, add_special_tokens=True) -> List[int]` | Encode a string to token ids |
| `.decode(token_ids, skip_special_tokens=True) -> str` | Decode token ids back to a string |
| `.encode_batch(texts, padding=True, max_length=None, truncation=False) -> dict` | Encode a list of strings with padding + attention masks |
| `.decode_batch(batch_ids, skip_special_tokens=True) -> List[str]` | Decode a batch of id sequences |
| `.sequence_length(text) -> int` | Token count for a string |
| `.vocabulary_coverage(text) -> float` | Fraction of tokens that aren't `<UNK>` |
| `.token_to_id(token: bytes) -> Optional[int]` | Look up a byte token's id |
| `.id_to_token(token_id: int) -> Optional[bytes]` | Reverse lookup |
| `.get_special_token_id(name: str) -> Optional[int]` | e.g. `get_special_token_id("<BOS>")` |
| `.save(path, use_msgpack=True)` | Atomically persist the trained model |
| `.load(path)` | Load a previously saved model |
| `.validate() -> bool` | Sanity-check vocab/merge-rank consistency |
| `.stats() -> dict` | Vocab size, merge count, cache hit rate, etc. |
| `.get_vocab_size() -> int` | Total addressable ids (specials + bytes + merges) |
| `.clear_caches()` | Reset the word cache and hit/miss counters |

`MS3P_sushima_tokenizer_v1` and `_v2` expose the same API and are functionally equivalent to `_v3` — they're retained for compatibility with models/pipelines pinned to an earlier engine version. New projects should use `_v3`.

---

## Project Structure

```
ms3p_sushima_engine_tokenizer/
├── pyproject.toml
├── README.md
├── LICENSE
└── ms3p_sushima_tokenizer/
    ├── __init__.py
    ├── ms3p_sushima_data_center/
    │   └── ms3p_corpus_generate.py      # synthetic corpus generator + CLI
    ├── ms3p_sushima_corpus_center/      # generated .txt corpora land here
    ├── ms3p_sushima_engine/
    │   ├── ms3p_token_engine_v1.py
    │   ├── ms3p_token_engine_v2.py
    │   └── ms3p_token_engine_v3.py      # current tokenizer implementation
    └── ms3p_sushima_model/              # trained .ms3p model files land here
```

---

## Development

```bash
pip install -e ".[dev]"

pytest                 # run tests
black .                # format
ruff check .           # lint
mypy ms3p_sushima_tokenizer  # type-check
```

---

## Model File Format

`.ms3p` files start with an 8-byte magic header (`MSGPACK1` or `JSON0001`) followed by the serialized payload: schema version, tokenizer config, special tokens, vocabulary, and ordered merge rules. `msgpack` is preferred for size and native binary support; the JSON fallback hex-encodes byte tokens for text-safety. Files saved by one format are only loadable if the corresponding library is installed (`msgpack` is optional) — save with `use_msgpack=False` if you need a format with no extra dependency.

---

## License

MIT — see [`LICENSE`](./LICENSE).

## Links

- **Repository**: https://github.com/smcgandco/ms3p_sushima_engine_tokenizer
- **Issues**: https://github.com/smcgandco/ms3p_sushima_engine_tokenizer/issues
