Metadata-Version: 2.5
Name: indic-bpe
Version: 0.0.1
Summary: A lightweight BPE tokenizer for Indic languages, starting with Devanagari.
Author: Raktim Kalita
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: streamlit>=1.61.1
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/favicon.png" alt="Indic-BPE logo" width="110">
</p>

<h1 align="center">Indic-BPE</h1>

<p align="center">
  A lightweight Byte Pair Encoding (BPE) tokenizer built from scratch for
  <strong>Hindi written in Devanagari</strong>.
</p>

<p align="center">
  Simple implementation • Efficient BPE training • Small Python API
</p>

## Features

- Hindi / Devanagari BPE tokenizer
- Train BPE merges from a text corpus
- Heap-based most-frequent pair selection
- Linked-list based merge operations
- Incremental local pair updates during training
- Encode and decode support
- Vocabulary and merge serialization
- Simple Python package API
- Streamlit tokenizer playground
- Automated test suite
- Training and tokenizer benchmarks

## Architecture

<p align="center">
  <img src="assets/Diagram.png" alt="Indic-BPE Architecture" width="850">
</p>

## Installation

### From PyPI

```bash
pip install indic-bpe
```

### Development Installation

```bash
git clone https://github.com/Rktim/indic-bpe.git
cd indic-bpe
uv sync
```

## Quick Start

```python
from indic_bpe import BPETokenizer

tokenizer = BPETokenizer()

tokenizer.load(
    "corpus/processed/hindi_vocab.json",
    "corpus/processed/hindi_merges.json",
)

text = "यह एक हिंदी भाषा का परीक्षण है।"

tokens = tokenizer.encode(text)
print(tokens)

decoded = tokenizer.decode(tokens)
print(decoded)
```

Example:

```text
Tokens:
['य', 'ह ', 'ए', 'क ', 'ह', 'ि', 'ंद', 'ी ', 'भ', 'ा', 'ष', 'ा क', 'ा ',
 'प', 'र', 'ी', 'क्ष', 'ण', ' ', 'है', '।']

Decoded:
यह एक हिंदी भाषा का परीक्षण है।
```

## Training

A tokenizer can be trained directly from a Hindi text corpus:

```python
from indic_bpe import BPETokenizer

tokenizer = BPETokenizer()

tokenizer.train(
    loader,
    num_merges=1000,
)
```

The trainer repeatedly selects the most frequent adjacent symbol pair and merges it into a new token.

The implementation maintains pair statistics incrementally rather than rebuilding all pair frequencies after every merge.

## Save and Load

Save a trained tokenizer:

```python
tokenizer.save(
    "hindi_vocab.json",
    "hindi_merges.json",
)
```

Load it later:

```python
tokenizer.load(
    "hindi_vocab.json",
    "hindi_merges.json",
)
```

The tokenizer stores:

- Vocabulary in `vocab.json`
- Merge rules in `merges.json`

## API

### `train()`

```python
tokenizer.train(loader, num_merges=1000)
```

Train the BPE tokenizer from a corpus.

### `encode()`

```python
tokens = tokenizer.encode(text)
```

Convert Hindi text into BPE tokens.

### `decode()`

```python
text = tokenizer.decode(tokens)
```

Convert tokens back into the original text.

### `save()`

```python
tokenizer.save(vocab_path, merges_path)
```

Save the vocabulary and merge rules.

### `load()`

```python
tokenizer.load(vocab_path, merges_path)
```

Load a previously trained tokenizer.

## Tokenizer Evaluation

Example:

```text
यह एक हिंदी भाषा का परीक्षण है।
```

Current evaluation:

```text
Characters: 31
Tokens: 21
```

Example tokens:

```text
य | ह  | ए | क  | ह | ि | ंद | ी  | भ | ा | ष | ा क | ा  | प | र | ी | क्ष | ण |   | है | ।
```

The evaluation verifies that:

```python
decoded == original_text
```

## Streamlit Demo

Indic-BPE includes a small Hindi tokenizer playground.

Run:

```bash
uv run streamlit run scripts/tokenizer_app.py
```

The demo provides:

- Hindi text input
- Character count
- Token count
- Colored token visualization
- Token text view
- Token ID view
- Token breakdown

## Package Validation

The `0.0.1` package has been validated by:

1. Building a wheel and source distribution.
2. Installing the wheel into a clean virtual environment.
3. Importing `BPETokenizer` from the installed package.
4. Running a Hindi encode/decode round-trip.
5. Loading trained Hindi vocabulary and merge files.
6. Running a trained-tokenizer Hindi round-trip.

Example:

```python
from indic_bpe import BPETokenizer

tokenizer = BPETokenizer()

text = "यह एक हिंदी भाषा का परीक्षण है।"

tokens = tokenizer.encode(text)
decoded = tokenizer.decode(tokens)

assert decoded == text
```

## Project Structure

```text
indic-bpe/
├── indic_bpe/
│   ├── __init__.py
│   ├── corpus.py
│   ├── decoder.py
│   ├── encoder.py
│   ├── merges.py
│   ├── serialization.py
│   ├── tokenizer.py
│   ├── trainer.py
│   ├── utils.py
│   ├── version.py
│   └── vocabulary.py
│
├── corpus/
│   ├── raw/
│   └── processed/
│
├── scripts/
│   ├── benchmark_tokenizer.py
│   ├── evaluate_tokenizer.py
│   └── tokenizer_app.py
│
├── tests/
├── assets/
├── README.md
├── LICENSE
├── pyproject.toml
└── .gitignore
```

## Current Status

**Version:** `0.0.1`

Indic-BPE `0.0.1` is an early release focused on Hindi / Devanagari BPE tokenization.

Completed:

- BPE trainer
- Optimized pair tracking
- Encoder
- Decoder
- Vocabulary handling
- Serialization
- Tokenizer API
- Test suite
- Benchmark scripts
- Streamlit demo
- Python package build
- Clean wheel installation validation

## License

MIT License. See [LICENSE](LICENSE).
