Metadata-Version: 2.1
Name: rs_bytepiece
Version: 0.2.2
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
License-File: LICENSE
Summary: bytepiece-rs Python binding
Keywords: NLP,tokenizer,bytepiece,Deep Learning
Author: Yam(长琴) <haoshaochun@gmail.com>
Author-email: Yam(长琴) <haoshaochun@gmail.com>
License: MIT
Requires-Python: >=3.7
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: homepage, https://github.com/hscspring/bytepiece-rs
Project-URL: documentation, https://github.com/hscspring/bytepiece-rs
Project-URL: repository, https://github.com/hscspring/bytepiece-rs

# rs-bytepiece

## Install

```bash
pip install rs_bytepiece
```

## Usage

```python
from rs_bytepiece import Tokenizer

tokenizer = Tokenizer()
# a custom model
tokenizer = Tokenizer("/path/to/model")
ids = tokenizer.encode("今天天气不错")
text = tokenizer.decode(ids)
```

## Performance

The performance is a bit faster than the original implementation. I've tested (on my M2 16G) the《鲁迅全集》which has 625890 chars. The time unit is millisecond.

| length | jieba    | aho_py  | aho_cy | aho_rs |
| ------ | -------- | ------- | ------ | ------ |
| 100    | 17062.12 | 1404.37 | 564.31 | 112.94 |
| 1000   | 17104.38 | 1424.6  | 573.32 | 113.18 |
| 10000  | 17432.58 | 1429.0  | 574.93 | 110.03 |
| 100000 | 17228.17 | 1401.01 | 574.5  | 110.44 |
| 625890 | 17305.95 | 1419.79 | 567.78 | 108.54  |


