Metadata-Version: 2.4
Name: memchunk
Version: 0.2.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing
Summary: The fastest semantic text chunking library
Keywords: chunking,text,simd,nlp,tokenization,rag
Author: Bhavnick Minhas
License: MIT OR Apache-2.0
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/chonkie-inc/memchunk
Project-URL: Repository, https://github.com/chonkie-inc/memchunk

<p align="center">
  <img src="../../assets/memchunk_wide.png" alt="memchunk" width="500">
</p>

<h1 align="center">memchunk</h1>

<p align="center">
  <em>the fastest text chunking library — up to 1 TB/s throughput</em>
</p>

<p align="center">
  <a href="https://crates.io/crates/memchunk"><img src="https://img.shields.io/crates/v/memchunk.svg?color=e74c3c" alt="crates.io"></a>
  <a href="https://pypi.org/project/memchunk"><img src="https://img.shields.io/pypi/v/memchunk.svg?color=e67e22" alt="PyPI"></a>
  <a href="https://www.npmjs.com/package/memchunk"><img src="https://img.shields.io/npm/v/memchunk.svg?color=2ecc71" alt="npm"></a>
  <a href="https://github.com/chonkie-inc/memchunk"><img src="https://img.shields.io/badge/github-memchunk-3498db" alt="GitHub"></a>
  <a href="LICENSE-MIT"><img src="https://img.shields.io/badge/license-MIT%2FApache--2.0-9b59b6.svg" alt="License"></a>
</p>

---

you know how every chunking library claims to be fast? yeah, we actually meant it.

**memchunk** splits text at semantic boundaries (periods, newlines, the usual suspects) and does it stupid fast. we're talking "chunk the entire english wikipedia in 120ms" fast.

want to know how? [read the blog post](https://minha.sh/posts/so,-you-want-to-chunk-really-fast) where we nerd out about SIMD instructions and lookup tables.

## 📦 installation

```bash
pip install memchunk
```

looking for [rust](https://github.com/chonkie-inc/memchunk) or [javascript](https://github.com/chonkie-inc/memchunk/tree/main/packages/wasm)?

## 🚀 usage

```python
from memchunk import chunk

text = "Hello world. How are you? I'm fine.\nThanks for asking."

# with defaults (4KB chunks, split at \n . ?)
for slice in chunk(text):
    print(bytes(slice))

# with custom size
for slice in chunk(text, size=1024):
    print(bytes(slice))

# with custom delimiters
for slice in chunk(text, delimiters=".?!\n"):
    print(bytes(slice))

# with both
for slice in chunk(text, size=8192, delimiters="\n"):
    print(bytes(slice))

# collect all chunks
chunks = list(chunk(text))
```

chunks are returned as `memoryview` objects (zero-copy slices of the original text).

## 📝 citation

if you use memchunk in your research, please cite it as follows:

```bibtex
@software{memchunk2025,
  author = {Minhas, Bhavnick},
  title = {memchunk: The fastest text chunking library},
  year = {2025},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/chonkie-inc/memchunk}},
}
```

## 📄 license

licensed under either of [Apache License, Version 2.0](LICENSE-APACHE) or [MIT license](LICENSE-MIT) at your option.

