Metadata-Version: 2.4
Name: khmer_segmenter
Version: 0.1.0
Summary: The package is for segment khmer words (adding space between words) with two method: compound-based and morpheme-based.
Author-email: Natt Korat <natt.korat@cadt.edu.kh>
Project-URL: Homepage, https://github.com/nattkorat/khmer_segmenter.git
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: tqdm>=4.67.3
Requires-Dist: scipy>=1.15.3
Requires-Dist: scikit-learn>=1.7.2
Requires-Dist: sklearn-crfsuite>=0.5.0

# Khmer Segmenter

A lightweight Khmer word segmentation package for `Python ≥ 3.10`, designed for simple and efficient tokenization.

This project is adapted and simplified from the original [khnlp](https://pypi.org/project/khnlp/) package, with the following goals:

- Support modern Python versions (≥ 3.10)
- Simplified installation
- Focus on a single task: Khmer word segmentation
- Lightweight and easy to integrate into NLP pipelines

This package is intended as a small academic contribution to support Khmer NLP research and practical applications.

## Installation

```bash
pip install khmer-segmenter
```

Requires:
- Python >= 3.10

## Usage
```python
from khmer_segmenter import Tokenizer

tokenizer = Tokenizer(seg_type="com")

print(tokenizer.tokenize("សួស្ដីអ្នកទាំងអស់យើង"))
```

## Segmentation Modes

The tokenizer supports two segmentation strategies:

### 1. Compound-based (`seg_type="com"`)

- Segments text into compound words
- Suitable for general word-level NLP tasks
- Recommended for downstream applications such as:
    - Text classification
    - Named Entity Recognition
    - Information retrieval


### 2. Morpheme-based (`seg_type="mor"`)

- Performs finer-grained segmentation
- Splits text into smaller morphological units
- Useful for:
    - Linguistic analysis
    - Subword modeling
    - Research-focused NLP tasks


## Motivation

Khmer is a low-resource language with no explicit word boundary markers (spaces are not consistently used to separate words). This creates challenges for:
- Automatic Speech Recognition (ASR)
- Language Modeling
- Machine Translation
- Information Extraction

Existing tools for Khmer segmentation often have:
- Limited Python version support
- Heavy dependencies
- Broader NLP scope than necessary

This project provides a focused, minimal, and modern alternative dedicated solely to segmentation.

