Metadata-Version: 2.4
Name: mizosbd
Version: 0.1.0
Summary: Sentence Boundary Detection (SBD) and Tokenizer for the Mizo Language
Author-email: robzchhangte <robzchhangte@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/robzchhangte/mizosbd
Project-URL: Repository, https://github.com/robzchhangte/mizosbd
Project-URL: Issues, https://github.com/robzchhangte/mizosbd/issues
Keywords: mizo,lushai,nlp,sentence-boundary-detection,tokenizer,sentence-splitter,crf
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: sklearn-crfsuite>=0.3.6
Requires-Dist: scikit-learn>=1.3.0
Requires-Dist: numpy>=1.24.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Provides-Extra: train
Requires-Dist: torch>=2.0; extra == "train"
Requires-Dist: transformers>=4.30; extra == "train"
Requires-Dist: nltk>=3.8; extra == "train"
Requires-Dist: spacy>=3.5; extra == "train"
Dynamic: license-file

# MizoSBD: Sentence Boundary Detection for the Mizo Language

A benchmark and system for **Sentence Boundary Detection (SBD)** in Mizo (Lushai), a Tibeto-Burman language spoken primarily in Mizoram, India. The system focuses on **robust abbreviation handling** — the core challenge in Mizo text where a single period is deeply ambiguous across sentence endings, name initials, honorifics, currency, ordinals, and no-space word joins.

---

## The Problem

In Mizo text, `.` appears in many non-boundary contexts:

| Type | Example | Count in corpus |
|---|---|---|
| Name initials | `K. Vanlalvena`, `R. Lalnuntluanga` | 247 in test |
| Ordinals / list numbers | `No. 42`, `Sl. 4` | 285 in test |
| Honorifics | `Dr. Lalthlengliana`, `Er. Zothansanga` | 126 in test |
| Currency | `Rs. 500` | 59 in test |
| No-space joins | `inhnukdawk.Kum 26-a upa` | 1,474 paragraphs |
| Decimals / scores | `4.5`, `7-0` | — |

A system must cut true sentence boundaries while protecting all of these.

---

## Dataset

**Source:** 8,000 paragraphs with gold `<s>` sentence boundary annotations, collected from Mizo news and general-domain text.

**Split:** Paragraph-level (no sentence-level leakage):

| Split | Paragraphs | Sentences | Candidates |
|---|---|---|---|
| Train | 6,400 | ~27,700 | 36,595 |
| Dev | 800 | ~3,560 | 4,689 |
| Test | 800 | ~3,180 | 4,432 |

**Task formulation:** Candidate-based classification at punctuation positions. Every `.?!…` in the raw paragraph text is a candidate; the model classifies it as a sentence boundary or not.

Positive rate: ~75% of candidates are true boundaries. The always-True trivial baseline scores F1=0.835 — all reported systems are compared against this.

---

## Results

Test set results (paragraph-level split, seed=42):

| Model | P | R | F1 | MacroF1 | Training data |
|---|---|---|---|---|---|
| Always-True (trivial) | 0.717 | 1.000 | 0.835 | 0.418 | — |
| Naive Rule | 0.803 | 0.599 | 0.686 | 0.580 | None |
| spaCy Sentencizer | 0.844 | 0.796 | 0.819 | 0.702 | None |
| NLTK Punkt | 0.852 | 0.796 | 0.823 | 0.711 | Raw text (unsupervised) |
| Rule+Dict (curated) | 0.959 | 0.757 | 0.846 | 0.785 | Hand-curated gazetteer |
| Logistic Regression | 0.991 | 0.991 | 0.991 | 0.982 | Labeled candidates |
| **CRF** | **0.992** | **0.990** | **0.991** | **0.984** | Labeled candidates |
| BiLSTM | — | — | — | — | *(in progress)* |
| XLM-R (fine-tuned) | — | — | — | — | *(in progress)* |

**Abbreviation hard-subset error rates** (false boundary rate on known non-boundary categories):

| Category | Naive Rule | Punkt | Rule+Dict | CRF |
|---|---|---|---|---|
| Titles (Dr., Er., Rev.) | 97.6% | 95.2% | 0.0% | 0.0% |
| Initials (K., R.) | 51.0% | 3.2% | 0.4% | 0.0% |
| Currency (Rs.) | 0.0% | 91.5% | 0.0% | 0.0% |
| Ordinals (No., Sl.) | 31.8% | 34.4% | 0.2% | 0.0% |
| Multi-dot (w.e.f) | 35.0% | 22.9% | 0.0% | 0.0% |
| Other (uncategorised) | 23.6% | 31.0% | 27.2% | **7.1%** |

Key finding: NLTK Punkt and spaCy Sentencizer both fail almost completely on Mizo titles and currency abbreviations — categories that require Mizo-specific knowledge. CRF with Mizo-aware features is the only model that handles all named categories.

---

## Installation

### Via pip

```bash
# Install directly from PyPI (once published)
pip install mizosbd

# Or install directly from GitHub
pip install git+https://github.com/robzchhangte/mizosbd.git

# Or install locally in editable mode for development
git clone https://github.com/robzchhangte/mizosbd.git
cd mizosbd
pip install -e ".[dev]"
```

---

## Quickstart (Python API)

```python
from mizosbd import split_sentences, MizoSentenceSplitter

text = (
    "Dr. Lalthlengliana chuan thu a sawi a. "
    "K. Vanlalvena pawh a tel ve a ni. "
    "A hmun chu Aizawl a ni a, Rs. 500 an chawi a nih chu!"
)

# Convenience function (uses pre-trained bundled CRF model)
sentences = split_sentences(text)
for s in sentences:
    print("-", s)

# Or class-based interface
splitter = MizoSentenceSplitter()
sentences = splitter.split(text)
```

---

## CLI & Research Usage

### Build candidate datasets

```bash
python -m mizosbd.cli build-dataset
```

Reads `all/mizosbd_data.jsonl`, produces paragraph-level splits, enumerates candidates.
Output: `results/candidates/{train,dev,test}_candidates.jsonl`

### Train a model

```bash
python -m mizosbd.cli train --model punkt
python -m mizosbd.cli train --model crf
python -m mizosbd.cli train --model logreg
python -m mizosbd.cli train --model bilstm
python -m mizosbd.cli train --model transformer --model-name xlm-roberta-base
```

Stateless models (no training needed): `naive_rule`, `spacy_sentencizer`, `rule_based`

### Evaluate

```bash
python -m mizosbd.cli eval --model naive_rule
python -m mizosbd.cli eval --model spacy_sentencizer
python -m mizosbd.cli eval --model punkt
python -m mizosbd.cli eval --model rule_based
python -m mizosbd.cli eval --model crf
python -m mizosbd.cli eval --model logreg
python -m mizosbd.cli eval --model bilstm
python -m mizosbd.cli eval --model transformer
```

Outputs: metrics JSON + error analysis CSV to `results/`.

### Run tests

```bash
pytest tests/
```

---

## Project Structure

```
mizosbd/
├── all/
│   └── mizosbd_data.jsonl        # 8,000 annotated paragraphs (source)
├── src/mizosbd/
│   ├── data.py                   # Paragraph-level split loader
│   ├── candidates.py             # Candidate enumeration and serialization
│   ├── abbrev.py                 # Abbreviation gazetteer + protection API
│   ├── features.py               # Per-candidate features for CRF/LogReg
│   ├── eval.py                   # P/R/F1, macro-F1, abbrev subset, McNemar
│   ├── cli.py                    # Command-line interface
│   └── models/
│       ├── rule_based.py         # Rule+Dict baseline (curated gazetteer)
│       ├── punkt_spacy.py        # NaiveRule, NLTK Punkt, spaCy Sentencizer
│       ├── crf.py                # CRF and Logistic Regression
│       ├── bilstm.py             # Char+token BiLSTM classifier
│       └── transformer.py        # Fine-tuned XLM-R / MuRIL / mBERT
├── results/
│   ├── candidates/               # Serialized candidate datasets
│   ├── *.json                    # Per-model metrics
│   └── *_errors.csv              # Error analysis per model
├── tests/                        # pytest suite
├── notebooks/                    # Colab-portable transformer training
└── paper/                        # Paper drafts (ARS pipeline)
```

---

## Design Decisions

**Why candidate-based, not sequence labeling?**
Candidate-based classification focuses evaluation precisely on the ambiguous cases (abbreviations, initials, decimals) rather than diluting metrics across the full token sequence. It also makes the abbreviation hard-subset metric straightforward to compute.

**Why paragraph-level splitting?**
Splitting at the sentence level (shuffling individual sentences) causes document-level leakage — sentences from the same source paragraph appear in both train and test. Paragraph-level splitting is the methodologically correct choice.

**Why is Rule+Dict not the paper's main baseline?**
The curated abbreviation list (`Dr`, `Mr`, `Rs`, `No`, all 26 initials…) is hand-written domain knowledge — not learned from data. It is not reproducible for a new language without manual effort. NLTK Punkt and the Naive Rule are the fair, reproducible baselines. Rule+Dict is kept as an ablation showing the value of domain-specific knowledge.

---

## Citation

*(Paper under preparation)*

---

## License

*(To be added)*
