Metadata-Version: 2.4
Name: tajiknlp
Version: 1.2.0
Summary: Production-ready NLP library for Tajik language (Cyrillic script)
Author-email: Arabov Mullosharaf Kurbonovich <cool.araby@gmail.com>
Maintainer-email: Arabov Mullosharaf Kurbonovich <cool.araby@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/TajikNLPWorld/tajiknlp
Project-URL: Repository, https://github.com/TajikNLPWorld/tajiknlp
Project-URL: Issues, https://github.com/TajikNLPWorld/tajiknlp/issues
Project-URL: Documentation, https://TajikNLPWorld.github.io/tajiknlp
Project-URL: Changelog, https://github.com/TajikNLPWorld/tajiknlp/blob/main/CHANGELOG.md
Keywords: tajik,nlp,natural-language-processing,tokenization,lemmatization,low-resource,cyrillic,morphology,text-processing,persian-family
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Education
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Natural Language :: Persian
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: regex>=2023.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff>=0.6.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4.0; extra == "dev"
Requires-Dist: numpy>=1.20; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5; extra == "docs"
Requires-Dist: mkdocs-material>=9.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24; extra == "docs"
Provides-Extra: neural
Requires-Dist: tokenizers>=0.19.0; extra == "neural"
Requires-Dist: gensim>=4.3.0; extra == "neural"
Requires-Dist: huggingface-hub>=0.20.0; extra == "neural"
Dynamic: license-file

# 🇹🇯 TajikNLP

**Production-ready NLP library for the Tajik language** (Cyrillic script).

[![PyPI version](https://img.shields.io/pypi/v/tajiknlp.svg)](https://pypi.org/project/tajiknlp/)
[![Python 3.8+](https://img.shields.io/badge/python-3.8%2B-blue)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Ruff](https://img.shields.io/badge/code%20style-ruff-261230.svg)](https://github.com/astral-sh/ruff)
[![CI](https://github.com/TajikNLPWorld/tajiknlp/actions/workflows/ci.yml/badge.svg)](https://github.com/TajikNLPWorld/tajiknlp/actions/workflows/ci.yml)
[![Downloads](https://static.pepy.tech/badge/tajiknlp)](https://pepy.tech/project/tajiknlp)

TajikNLP provides a complete suite of text processing tools specifically designed for the unique challenges of the Tajik language (Cyrillic alphabet). Whether you're building a search engine, a chatbot, or conducting linguistic research, TajikNLP gives you robust, customizable components that work out of the box.

---

## ✨ Features

### 🧹 Text Cleaning (100% Complete)
- Remove URLs, emails, HTML tags, mentions, and hashtags
- Strip invisible Unicode characters and normalize whitespace
- Smart bracket cleaning (empty brackets and punctuation‑only brackets are removed)
- Remove spaces before punctuation marks

### 🔤 Normalization (100% Complete)
- Tajik Cyrillic standardization (e.g., `љ` → `ҷ`, `ї` → `ӣ`)
- Persian/Arabic digit conversion (`۱۲۳` → `123`)
- Quote and dash normalization (`« »` → `"`, `—` → `-`)
- Optional lowercasing and custom character replacement maps
- `check_normalization()` to validate text normalization

### ✂️ Tokenization (100% Complete)
- Regex‑based word tokenizer aware of Tajik‑specific letters (`ғ`, `ӣ`, `қ`, `ӯ`, `ҳ`, `ҷ`)
- Handles hyphenated compounds (`китоб-ҳо`)
- Preserves punctuation as separate tokens (optional)
- Provides accurate character offsets for each token
- Morpheme-level tokenization for linguistic analysis
- **Subword tokenization (BPE)** – pretrained BPE tokenizer for Tajik

### 📝 Sentence Splitting (100% Complete)
- Rule-based sentence boundary detection
- Protects Tajik-specific abbreviations (`т.д.`, `и.т.п.`, `ва ғ.`)
- Handles initials, decimals, dates, URLs, and emails

### 🏷️ Part‑of‑Speech Tagging (100% Complete)
- Rule‑based tagger with dictionary lookup and morphological fallback
- Category priority system from suffix rules
- Works on raw text without prior annotation

### 🔍 Morpheme Segmentation (100% Complete)
- Splits words into prefixes, root, and suffixes
- Uses linguistic rules from external JSON files (fully customizable)
- Returns structured morpheme data for each token

### 📚 Lemmatization (100% Complete)
- Hybrid approach: direct dictionary lookup + iterative affix stripping
- POS‑aware suffix removal for higher accuracy
- Built‑in dictionary of irregular forms

### 🌿 Stemming (100% Complete)
- Conservative-to-aggressive rule-based stemmer
- Safe mode for grammatical reductions
- Deep mode for derivational stripping
- Corpus-aware scoring for better accuracy

### 🛑 Stop Words Filtering (100% Complete)
- Comprehensive stop word list (100+ items)
- Automatically removes function words while preserving content words

### 🎯 Named Entity Recognition (NER) Ready (100% Complete)
- Character-level span alignment
- BIO tag generation for entity spans
- Offset preservation for training data preparation

### 🧠 Neural Pipeline
- **Pretrained Word2Vec / FastText embeddings** – 300‑dimensional vectors for Tajik words
- **Cosine similarity** – compute semantic similarity between words or custom vectors
- **Most similar words** – find nearest neighbours in the embedding space
- Automatic download of models from Hugging Face Hub (cached locally)

### 📊 Feature Extraction
- `TajikCountVectorizer` – Bag‑of‑Words with n‑gram support
- `TajikTfidfVectorizer` – TF‑IDF feature extraction
- Seamless integration with `Doc` objects or raw strings

### 🏷️ Keyword Classification
- Lightweight `KeywordClassifier` – assign categories based on user‑defined keywords
- Returns confidence scores and supports case‑insensitive matching
- Works directly with tokenized `Doc` objects (uses lemmas when available)

### 🧬 Unified Morphology Engine (NEW in 1.2.0)
- **TajikMorphEngine** – unified prefix+suffix rule‑based engine with three modes:
  - `lemma` – safe lemmatization
  - `controlled` – broader stripping with POS/dictionary checks
  - `deep` – aggressive stemming
- **UnifiedMorphLemmatizer** and **UnifiedMorphStemmer** – alternative experimental components
- Provides detailed morphological analysis (prefixes, suffixes, validity)

### 💬 Sentiment Analysis (NEW in 1.2.0)
- **LexiconSentimentAnalyzer** – lexicon‑based sentiment analysis
- Hand‑curated Tajik sentiment lexicon with **4,000+ entries** and intensity scores
- Handles negations (`на`, `не`), intensifiers (`хеле`, `бисёр`), and punctuation
- Optional stemming via `TajikMorphEngine` for recognizing inflected forms (`бадро`, `хубро`)
- Returns sentiment label (`positive`/`negative`/`neutral`), score, and matched words

### 🛠️ Utilities (100% Complete)
- **Script detection** – identify if text is Cyrillic, Latin, Arabic, or mixed
- **Language detection** – distinguish Tajik from Russian, Persian, English, Arabic
- **Quality scoring** – evaluate how "Tajik‑like" a piece of text is
- **Validation helpers** – quickly check if a string is valid Tajik text
- **Evaluation Metrics** – Precision, Recall, F1, Edit Distance, Word Error Rate, BLEU score, Classification Report

---

## 📊 Project Status

| Module                  | Status        | Tests   | Coverage |
|-------------------------|---------------|---------|----------|
| Preprocessing           | ✅ Complete   | 33/33   | 100%     |
| Tokenization            | ✅ Complete   | 22/22   | 88%      |
| Subword Tokenization    | ✅ Complete   | 12/12   | 100%     |
| Sentence Splitting      | ✅ Complete   | 7/7     | 93%      |
| POS Tagging             | ✅ Complete   | 9/9     | 87%      |
| Morpheme Segmentation   | ✅ Complete   | 8/8     | 98%      |
| Lemmatization           | ✅ Complete   | 23/23   | 92%      |
| Stemming                | ✅ Complete   | 6/6     | 87%      |
| Stop Words Filtering    | ✅ Complete   | 6/6     | 95%      |
| NER / Alignment         | ✅ Complete   | 5/5     | 84%      |
| Script Detection        | ✅ Complete   | 47/47   | 95%      |
| Validation & Metrics    | ✅ Complete   | 18/18   | 99%      |
| Embeddings              | ✅ Complete   | 15/15   | 83%      |
| Feature Extraction      | ✅ Complete   | 13/13   | 96%      |
| Keyword Classification  | ✅ Complete   | 20/20   | 98%      |
| Unified Morphology      | ✅ Complete   | 13/13   | 93%      |
| Sentiment Analysis      | ✅ Complete   | 15/15   | 94%      |
| Core & Pipeline         | ✅ Complete   | 39/39   | 99%      |
| Configuration & Errors  | ✅ Complete   | 23/23   | 85%      |
| Utilities & Resources   | ✅ Complete   | 26/26   | 100%     |
| **Overall**             | **✅ 100%**   | **616** | **93%**   |

---

## 📦 Installation

TajikNLP requires Python 3.8 or later.

### Basic Installation

```bash
pip install tajiknlp
```

### With Neural Components (Word2Vec, BPE tokenizer)

```bash
pip install "tajiknlp[neural]"
```

### Development Installation

```bash
pip install "tajiknlp[dev]"
```

---

## 🧪 Try It Now in Google Colab

Click the badge below to run a complete demo notebook directly in your browser:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/TajikNLPWorld/tajiknlp/blob/main/examples/demo/tajiknlp_demo.ipynb)

---

## 🚀 Quick Start

### Load the default pipeline

```python
from tajiknlp import load_pipeline

pipe = load_pipeline("default")
text = "Китобҳоямонро хондам, аммо нафаҳмидам."
doc = pipe(text)

for token in doc.tokens:
    print(f"{token.text:<15} → lemma: {token.lemma:<10} pos: {token.pos}")
```

**Output:**
```
китобҳоямонро   → lemma: китоб      pos: NOUN
хондам          → lemma: хондан     pos: VERB
,               → lemma: ,          pos: PUNCT
аммо            → lemma: аммо       pos: CONJ
нафаҳмидам      → lemma: фаҳмидан   pos: VERB
.               → lemma: .          pos: PUNCT
```

### Available Pipeline Presets

| Preset               | Components                                                       |
|----------------------|------------------------------------------------------------------|
| `minimal`            | Cleaner + Tokenizer + Morpheme splitter                          |
| `default`            | Full preprocessing + POS + Morphemes + Stopwords + Lemmatizer    |
| `full`               | Alias for `default`                                              |
| `stemming`           | Same as default but with stemmer instead of lemmatizer           |
| `ner`                | Optimized for NER (preserves case)                               |
| `neural`             | BPE tokenizer + Word2Vec embeddings (requires `[neural]`)        |
| `lemmatizer_unified` | Experimental lemmatizer based on TajikMorphEngine                |
| `stemmer_unified`    | Experimental stemmer based on TajikMorphEngine                   |
| `sentiment`          | Lexicon‑based sentiment analysis                                 |

### Neural Pipeline Example

```python
nlp = load_pipeline("neural")
doc = nlp("Эргашбой Мирзоевич Муҳамадиев - риёзидони бузурги тоҷик.")
for token in doc:
    vec = token.metadata.get("vector")
    if vec:
        print(f"{token.text}: vector dim={len(vec)}")
```

### Unified Morphology Example

```python
from tajiknlp.morphology import TajikMorphEngine

engine = TajikMorphEngine(mode="controlled")
result = engine.analyze("намеравам", pos="VERB")
print(f"Stem: {result['stem']}, Prefixes: {result['prefixes']}, Suffixes: {result['suffixes']}")
```

### Sentiment Analysis Example

```python
pipe = load_pipeline("sentiment")
doc = pipe("Китоби хеле хуб!")
print(f"Sentiment: {doc.metadata['sentiment']}, Score: {doc.metadata['sentiment_score']}")
```

### Feature Extraction

```python
from tajiknlp import load_pipeline, TajikCountVectorizer

pipe = load_pipeline("default")
docs = [pipe("Ман китоб хондам."), pipe("Ту китоб нахонди.")]
cv = TajikCountVectorizer(ngram_range=(1, 2))
X = cv.fit_transform(docs)
print(cv.get_feature_names_out())
```

### Keyword Classification

```python
from tajiknlp import KeywordClassifier

clf = KeywordClassifier()
clf.add_category("спорт", {"футбол", "варзиш", "бозӣ"})
clf.add_category("сиёсат", {"президент", "интихобот", "ҳукумат"})
print(clf.predict("Дастаи футбол бозӣ бурд."))  # 'спорт'
```

---

## 📖 Documentation

Full documentation is available at **[TajikNLPWorld.github.io/tajiknlp](https://TajikNLPWorld.github.io/tajiknlp)**.

It includes:
- Detailed API references
- Component configuration guides
- Advanced usage examples
- NER and alignment tutorials

---

## 🧪 Running Tests

If you cloned the repository, you can run the test suite with:

```bash
# Install development dependencies
pip install -e ".[dev]"

# Run all tests
pytest tests/ -v

# Run with coverage report
pytest tests/ --cov=tajiknlp --cov-report=html
```

---

## 🤝 Contributing

We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on:

- Setting up the development environment
- Coding standards and style guides
- Testing requirements
- Pull request process

---

## 📄 License

This project is licensed under the MIT License – see the [LICENSE](LICENSE) file for details.

Copyright (c) 2026 TajikNLPWorld
Copyright (c) 2026 Arabov Mullosharaf Kurbonovich

---

## 🙏 Acknowledgements

TajikNLP was inspired by the need for robust, open‑source NLP tools for low‑resource languages. Special thanks to all contributors and the Tajik linguistic community.

---

## 📊 Citation

If you use TajikNLP in your research, please cite:

```bibtex
@software{tajiknlp2026,
  author = {Arabov, Mullosharaf Kurbonovich},
  title = {TajikNLP: Production-ready NLP library for Tajik language},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/TajikNLPWorld/tajiknlp}
}
```

---

**Made with ❤️ for the Tajik language.**
