Metadata-Version: 2.4
Name: uzbek2turkic
Version: 0.2.2
Summary: An advanced text normalizer converting standard Uzbek Latin into the Common Turkic Alphabet (CTA) for highly optimized NLP LLM tokenization natively supporting the O(1) NG ambiguity exclusion algorithms.
Author-email: "Sizning Ismingiz (Your Name)" <your_email@example.com>
License: MIT
Project-URL: Homepage, https://github.com/SizningUzeringiz/uzbek2turkic
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.7
Description-Content-Type: text/markdown

# uzbek2turkic: Uzbek Normalizer for NLP Models

A robust, rule-based text normalization library designed to prepare and standardize Uzbek text for modern AI and NLP models (BERT, mDeBERTa, Llama, WordPiece, BPE). 

The library intelligently maps Uzbek Latin sequences to the **Common Turkic Alphabet (CTA)**. By unifying the Turkic language subword ecosystem, this package prevents tokenizers from hopelessly fragmenting multi-character digraphs (like `o'`, `g'`, `sh`, `ch`, and `ng`), drastically improving training efficiency and downstream NLP performance.

---

##  Key Features

* **Algorithmic Apostrophe Unification**: Flattens all erratic variations of apostrophes (`ʻ`, `‘`, `’`, `´`, `` ` ``) into the internationally recognized W3C modifier mark: **`ʼ` (U+02BC)**.
* **Digraph Compression**: Securely converts sequential digraphs into single-character CTA equivalents (`o'` → `ö`, `g'` → `ğ`, `sh` → `ş`, `ch` → `ç`).
* **Advanced `NG` Disambiguation**: Uses a dual-layered algorithm (Grammatical Suffix Matrix + O(1) Dictionary Hash) to distinguish between the velar nasal (`ñ`) and dative/participle morphology boundaries (`n` + `g`).
* **Case-Sensitivity**: Fully preserves logical structural capitalization (`Sh` → `Ş` or `SH` → `Ş`).
* **Reversible**: Supports flawless denormalization (`trc2uz`) to restore model outputs back to standard standard Uzbek Latin.

---

##  Installation

Available on PyPI. Install via pip:
```bash
pip install uzbek2turkic
```

---

## Quick Start

```python
from uzbek2turkic.normalizer import UzbekNormalizer

# Initialize the normalizer
normalizer = UzbekNormalizer()

original_text = "O'g'il bola shahar tomon choy ichgani yo'l oldi."

# Forward Conversion (Original -> CTA)
normalized = normalizer.uz2trc(original_text)
print(normalized)
# Output: Öğil bola şahar tomon çoy içgani yöl oldi.

# Backward Conversion (CTA -> Original)
restored = normalizer.trc2uz(normalized)
print(restored)
# Output: O'g'il bola shahar tomon choy ichgani yo'l oldi.
```

---

##  Deep Dive: Semantic Disambiguation of NG (`ñ` vs `n` + `g`)

Converting `ng` to the nasal `ñ` is one of the most profound challenges in computational Uzbek text processing. `n` and `g` frequently appear sequentially as separate grammatical units (for example, a root ending in `n` appended by the dative suffix `ga`, or a passive verb ending in `n` appended by `-gan`). 

A simplistic replacement script would erroneously destroy morphology by mapping `jonga` into `joña` or `ishlangan` into `işlañan`.

**`uzbek2turkic`** solves this dynamically by analyzing **every single occurrence of `ng` independently**, applying two powerful mechanisms:

### 1. Grammatical Suffix Concurrency Matrix
The algorithm detects localized combinations of `N-suffixes` (like `-lan`, `-gan`, `-qan`) colliding with `G-suffixes` (like `-gach`, `-gan`, `-ga`). It automatically isolates these morphological boundaries without relying on absolute dictionary lookups.

```python
# Example of Grammatical Matrix working its magic:
text = "quvonganning ishlanganlariga"

print(normalizer.uz2trc(text))
# Output: quvonganniñ işlanganlariga
```
*(Notice how the legitimate initial `n+g` breaks (`quvon+gan`, `ishlan+gan`) successfully bypass `ñ` conversion, while the true genitive suffix `-ning` at the end smoothly transitions into `niñ` within the exact same lexical word).*

### 2. O(1) Root Dictionary Filtering
For native nouns that structurally end in `n` (like `tun`, `kun`, `jon`, `vatan`), the text is instantly validated against `exceptions.txt`. 

```python
# Example of Dictionary Exclusion:
text = "Tungi sovuq kunga va buguning bir qismiga ta'sir qildi."

print(normalizer.uz2trc(text))
# Output: Tungi sovuq kunga va buguniñ bir qismiga taʼsir qildi.
```

*(You can also load your own root words by passing a custom dictionary during initialization: `UzbekNormalizer(exceptions_file='/path/to/custom_roots.txt')`).*

---

##  W3C Typography and BPE Tokenizer Compliance

[According to W3C Typography and International Unicode Standards](https://www.w3.org/TR/typography/), letters that act as an integral part of a word (like the Uzbek "tutuq") **must not** be written using standard punctuation apostrophes (`'`, U+0027 or `’`, U+2019). The international standard mandates the use of **U+02BC (Modifier Letter Apostrophe — `ʼ`)**.

The Uzbek tutuq mark (used in words like `ma'no`, `sur'at`) is treated as a "punctuation boundary" by standard tokenizers (like HuggingFace BPE), unconditionally shattering the word into useless bytes (e.g., `["ma", "'", "no"]`). 

By converting standalone apostrophes into **U+02BC (`ʼ`)**, which is officially classed as a **Letter Modifier**, models treat `maʼno` as a unified, physically unbroken linguistic token. This single fix dramatically reduces context window loss during LLM training.

---

## License
MIT License. Open-source and ready for large-scale NLP pipelining.
