Metadata-Version: 2.4
Name: TAN_LANG
Version: 0.1.0
Summary: Tanzania Local Language (TLL) Python library for Kiswahili and Kisukuma.
Author: TLL Contributors
License: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# TLL — Tanzania Local Language

TLL is a Python library ecosystem for Tanzanian local-language technology. The project starts with Kiswahili (`sw`) and Kisukuma (`suk`) and is designed to support additional Tanzanian languages without rewriting the core architecture.

This repository currently focuses on the foundational pieces required for a serious, reusable library:

- language registry
- dataset provenance and validation
- raw/cleaned/reviewed/verified data pipeline
- dictionary-oriented data access
- translation API with conservative behavior
- text normalization and tokenization hooks
- CLI and documentation scaffolding

Important: some features are intentionally scaffolded and documented as future work rather than pretending they are fully implemented.

## Quick start

PowerShell:

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -e .
```

To install the published package from PyPI:

```powershell
pip install TAN_LANG
```

The package includes the cleaned Sukuma dataset needed by the dictionary
translation API, so `LOCAL_LANGUAGE` and the repository's root `data/` folder
are not required after installation.

Then:

```python
from tll import get_language, list_languages

print(get_language("suk").name)
print([lang.code for lang in list_languages()])
```

## Translation and chatbot usage

The translation API now uses the dataset-first algorithm from the
`LOCAL_LANGUAGE` project:

1. exact match
2. word-by-word composition for phrases
3. fuzzy spelling match
4. explicit "not found" response instead of invented translations

PowerShell examples:

```powershell
tll translate --from suk --to sw mwalimu
tll translate --from suk --to sw "mwalimu shule"
tll chat --from suk --to sw mwalimu
```

Python usage:

```python
from tll import LocalLanguageChatbot, Translator

translator = Translator(source="suk", target="sw")
print(translator.translate("mwalimu shule"))

chatbot = LocalLanguageChatbot(language="suk")
print(chatbot.chat("mwalimu", source_language="suk", target_language="sw"))
```

The chatbot is grounded in the local dataset. `grounded=True` means the
response came from a dataset match; a `grounded=False` response must be
reviewed by a native speaker and should not be treated as a verified
translation.

## Current validation workflow

```powershell
python scripts/validate_dataset.py --dataset data/raw/suk/sukuma_deduplicated.csv
```

This validates the provided Kisukuma dataset without modifying the original raw file.

## Reviewed and verified dataset workflow

```powershell
python scripts/review_dataset.py --input data/cleaned/suk/cleaned_sukuma_dataset.csv --output-dir data/reviewed/suk --language suk
```

This creates the reviewed stage and then promotes the verified subset into a dedicated verified dataset. The library keeps all intermediate data, so the raw source remains untouched and the review decisions remain auditable.

## Project status

This is Phase 1 of the roadmap: dataset inspection, architecture, language registry, and validation. The cleaning stage begins only after validation is complete and reviewed.

## License

This project is licensed under the MIT license. Dataset licenses are handled separately and must be recorded carefully; if unknown, the metadata must state `unknown`.
