Metadata-Version: 2.4
Name: mnk-persian-words
Version: 1.1.0
Summary: A zero-config Persian word toolkit with random sampling, filtering, normalization, and dataset metadata
Author-email: Masoud Najafzadeh Kalat <masoudnk2@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/masoudnk/PersianWordsLib
Project-URL: Documentation, https://github.com/masoudnk/PersianWordsLib#readme
Project-URL: Issues, https://github.com/masoudnk/PersianWordsLib/issues
Project-URL: Source, https://github.com/masoudnk/PersianWordsLib
Keywords: persian,farsi,random-words,text-generation,nlp,rtl,unicode
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Natural Language :: Persian
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# PersianWords

[![PyPI version](https://img.shields.io/pypi/v/mnk-persian-words.svg)](https://pypi.org/project/mnk-persian-words/)
[![Python versions](https://img.shields.io/pypi/pyversions/mnk-persian-words.svg)](https://pypi.org/project/mnk-persian-words/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**PersianWords** (`mnk-persian-words`) is a zero-configuration Python toolkit for working with a large Persian word dataset. It provides random sampling, filtering, deterministic seeds, Persian Unicode normalization, dataset statistics, placeholder text, and a command-line interface.

The production dataset contains **628,419 unique Persian words**. The library is designed for UI/UX placeholder text, educational software, word games, RTL/font testing, and language-processing experiments.

> **PyPI 1.0.2 warning**  
> The artifacts published for version `1.0.2` accidentally omitted the bundled database and are only a few kilobytes. The next release is `1.1.0`, which includes corrected packaging plus the new single-database raw/normalized/clean architecture. Until it is published, use the repository version when the complete dataset is required.

## Installation

After `1.1.0` is published:

```bash
pip install mnk-persian-words
```

Until then:

```bash
pip install "git+https://github.com/masoudnk/PersianWordsLib.git"
```

Python **3.9+** is supported.

## Quick start

No manual database loading is required:

```python
from mnk_persian_words import get_random_persian_word

print(get_random_persian_word())
print(get_random_persian_word(5))
```

The dataset is prepared lazily on first use and then reused from a versioned local cache.

## Public API

### Random words

```python
from mnk_persian_words import get_random_persian_words

words = get_random_persian_words(10, seed=42)
print(words)
```

Use `unique=True` to sample without replacement:

```python
get_random_persian_words(20, unique=True, seed=42)
```

A seed is handled with an independent `random.Random` instance, so the application's global random state is not modified.

### Filtering

Filters can be combined:

```python
get_random_persian_words(
    10,
    min_length=4,
    max_length=8,
    starts_with="م",
    contains="ار",
    unique=True,
    seed=42,
)
```

Supported filters:

- `min_length`
- `max_length`
- `starts_with`
- `ends_with`
- `contains`
- `unique`
- `seed`

If no word matches a filter, `ValueError` is raised. Invalid programmer inputs raise normal Python exceptions rather than returning warning strings.

### Backward-compatible string API

```python
from mnk_persian_words import get_random_persian_word

text = get_random_persian_word(3, seed=42)
# Three words separated by spaces
```

`load_words_from_db()` still exists for compatibility with older code, but it is no longer required. Dataset loading is automatic.

### Random paragraphs

```python
from mnk_persian_words import get_random_persian_paragraph

print(get_random_persian_paragraph(words_count=8, paragraphs=2, seed=42))
```

This function intentionally generates **random word paragraphs**, not grammatical natural-language prose.

### Layout placeholder text

```python
from mnk_persian_words import get_persian_placeholder

print(get_persian_placeholder(words_count=20, paragraphs=2, seed=42))
```

The placeholder helper adds light Persian-style punctuation for UI, RTL, typography, and layout testing. It does **not** claim to generate natural Persian language.

## Persian normalization

```python
from mnk_persian_words import normalize_persian

normalize_persian("علي و كتاب")
# 'علی و کتاب'

normalize_persian("عَلِی", remove_diacritics=True)
# 'علی'
```

Normalization currently supports:

- Arabic `ي` / `ى` → Persian `ی`
- Arabic `ك` → Persian `ک`
- optional diacritic removal
- optional preservation or conversion of ZWNJ / half-space
- whitespace cleanup
- Unicode normalization

The default behavior avoids aggressive transformations that could change wording.

## Dataset information

```python
from mnk_persian_words import dataset_info

print(dataset_info())
```

The returned dictionary includes:

- dataset version
- total and unique words
- minimum and maximum word length
- words containing ZWNJ
- words containing diacritics
- storage format
- bundled dataset fingerprint

The schema-v2 production dataset identifier is `628419-v2-canonical`. Library and dataset versions are tracked separately.

## Command line interface

After installation:

```bash
persian-words word
persian-words word --count 10
persian-words word --count 20 --unique --seed 42
persian-words word --starts-with م --min-length 4
persian-words info
persian-words normalize "علي و كتاب"
```

## Performance design

The library does **not** load all 628,419 database rows into a Python `list[str]` for normal random sampling. It uses SQLite IDs and batched lookups, while filtered candidate-ID sets are cached in a small bounded cache.

This design aims to keep:

- imports fast
- normal memory use low
- repeated calls fast
- large random batches practical

A reproducible benchmark helper is provided at `benchmarks/benchmark.py`. Benchmark results are environment-specific and are not performance guarantees.

## Dataset cache

The compressed SQLite database remains inside the installed package and is extracted lazily into a versioned application cache on first use.

The cache location follows platform conventions:

- Windows: `%LOCALAPPDATA%/mnk_persian_words`
- macOS: `~/Library/Caches/mnk_persian_words`
- Linux: `$XDG_CACHE_HOME/mnk_persian_words` or `~/.cache/mnk_persian_words`

For controlled environments, set:

```text
MNK_PERSIAN_WORDS_CACHE_DIR=/custom/cache/path
```

Extraction is validated and uses an atomic replacement step to reduce corruption risk when multiple processes start at the same time.

## Data sources and licensing

The Python source code is licensed under MIT. The bundled dataset contains third-party data whose licensing status is separate from the source-code license.

See [DATA_SOURCES.md](DATA_SOURCES.md) for source provenance and licensing notes.

Primary referenced sources include:

- https://github.com/shahind/Persian-Words-Database
- http://khodam.altervista.org/لیست-همه-کلمات-فارسی-مجموعه-کامل/

Public availability does not automatically mean third-party dataset content is MIT-licensed.

## Development

Clone and install in editable mode:

```bash
git clone https://github.com/masoudnk/PersianWordsLib.git
cd PersianWordsLib
python -m pip install -e .
```

Prepare the bundled ZIP from the local production SQLite database when needed:

```bash
python tools/prepare_dataset.py mnk_persian_words/data/words.db
```

For schema v2, `words` preserves every raw record and adds `canonical_id`; `canonical_words` stores each normalized form once together with `is_clean`. `prepare_dataset.py` validates either legacy schema v1 or the new schema v2 before replacing `words.db.zip`. The uncompressed `words.db` is ignored by Git.

Run a dataset quality audit without modifying the data:

```bash
python tools/audit_dataset.py mnk_persian_words/data/words.db
```

Run tests:

```bash
python -m unittest discover -s tests -v
```

Run the benchmark:

```bash
python benchmarks/benchmark.py
```

Build distributions:

```bash
python -m build
python -m twine check dist/*
```

CI tests Python 3.9 through 3.13, builds the wheel/sdist, verifies that the dataset is physically present in the wheel, installs that wheel in a clean virtual environment, and performs installed-package and CLI smoke tests.

## Examples

Executable examples are available under `examples/`:

- `basic_usage.py`
- `filters.py`
- `normalization.py`
- `reproducible_random.py`
- `dataset_info.py`
- `dataset_modes.py`

## Release status

- Current PyPI release: `1.0.2` — **known broken dataset packaging**
- Next release: `1.1.0` — **corrected packaging + single-database canonical/clean architecture**
- Supported Python: `>=3.9`
- Production dataset: `628,419` unique words

See [CHANGELOG.md](CHANGELOG.md) for release notes.

## License

The Python source code in PersianWords is released under the [MIT License](LICENSE). See [DATA_SOURCES.md](DATA_SOURCES.md) for separate notes about third-party dataset content.
## Raw, normalized, and clean modes

Version `1.1.0` keeps one SQLite database and exposes three logical views:

- `raw`: all original 628,419 source records, unchanged. This is the default for backward compatibility.
- `normalized`: unique canonical forms after conservative Unicode cleanup such as Arabic Yeh/Kaf normalization and removal of non-semantic format controls.
- `clean`: the normalized canonical set restricted to structurally word-like Arabic-script entries. This is a structural quality profile, not a claim that every entry is a modern dictionary-standard Persian word.

```python
from mnk_persian_words import get_random_persian_words

raw = get_random_persian_words(5, mode="raw", seed=42)
normalized = get_random_persian_words(5, mode="normalized", seed=42)
clean = get_random_persian_words(5, mode="clean", seed=42)
```

Filters work with all three modes. Filter text is normalized automatically in `normalized` and `clean` modes.

The database stores raw and canonical data without duplicating whole database files:

```text
words
  id
  word            # original source form
  canonical_id    # points to canonical_words, NULL only if normalization becomes empty

canonical_words
  id
  word            # unique normalized form
  is_clean        # 0/1 structural-clean flag
```

Build and validate schema v2 without modifying the source database:

```bash
python tools/upgrade_dataset_schema.py --output build/dataset_v2/words.db
```

After reviewing the report, install it into the package while creating a schema-v1 backup:

```bash
python tools/upgrade_dataset_schema.py --install
```

Then regenerate the packaged ZIP:

```bash
python tools/prepare_dataset.py
```

The deep audit remains available and does not modify the source data:

```bash
python tools/audit_dataset.py --output build/dataset_audit.json
```
