Metadata-Version: 2.2
Name: tokmor
Version: 1.4.1
Summary: Dependency-free deterministic tokenizer and morphology hints for 300+ configured languages
Author-email: Zeus Kim <zeus@zeus.kim>
License: MIT
Project-URL: Homepage, https://github.com/zeus-kim/tokmor
Project-URL: Documentation, https://github.com/zeus-kim/tokmor#readme
Project-URL: Source, https://github.com/zeus-kim/tokmor
Project-URL: Issues, https://github.com/zeus-kim/tokmor/issues
Keywords: nlp,preprocessing,multilingual,tokenizer,tokenization,segmentation,morphology,lemmatization,ner,rag,information-extraction,offline
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Provides-Extra: api
Requires-Dist: fastapi>=0.100; extra == "api"
Requires-Dist: uvicorn>=0.23; extra == "api"

# TokMor

**Dependency-free deterministic tokenizer and morphology hints for multilingual text.**

[![PyPI](https://img.shields.io/pypi/v/tokmor.svg)](https://pypi.org/project/tokmor/)
[![Python](https://img.shields.io/pypi/pyversions/tokmor.svg)](https://pypi.org/project/tokmor/)
[![License: MIT](https://img.shields.io/pypi/l/tokmor.svg)](https://pypi.org/project/tokmor/)

TokMor is a compact preprocessing library for deterministic tokenization, offsets, language-aware segmentation, and rule-based morphology hints. It is built for offline product use: deterministic output, no model downloads, and no runtime dependencies in the core package.

TokMor is not a Transformer model, POS tagger, NER classifier, crawler, dashboard, or news-analysis pipeline. Downstream systems can use TokMor spans and morphology hints before entity recognition, search, RAG, or analytics.

## Current Scope

- 300+ configured language codes with deterministic fallback tokenization.
- 121-language product contract gate for representative real-world text patterns.
- Repository regression suites for morphology hints, boundaries, offsets, and multilingual product-contract cases.
- Focused no-space language support for Korean, Japanese, Chinese, Thai, and related CJK/SEA scripts.
- Stable handling for offsets, punctuation, dates, URLs, versions, units, abbreviations, and common clitic/MWT patterns.
- CLI, Python API, optional FastAPI server, and stdio MCP server.

Language quality is tiered. Some languages have tuned segmentation and morphology packs; others intentionally use conservative deterministic fallback rules. See [docs/LANGUAGE_TIERS.md](https://github.com/zeus-kim/tokmor/blob/main/docs/LANGUAGE_TIERS.md) and [docs/COMMERCIAL_MVP.md](https://github.com/zeus-kim/tokmor/blob/main/docs/COMMERCIAL_MVP.md).

## Install

```bash
pip install tokmor
```

The core wheel includes compact starter dictionaries for CJK/SEA segmentation,
frequent-word POS/UPOS hints, MWT/clitic splits, lemma manifests, and small
domain hint assets. No separate dictionary download is required for default
local use. Larger lemma dictionaries and language resources remain optional
offline data packs.

Run `tokmor assets --pretty` after installation to inspect the bundled dictionary inventory.

From a source checkout, install editable mode from the repository root:

```bash
python3 -m pip install -e ".[dev]"
```

## Python

```python
import tokmor

out = tokmor.unified_tokenize("우리는 2025-01-10에 서울을 방문했다.", lang="ko", sns=False)
print([t["text"] for t in out["tokens"]])

prep = tokmor.ner_preprocess("東京の新製品を25℃でテストした。", lang="ja")
print(prep[:3])
```

Token dictionaries include `text`, `start`, and `end`; offsets slice back to the original string. `ner_preprocess` prepares spans for downstream NER/search systems; it does not assign entity labels.

## CLI

```bash
tokmor tokenize --lang ko --text "로마로부터 정명석의" --text-output
tokmor analyze --lang ja --text "東京の新製品" --pretty
tokmor ner-preprocess --lang zh --text "我们在2025-01-10访问了首尔。" --pretty
tokmor explain --lang auto --text "TokMor checks bundled assets." --pretty
tokmor explain-document --lang auto --text "TokMor checks bundled assets. It keeps offsets stable." --summary-only --pretty
tokmor assets --pretty
```

If the console script is not on `PATH`, use the equivalent module form: `python -m tokmor tokenize ...`.

See [docs/CLI.md](https://github.com/zeus-kim/tokmor/blob/main/docs/CLI.md).

## MCP and API

Run the MCP server for AI hosts:

```bash
tokmor-mcp
```

Run the optional HTTP API after installing the `api` extra:

```bash
python3 -m pip install "tokmor[api]"
tokmor-api --host 127.0.0.1 --port 8000
```

Contract details are in [docs/MCP.md](https://github.com/zeus-kim/tokmor/blob/main/docs/MCP.md) and [docs/TOKMOR_API_CONTRACT.md](https://github.com/zeus-kim/tokmor/blob/main/docs/TOKMOR_API_CONTRACT.md).

## Verify a Checkout

Fast product checks:

```bash
PYTHONPATH=TokMor_v1 python3 examples/quickstart_tier1.py
PYTHONPATH=TokMor_v1 python3 scripts/run_token_invariants_smoke.py
PYTHONPATH=TokMor_v1 python3 scripts/run_multilang_product_contract.py
PYTHONPATH=TokMor_v1 python3 scripts/run_morph_regress.py --langs en,ko,ja,zh,th,ar,hi
PYTHONPATH=TokMor_v1 python3 scripts/audit_repo_scope.py --strict --include-history
```

Full release gate:

```bash
python3 scripts/run_productization_checks.py \
  --python python3 \
  --with-pos-eval \
  --pos-ud-root /path/to/ud_top10 \
  --with-bench \
  --with-package \
  --bench-iters 300
```

## Repository Hygiene

This repository is kept to TokMor core runtime, product contracts, regression fixtures, packaging, and product documentation. Non-core dashboards, news-analysis apps, generated build outputs, local data, and session artifacts are intentionally excluded. Run the scope audit before release:

```bash
PYTHONPATH=TokMor_v1 python3 scripts/audit_repo_scope.py --strict --include-history --json
```

The expected Git author/committer identity for this repository is `Zeus Kim <zeus@zeus.kim>`.

## Docs

- [docs/COMMERCIAL_MVP.md](https://github.com/zeus-kim/tokmor/blob/main/docs/COMMERCIAL_MVP.md): conservative product positioning and limits.
- [docs/AUTODEV_AUTOMATION.md](https://github.com/zeus-kim/tokmor/blob/main/docs/AUTODEV_AUTOMATION.md): automated d2 development loop.
- [docs/NEXT_DEVELOPMENT_PLAN.md](https://github.com/zeus-kim/tokmor/blob/main/docs/NEXT_DEVELOPMENT_PLAN.md): parallel development tracks and gates.
- [docs/LANGUAGE_TIERS.md](https://github.com/zeus-kim/tokmor/blob/main/docs/LANGUAGE_TIERS.md): language support model.
- [docs/TOKMOR_API_CONTRACT.md](https://github.com/zeus-kim/tokmor/blob/main/docs/TOKMOR_API_CONTRACT.md): Python/API/MCP contract expectations.
- [docs/AUTONER_TOKMOR_HANDOFF.md](https://github.com/zeus-kim/tokmor/blob/main/docs/AUTONER_TOKMOR_HANDOFF.md): downstream AutoNER integration notes.
- [docs/DATA_SOURCES_AND_LICENSES.md](https://github.com/zeus-kim/tokmor/blob/main/docs/DATA_SOURCES_AND_LICENSES.md): data source and license notes.
- [docs/FAQ.md](https://github.com/zeus-kim/tokmor/blob/main/docs/FAQ.md): what TokMor is and is not.

## License

MIT. See [LICENSE](LICENSE).
