Metadata-Version: 2.5
Name: romanizer
Version: 0.1.0
Summary: Fast, CPU-only romanization of names and text from any script into plain Latin letters
Project-URL: Homepage, https://github.com/unicef/romanizer
Project-URL: Repository, https://github.com/unicef/romanizer
Project-URL: Issues, https://github.com/unicef/romanizer/issues
Project-URL: Changelog, https://github.com/unicef/romanizer/blob/main/CHANGELOG.md
Author-email: Jan Romaniak <romaniakjan@gmail.com>
License: MIT
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Keywords: arabic,ascii,cjk,cyrillic,names,romanization,transliteration,unicode
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: anyascii>=0.3
Provides-Extra: all
Requires-Dist: arabic-name-romanizer>=0.2; extra == 'all'
Requires-Dist: korean-romanizer>=0.25; extra == 'all'
Requires-Dist: pyicu>=2.11; extra == 'all'
Requires-Dist: pykakasi>=2.2; extra == 'all'
Requires-Dist: pypinyin>=0.50; extra == 'all'
Requires-Dist: pythainlp>=5.0; extra == 'all'
Provides-Extra: arabic
Requires-Dist: arabic-name-romanizer>=0.2; extra == 'arabic'
Provides-Extra: cjk
Requires-Dist: korean-romanizer>=0.25; extra == 'cjk'
Requires-Dist: pykakasi>=2.2; extra == 'cjk'
Requires-Dist: pypinyin>=0.50; extra == 'cjk'
Provides-Extra: icu
Requires-Dist: pyicu>=2.11; extra == 'icu'
Provides-Extra: thai
Requires-Dist: pythainlp>=5.0; extra == 'thai'
Description-Content-Type: text/markdown

# romanizer

[![CI](https://github.com/unicef/romanizer/actions/workflows/ci.yml/badge.svg)](https://github.com/unicef/romanizer/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/romanizer.svg)](https://pypi.org/project/romanizer/)
[![Python](https://img.shields.io/pypi/pyversions/romanizer.svg)](https://pypi.org/project/romanizer/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Fast, CPU-only romanization of names and text from any script into **plain Latin letters**.

```python
from romanizer import romanize

romanize("Володимир Зеленський")  # 'Volodymyr Zelenskyi'
romanize("Александр Пушкин")  # 'Aleksandr Pushkin'
romanize("Γιώργος Παπανδρέου")  # 'Giorgos Papandreou'
romanize("محمد صلاح")  # 'Mohamed Salah'
romanize("习近平")  # 'Xi Jinping'
romanize("김정은")  # 'Kim Jeongeun'
romanize("नरेन्द्र मोदी")  # 'Narendra Modi'
romanize("Nguyễn Văn Đức")  # 'Nguyen Van Duc'
romanize("Zeleнський in Київ")  # 'Zelenskyi in Kyiv'
```

The output is always printable ASCII. No apostrophes standing in for soft signs, no backticks
for ayin, no stray spaces before commas, no emoji, no `№`, no `€`.

## Why another one

Single-library solutions all fail somewhere:

- `unidecode` / `anyascii` work character by character. Ukrainian comes out Russian-flavoured
  (`Volodimir`, `Kiyiv`), soft signs become `'`, Kazakh gets `@`, Japanese kanji are read as
  Chinese (`東京` → `Dong Jing`), and CJK output has stray spaces.
- `uroman` is good but 50–100× slower than everything else (~0.5 ms per name, 1 s to load).
- Dedicated libraries (`pypinyin`, `pykakasi`, `korean-romanizer`, `arabic-name-romanizer`) are
  excellent for their one script and know nothing about the others or about mixed input.

`romanizer` splits input into same-script runs, guesses the language inside a script (Ukrainian vs
Russian vs Bulgarian, Japanese vs Chinese, Persian vs Arabic), routes each run to the best engine,
and passes everything through one final ASCII-only layer.

## Installation

```bash
pip install romanizer                 # Cyrillic, Greek, Latin, Armenian, Georgian, Ethiopic
pip install "romanizer[arabic]"       # + Arabic, Persian, Dari, Pashto, Urdu (arabic-name-romanizer, ONNX CPU)
pip install "romanizer[cjk]"          # + Chinese, Japanese, Korean
pip install "romanizer[icu]"          # + Indic and South-East Asian scripts (needs libicu)
pip install "romanizer[thai]"         # + Thai (pythainlp, rule-based engine only)
pip install "romanizer[all]"
```

Nothing here touches a GPU. Missing extras never crash: the affected script falls back to a
readable approximation and is flagged as unsupported in `analyze()`.

## Language support

Status: ✅ supported · ⚠️ supported with caveats · ❌ not supported for now.

| Language / script | Detection | Engine | What we add on top | Example | Status |
|---|---|---|---|---|---|
| Ukrainian | Cyrillic with `іїєґ`, or `ськ`/`ння`/apostrophe before iotated vowel | own table (KMU 2010 national system) | `Ye/Yi/Yu/Ya` word-initially, `зг`→`zgh`, soft sign and apostrophe dropped | `Volodymyr Zelenskyi, Kyiv, Yevhen, Yuliia` | ✅ |
| Russian | Cyrillic default, or `ыэъё` | own table (BGN/PCGN, simplified) | `е`→`ye` after vowels and word-initially, `ё`→`yo` (`e` after hushers), hard/soft signs dropped | `Aleksandr Pushkin, Yeltsin, Yuriy, Ilya` | ✅ |
| Belarusian | `ў` | own table | `г`→`h`, `ў`→`u` | `Alyaksandr Lukashenka, Uladzimir` | ✅ |
| Bulgarian | `ъ` used as a vowel | own table (2009 official system) | `ъ`→`a`, `щ`→`sht`, `-ия`→`-ia` | `Balgarski, Tarnovo`; with `lang="bg"` also `Hristo Botev, Sofia` | ✅ |
| Serbian / Macedonian | `ђћљњџј` / `ѓќѕ` | own table (Latin alphabet, diacritics stripped as in passports) | `Đ`→`Dj`, `Ћ`→`C` | `Djordje Petrovic, Cirilica, Gjorgji` | ✅ |
| Kazakh, Kyrgyz, Mongolian, Tajik, other Cyrillic | `әғқңөұүһ`, `ҳҷӣӯ` | own table (Kazakh 2021 Latin) + `anyascii` for rare letters | `қ`→`q`, `ғ`→`gh`, `ң`→`ng` | `Nursultan Nazarbaev, Qazaqstan, Jumhuri` | ✅ |
| Greek | Greek script | own table (ELOT 743) | word-initial `μπ/ντ/γκ`→`b/d/g`, medial `mb/nt/ng`, `αυ/ευ`→`av/ev` or `af/ef` by context | `Giorgos Papandreou, Bambis, Dinos, Evangelos, Nafplio, Pavlos` | ✅ |
| Arabic | Arabic script | `arabic-name-romanizer` (dictionary of ~60k attested spellings, tiny ONNX model for the rest), batched | script segmentation so Latin words, digits and punctuation never enter the model; single letters spelled (`د.` → `D.`); `arabic_country` hint | `Mohamed Salah, Abdulrahman, Abdelkader (DZ)` | ✅ |
| Persian, Dari | Arabic script with `پچژگ` | `arabic-name-romanizer` 0.2, Persian-script dictionary and model | `arabic_country="AF"` picks Afghan spellings (`Ahmad`, `Mohammad`) over Iranian ones | `Golshifteh Farahani, Mohammad Reza Pahlavi, Hamid Karzai, Zalmay Khalilzad` | ✅ 79% agreement with Wikidata |
| Pashto | Arabic script with `ټډړږښڼېۍځڅ` | same Persian-script model, country defaults to `AF` | | `ملالۍ` → `Malali` | ⚠️ names work, Pashto-specific consonants (`ښ`, `ډ`) are approximate |
| Urdu | Arabic script with `ٹڈڑںہے` | same Persian-script model, country defaults to `PK` | | `Malala Yousafzai, Mohammad Ali Janah` | ✅ 83% agreement with Wikidata |
| Hebrew | Hebrew script | none | policy `fallback` gives a consonant skeleton, `drop` removes it, `raise` errors | `Vnymyn Ntnyhv` | ❌ no vowels without a dictionary or model |
| Chinese | Han without kana | `pypinyin` | in names mode a 2–3 character token starting with one of ~200 common surnames is split into surname + given name; `ü`→`u` | `Xi Jinping, Zhang Wei, Ouyang Xiu`; `北京市` → `Bei Jing Shi` | ✅ |
| Japanese | kana present, or `lang="ja"` | `pykakasi` (Hepburn) | long vowels collapsed (`toukyou`→`Tokyo`), words capitalised | `Tokyo, Yamada Taro` | ⚠️ kanji name readings are inherently ambiguous |
| Korean | Hangul | `korean-romanizer` (Revised Romanization) | a 3-syllable token starting with one of the 40 most common surnames is split, with the customary spelling (`Kim, Lee, Park`); compound surnames handled | `Kim Jeongeun, Park Chanho, Seoul`; `대통령` → `Daetongryeong` | ⚠️ consistent RR, but Koreans spell given names ad hoc (`Jong-un`, `Young Shik`): 26% agreement with Wikidata |
| Hindi, Marathi, Nepali | Devanagari | ICU `Devanagari-Latin` (ISO 15919) | schwa deletion on the ISO form (short `a` only, so `Sunita` keeps its vowel and `Ram` loses it), `c`→`ch`, `ś/ṣ`→`sh`, `siṁha`→`Singh`, title case | `Sachin Tendulkar, Amitabh Bachchan, Priyanka Chopra, Kolkata` | ✅ with `[icu]`, ⚠️ without |
| Bengali | Bengali script | ICU `Any-Latin` | diacritics stripped, title case | `শেখ হাসিনা` → `Sekha Hasina`, `ফাতেমা` → `Phatema`, `আব্দুল করিম` → `Abdula Karima` | ❌ scholarly ISO 15919 only: inherent vowel kept (`Rahima`), `ph` for `f`, `s` for `sh`, `j` for `z`; 7% agreement with Wikidata |
| Punjabi, Gujarati, Tamil, Telugu, Kannada, Malayalam, Sinhala | Indic scripts | ICU `Any-Latin` | diacritics stripped, title case | `Sri` | ⚠️ ISO-style, no inherent-vowel handling; Tamil 15% agreement with Wikidata |
| Thai | Thai script | `pythainlp` RTGS engine | title case | `Somchai Prayut Chanocha` | ⚠️ 60 µs/name; long compounds and `ทร` clusters come out wrong (`Chinwatn` for Chinnawat); 13% agreement with Wikidata |
| Lao, Khmer, Burmese, Tibetan, Mongolian script | respective scripts | ICU `Any-Latin` | | | ⚠️ readable, not standardised |
| Armenian, Georgian | respective scripts | `anyascii` | | `Erevan, Armen Sargsyan, Tbilisi` | ✅ |
| Amharic, Tigrinya | Ethiopic | `anyascii` | apostrophes removed | `Adis Abeba, Aby Ahmed` | ⚠️ |
| Vietnamese, Polish, German, Turkish, all Latin-script languages | Latin | NFKD + own map | `Đ Ł ß Ø Þ Æ Œ ı` and friends, which NFKD leaves alone | `Nguyen Van Duc, Lodz, Muller, Istanbul` | ✅ |
| Symbols, emoji, typographic punctuation | | final layer | `“ ” – — …` → ASCII, `№`→`No`, `€`→`EUR`, `١٢٣`→`123`, emoji removed, whitespace collapsed | | ✅ |
| Mixed-script input | per run | router | each run goes to its own engine, then one ASCII pass | `Zeleнський in Київ` → `Zelenskyi in Kyiv` | ✅ |

### Measured agreement with Wikidata

`scripts/evaluate_wikidata.py` romanizes the native-language labels of about 1,400 people per
language from Wikidata and compares them with the English labels. **Token recall** is the share of
English-label words that appear verbatim in our output. English labels are not gold
transliterations: they follow anglicised habits (`Vladimir` for `Володимир`), nicknames and
drop patronymics, so these are floors for comparing languages, not accuracies.

| Language | Token recall | Char similarity | Reading |
|---|---|---|---|
| Serbian, Macedonian, Bulgarian, Georgian | 84–88% | 96% | as good as the labels allow |
| Urdu, Persian | 79–83% | 91–94% | good (Persian-script model of `arabic-name-romanizer` 0.2) |
| Greek, Arabic | 76–77% | 91–95% | good |
| Chinese, Japanese (with `lang="ja"`), Armenian | 67–70% | 55–92% | good; Japanese order differs from Wikidata |
| Ukrainian, Kazakh, Hindi, Russian | 54–61% | 53–90% | mostly convention differences (`Zelenskyi` vs `Zelenskyy`) |
| Belarusian | 44% | 76% | Wikidata mixes Belarusian and Russian forms |
| Korean | 26% | 79% | see the table above |
| Tamil, Thai | 13–15% | 74–75% | ISO-style output, not everyday spelling |
| Bengali | 7% | 79% | not supported in practice |
| Hebrew | 0% | 59% | not supported |

### Known ambiguities

- **Han without kana is read as Chinese.** `山田太郎` alone gives `Shan Tian Tai Lang`; pass
  `lang="ja"` when you know the source is Japanese.
- **Cyrillic without language-specific letters is read as Russian.** `Володимир` has no Ukrainian-only
  letters and becomes `Volodimir`; with a surname (`Зеленський`) or `lang="uk"` it becomes `Volodymyr`.
  Likewise Bulgarian is only recognised by a vocalic `ъ` (`Търново`), so `Христо Ботев` needs `lang="bg"`
  to become `Hristo Botev` instead of `Khristo Botev`. When you know the language, pass it.
- **Names mode splits Chinese and Korean tokens that start with a common surname.** `田中` (Tanaka) is
  a Chinese surname too, so without `lang="ja"` it becomes `Tian Zhong`; `王府` becomes `Wang Fu`.
  Use `mode="text"` for running text.
- **Serbian written in Latin letters loses `Đ` → `D`** (`Đorđe` → `Dorde`) while the Cyrillic form gives
  `Djordje`: `Đ` is also Vietnamese, where `D` is right, and nothing in the text says which.
- **Dari and Iranian Persian share an alphabet.** Both are detected as `fa`; pass
  `arabic_country="AF"` for Afghan data to prefer `Ahmad Shah Massoud` over `Ahmed Shah Masoud`.
- **Hindi schwa deletion is a heuristic.** It gets `Tendulkar`, `Kolkata`, `Arvind` right and
  `Draupadi` wrong (`Draupdi`). Use `overrides` for the names that matter.

## API

```python
from romanizer import Romanizer, romanize, romanize_many, analyze

romanize("Київ")  # auto-detect
romanize("Володимир", lang="uk")  # language hint
romanize_many(["محمد", "أحمد", "Київ"])  # batches Arabic runs into one model call (20× faster)

r = Romanizer(
    mode="names",  # or "text": no surname splitting, no title-casing
    unsupported="fallback",  # or "raise" / "drop" for scripts with no engine (Hebrew)
    arabic_country="EG",  # regional spelling preference for Arabic names
    overrides={  # exact spellings that win over every engine, whole input or any token
        "Зеленський": "Zelenskyy",
        "محمد صلاح": "Mo Salah",
    },
)
r.romanize("Володимир Зеленський")  # 'Volodymyr Zelenskyy'

a = analyze("Zeleнський in Київ")
a.output  # 'Zelenskyi in Kyiv'
a.segments[1].script  # 'Cyrillic'
a.segments[1].lang  # 'uk'
a.segments[1].engine  # 'table-cyrillic'
a.segments[1].supported  # True
```

Command line:

```bash
romanize "Володимир Зеленський" "محمد صلاح"
romanize --lang uk < names.txt > names_latin.txt
romanize --overrides known_spellings.tsv < names.txt      # 'source<TAB>latin' per line
romanize --analyze "Zeleнський in Київ"
```

## Speed

Single thread, Apple M-series, warm engines, microseconds per string:

| Input | `romanize()` | `romanize_many()` |
|---|---|---|
| ASCII (fast path) | 0.9 | 0.6 |
| Latin with diacritics | 3 | 3 |
| Ukrainian / Russian | 7–9 | 7–8 |
| Greek | 10 | 9 |
| Chinese / Japanese / Korean | 6–13 | 6–12 |
| Hindi (ICU) | 8 | 7 |
| Mixed `Zeleнський in Київ` | 7 | 6 |
| Thai | 58 | 58 |
| Arabic, name in dictionary | 400 | 80 |
| Arabic, name unknown (ONNX model) | 1430 | 40 |

Ten million mixed records take minutes on one core. Always use `romanize_many` for Arabic.

## Output guarantees

1. Only printable ASCII leaves the library. Every character is either mapped by a table, decomposed
   by NFKD with its marks removed, mapped by an explicit table for letters NFKD cannot decompose,
   or, as a last resort, transliterated by `anyascii`.
2. Backticks, carets and tildes never survive: they only ever come from transliteration schemes.
3. Whitespace is collapsed to single spaces and trimmed.
4. Casing of cased scripts (Cyrillic, Greek, Latin) is preserved: `ЩЕРБАК` → `SHCHERBAK`,
   `Щербак` → `Shcherbak`. Caseless scripts are title-cased in names mode.

## Contributing

```bash
uv sync
uv run pytest          # 100% line and branch coverage is enforced
uv run ruff check .
uv run mypy            # strict
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the layout, the checklist for adding a language, and
the pull request expectations. Security reports: [SECURITY.md](SECURITY.md). Changes:
[CHANGELOG.md](CHANGELOG.md).

## License

MIT for this package (see [LICENSE](LICENSE); dependency licenses in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)). Optional engines keep their own licenses: `anyascii` (ISC), `pypinyin` (MIT),
`PyICU` (MIT), `pythainlp` (Apache-2.0), `arabic-name-romanizer` (MIT), and in the `[cjk]` extra
`pykakasi` and `korean-romanizer`, which are **GPL-3.0-or-later**. If GPL is a problem for your
deployment, skip the `[cjk]` extra: Chinese, Japanese and Korean then fall back to `anyascii`.
