Metadata-Version: 2.5
Name: arabic-name-romanizer
Version: 0.1.2
Summary: Offline Arabic → Latin personal-name transliteration (dictionary-first, tiny ONNX model fallback)
Project-URL: Homepage, https://github.com/unicef/arabic_name_romanizer
Project-URL: Repository, https://github.com/unicef/arabic_name_romanizer
Project-URL: Issues, https://github.com/unicef/arabic_name_romanizer/issues
Project-URL: Changelog, https://github.com/unicef/arabic_name_romanizer/blob/main/CHANGELOG.md
Author-email: Jan Romaniak <romaniakjan@gmail.com>
License: MIT
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Keywords: arabic,names,offline,onnx,romanization,transliteration
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: Arabic
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: <3.15,>=3.11
Requires-Dist: numpy>=1.26
Requires-Dist: onnxruntime>=1.17
Description-Content-Type: text/markdown

# arabic-name-romanizer

Offline, CPU-only transliteration of **Arabic personal names** (a given name, a
surname, or a full name) into their **likely real-world Latin spelling**.

```python
from arabic_name_romanizer import transliterate

transliterate("محمد صلاح")  # 'Mohamed Salah'
transliterate("حبيبة عطيفي")  # 'Habiba Atifi'
transliterate("عبد الرحمن", top_k=3)  # [Candidate('Abdulrahman', ...), Candidate('Abderrahmane', ...), ...]
transliterate("عبد القادر", country="DZ")  # 'Abdelkader'
```

It produces likely or common spellings, not legally authoritative
identity-document transliterations. It is not a general Arabic transliterator
and not a translator.

## Why Arabic → Latin names are ambiguous

Arabic writing omits short vowels, so `محمد` is attested as Mohamed, Mohammed,
Muhammad and Muhammed. Conventions differ by region: Egypt writes `Gamal` where
the Gulf writes `Jamal`; Algeria, Morocco and Tunisia follow French spelling
(`Youcef`, `Abdelkader`, `Benali`) where the Mashreq follows English
(`Yusuf`, `Abdulqadir`, `Bin Ali`). One Arabic name legitimately has several
correct Latin spellings, and the library returns ranked candidates rather than
one "true" form.

## How it works

```mermaid
flowchart TD
    IN["Arabic input<br/>e.g. محمد عبد الرحمن الزهراني<br/>+ optional country / role hint"] --> NORM["Normalization<br/>NFKC, strip harakat and tatweel,<br/>unify Persian letters, punctuation → space"]
    NORM --> WHOLE{"Whole string known?<br/>user dictionary, then shipped index"}
    WHOLE -- "yes, trusted (count ≥ 3)" --> OUT
    WHOLE -- no --> SPLIT["Split into units<br/>whitespace tokens; particles merge:<br/>عبد الرحمن · بن علي · سيف الدين"]
    SPLIT --> UNIT{"Each unit"}
    UNIT --> UD["User dictionary<br/>(TSV, always wins)"]
    UD -- miss --> IDX["Shipped index<br/>SQLite, ~60k attested spellings<br/>from Wikidata + Algerian corpora<br/>ranked by country → region → all"]
    IDX -- miss --> ML["ML fallback<br/>char-level GRU seq2seq with attention<br/>1.7M params, ONNX Runtime, CPU<br/>input: country + role tokens + letters<br/>batched beam search, width 2"]
    ML -- "no model installed" --> RULES["Rule-based romanization<br/>(last resort)"]
    UD --> RANK
    IDX --> RANK
    ML --> RANK
    RULES --> RANK
    RANK["Combine units<br/>score = product of unit scores,<br/>dictionary: frequency × trust,<br/>model: exp(mean log-prob)"] --> OUT["Top-k candidates<br/>text, score, source per unit"]
```

No ML runs for a name whose units are all known: the dictionary answers, in
0.1–0.4 ms. The network only sees units nobody in the data has ever spelled.
Training (PyTorch, `training/`) is offline; the package ships the exported
ONNX graphs and needs only `numpy` and `onnxruntime`.

## Why dictionary first

Known real-world spellings beat generated ones. The pipeline is:

```text
Arabic input
     |
     v
normalize
     |
     v
level 0: user override dictionary, whole string     (if hit: done)
     |
     v
level 1: built-in index, whole string               (if hit with count >= 3: done)
     |
     v
split on whitespace into units; merge particle + next token
(عبد الرحمن, أبو بكر, بن علي, آل سعود are one unit each)
     |
     v
for each unit:  user dictionary → built-in unit index → neural model (beam search) → rules
     |
     v
combine per-unit candidates (score product) → rank → join with spaces
```

The built-in index holds attested spellings from Wikidata; the neural model is a
1.7M-parameter GRU sequence-to-sequence model run with ONNX Runtime and is only
consulted for name units the index has never seen.

## Install

```bash
pip install arabic-name-romanizer      # or: pip install dist/arabic_name_romanizer-0.1.0-py3-none-any.whl
```

Runtime dependencies: `onnxruntime` and `numpy`. PyTorch is only needed for
training. Requires Python 3.11–3.14. The wheel is 9.3 MB compressed, 16 MB
installed (8.6 MB SQLite index, 7.1 MB ONNX graphs).

## Python API

```python
from arabic_name_romanizer import Transliterator, UserDictionary, transliterate, transliterate_many

transliterate("محمد")  # 'Mohamed'
transliterate("محمد", country="SA")  # 'Mohammed'
transliterate("حسن", role="surname")  # role hint: 'given' | 'surname'
transliterate_many(["محمد", "علي"])  # ['Mohamed', 'Ali']

for c in transliterate("محمد زقزوق", top_k=3):
    print(c.text, round(c.score, 3), c.source)  # source: dictionary | model | rules | mixed | user | passthrough
    for u in c.units:  # per-unit provenance
        print("  ", u.arabic, "->", u.latin, u.source, u.count)

# your own attested spellings always win
t = Transliterator(user_dictionary=UserDictionary.from_pairs([("حبيبة عطيفي", "Habiba Atify")]))
t.transliterate("حبيبة عطيفي")  # 'Habiba Atify'
```

`score` is a ranking heuristic (relative attested frequency for dictionary hits,
exp of the mean per-character log-probability for the model), not a calibrated
probability. Non-Arabic input is passed through unchanged.

## CLI

```bash
arabic-name-romanize "محمد صلاح"
arabic-name-romanize "محمد صلاح" --top-k 5
arabic-name-romanize "عبد القادر" --country DZ
arabic-name-romanize --json --top-k 3 "زقزوق"
cat names.txt | arabic-name-romanize --user-dict my_names.tsv     # one name per line on stdin
```

The user dictionary is a TSV: `arabic<TAB>latin[<TAB>count[<TAB>country]]`.

## CPU-only inference

Inference needs no GPU and no PyTorch. The model ships as two ONNX graphs
(encoder, decoder step) plus a character vocabulary. Decoding is batched beam
search across all name units of a batch; identical units are decoded once.
Output is deterministic for a fixed model file, ONNX Runtime version and CPU;
floating-point differences across CPUs can reorder near-tied candidates.

## Training data

| source | license | use |
|---|---|---|
| Wikidata: humans with citizenship in 22 Arabic-speaking countries, Arabic + English/French labels, given-name and family-name items | CC0 | primary: 46,245 name pairs, 19,535 unique name units |
| Algerian Name Transcription Corpus (Zerrouki) + DziriNames benchmark | CC0 / MIT | 11,172 Algerian (French-convention) pairs; +3 points EM@1 on Algerian names |
| google/transliteration ar2en | Apache-2.0 | not used: mostly foreign names written in Arabic |
| ANETAC | CC BY-NC 4.0 | research only, never in the default build |

Filters remove titles, regnal ordinals, academic romanizations with macrons and
length mismatches. Units are aligned to Latin tokens positionally, or by a small
DP aligner when a label drops a middle name or spells a particle unit as several
tokens (`عبد الرحمن → Abd al Rahman`); 99.9% of accepted pairs yield unit pairs.
People born before 1900 are excluded from the shipped index and training. Full
details in `DATA_LICENSES.md` and `DESIGN.md`.

## What the errors look like

Measured on 5,000 names composed from a hand-written lexicon (`data/llm_test/names.jsonl`,
country hint on, full report in `artifacts/llm_test/report.md`). A name counts as
correct only if every unit is one of the lexicon's accepted spellings; 3,923 of the
11,902 unit occurrences (33%) were not. Grouped by what actually differs:

| type of difference | share of wrong units | why it happens | example |
|---|---|---|---|
| vowels only | 37% | short vowels are not written in Arabic; both spellings are attested | ماجد → `Majd` vs `Majid`, بناني → `Bennani` vs `Bannani` |
| spacing / hyphen / case | 28% | same letters, different typography; Wikidata favours hyphens | الحارثي → `Al-Harthi` vs `Al Harthi`, عبد الحكيم → `Abdel Hakim` vs `Abdulhakim` |
| a consonant differs | 13% | rare regional spelling, a remaining data misalignment, or a gap in the reference list | قاسم → `Gacem` vs `Kacem` (both real in Algeria), الأشقر → `Achkar` vs `Al Ashqar` |
| single vs double consonant | 6% | the shadda (gemination mark) is not written | الحسن → `Al Hasan` vs `Al Hassan` |
| French vs English convention | 5% | ou/u, ch/sh, dj/j, c/k, k/q leaked across the region border | بشير → `Bashir` vs `Bachir` for Tunisia |
| article form (`Al` / `El` / `Al-` / dropped) | 4% | regional habit | الخالدي → `Khalidi` vs `Al Khalidi` |
| عبد compound form | 3% | `Abdul` / `Abdel` / `Abd al` are all in use | عبد الكافي → `Abd al-Kafi` vs `Abdel Kafi` |
| vowels and convention together | 2% | both of the above at once | الجزيري → `El Geziry` vs `Al Jaziri` |
| different number of tokens | 2% | a compound written joined or split | سيد أحمد → `Sid Ahmed` vs `Sidahmed` |

About two thirds of the differences are therefore another correct spelling of the
same name, not a wrong name. On this set the model is almost never the source:
98% of the units are found in the dictionary, so what is measured is which attested
variant gets ranked first. The `country` hint is worth two points of acceptable
matches and eight points once typography is ignored.

## Benchmark results

See `DESIGN.md` (section "Experiment log") for the full tables. Test split
(never seen in training, split by whole-name key): 2,472 full names and 4,225
name-unit groups, of which 807 units never occur in the training data. All rows
are measured on this (round 4, cleaned) test set.

| system | unseen-unit EM@1 | unseen-unit EM@3 | unseen-unit CER | bulk names/s |
|---|---|---|---|---|
| rule-based romanization | 0.103 | 0.103 | 0.346 | - |
| Transformer 5.6M (ONNX, beam 4, round 2) | 0.357 | 0.531 | 0.200 | 24 |
| GRU 1.7M, round 2 (ONNX FP32, beam 2) | 0.387 | 0.492 | 0.197 | 1727–2829 |
| GRU 1.7M + role token, round 3 (ONNX FP32, beam 2) | 0.408 | 0.504 | 0.184 | 2625 |
| **GRU 1.7M + role token, cleaned data, round 4 (ONNX FP32, beam 2, shipped)** | **0.420** | 0.519 | 0.168 | **2742** |
| GRU 1.7M + role token, round 4 (beam 4) | | | | 1403 |
| GRU 3.8M + role token (d_model 384) | 0.400 | 0.509 | 0.185 | not shipped |

Exact match is strict against the attested test labels; on full names 82% of
top-1 outputs are attested spellings of every unit ("expanded" match), and the
dictionary answers directly for every name whose units are known.

An independent set of 5,000 names composed from a hand-written lexicon
(`scripts/llm_lexicon.py`, common names only, country hint on) gives acceptable
EM@1 0.367, or 0.522 when hyphen/space/case differences are ignored, CER 0.088.
Two thirds of its remaining "errors" are a different attested variant (vowel
choice, `Al-` vs `Al `, `Abdel` vs `Abdul`); see `DESIGN.md` round 4 for the
breakdown and `artifacts/llm_test/report.md` for every failure. The model
also receives the unit's role (given name / surname, from its position or the
`role` argument), which is worth about one point on unknown units.

Throughput on an Apple M4, single process, machine idle, measured by
`scripts/benchmark.py`:

| measurement | p50 ms | p99 ms | names/s |
|---|---|---|---|
| dictionary hit, single call | 0.14 | 0.75 | 5,539 |
| full system, single call | 0.14 | 0.74 | 5,244 |
| full system, batch 256 | 35.0 per batch | 39.7 | 7,386 |
| model only, bulk, batch 256, beam 2 | | | 2,742 |

The project requirement was at least 1,000 names/s model-only bulk throughput;
the shipped configuration passes it with margin. Run-to-run variance on a laptop
is about ±30%.

## Limitations

- Foreign names written in Arabic (`جون سميث`) are out of scope; they will be
  transliterated as if Arabic.
- Historical figures are excluded from the data; academic romanizations
  (`Muḥammad`) are never produced.
- A dictionary entry seen once is one observation, not the truth. A whole-name
  entry below the trust threshold (3) is ranked jointly with the per-unit
  candidates; a unit entry is still preferred over the model, because attested
  spellings beat generated ones 9 points of the time (`Transliterator(model_below_trust=True)`
  changes this and gains 1.4 points of strict exact match).
- Scores are not calibrated probabilities.
- Regional spelling depends on the `country` hint; without it the most frequent
  attested spelling across all countries wins.

## Reproducing

```bash
uv sync
uv run python scripts/download_data.py --with-dzirinames
uv run python scripts/build_dataset.py
uv run python training/train.py --config training/configs/gru_role.yaml --run-dir artifacts/runs/gru_role
uv run python training/evaluate.py --model artifacts/runs/gru_role --beam-width 2 --out artifacts/eval/gru_role.json
uv run python scripts/export_onnx.py --run artifacts/runs/gru_role --out artifacts/onnx/gru_role --beam-width 2
uv run python scripts/build_lookup.py
uv run python scripts/benchmark.py --model-dir src/arabic_name_romanizer/resources
uv run pytest
```

## Licensing

Code: MIT (`LICENSE`). The shipped index and weights derive from Wikidata (CC0),
the Algerian Name Transcription Corpus (CC0) and the DziriNames benchmark (MIT,
which in turn credits the Amoura Kaggle dataset, MIT). The required MIT notices
are in `THIRD_PARTY_NOTICES.md`, which is packaged into the wheel's dist-info
directory; keep that file with any redistribution. Details per source in
`DATA_LICENSES.md`.

## CPU only

Inference, the test suite and the benchmarks run on CPU: ONNX Runtime is created
with `CPUExecutionProvider` only, the training smoke tests pin `device: cpu`, and
`scripts/benchmark.py` measures the shipped ONNX files, never a torch model.
A GPU is neither required nor used at runtime; `torch` is a dev dependency for
training only.
