Metadata-Version: 2.4
Name: arabic-lint
Version: 0.5.0
Summary: Find Arabic text corrupted by reshape+bidi before it was stored. Zero dependencies.
Author-email: Syamjith NK <hello@syamjithnk.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Syamjith-NK/arabic-lint
Project-URL: Issues, https://github.com/Syamjith-NK/arabic-lint/issues
Keywords: arabic,unicode,bidi,linter,i18n,rtl,text-shaping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Arabic
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# arabic-lint

Finds Arabic text that was corrupted **before it was stored** — in your JSON, your
localisation files, your database exports, your source code.

```bash
pip install arabic-lint
arabic-lint ./src
```

```
src/strings.json:3:20: 21 Arabic presentation forms stored  [UNSAFE TO AUTO-FIX]
    found     : ﺓﺪﺤﺘﻤﻟﺍ ﺔﻴﺑﺮﻌﻟﺍ ﺕﺍﺭﺎﻣﻹﺍ
    would be  : اإلمارات العربية المتحدة
    contains a lam-alef ligature; NFKC decomposition reorders the pair, so this
    recovery is wrong even though it looks like Arabic

3 corrupted span(s) in 1 file(s); 1 cannot be auto-fixed safely.
```

Exit code 1 when anything is found, so it drops into CI unchanged.
MIT. **Zero dependencies.** Python 3.9+.

## Which case are you in?

The findings below all depend on what draws your text, and that is not knowable by
reading code: it depends on your installed matplotlib version and on whether your
Pillow was built with Raqm, which is a property of the wheel rather than the version.

```bash
arabic-lint --doctor
```

```
  matplotlib 3.11.0
      pre-shaping BREAKS text here
      3.11 and later shape text with libraqm and apply bidi themselves

  Pillow 12.2.0
      pre-shaping BREAKS text here
      this build has Raqm, so ImageDraw.text() shapes and reorders for you. Note
      this is a property of the BUILD: the same version on another machine can
      answer differently

Verdict: on matplotlib and Pillow, remove the reshape/bidi step and pass the
logical string straight through. Leaving it in reverses the text, silently.
```

If one renderer shapes and the other does not, it says so, because then a single
shared helper cannot be correct for both.

## What counts as corruption, and what does not

Arabic Presentation Forms-A is **interleaved**: positional glyph forms and ordinary
semantic characters share the same block. The ornate parentheses ﴾ ﴿ that enclose a
Quranic quotation live at U+FD3E/U+FD3F, in the middle of it, and honorific ligatures
like ﵀ sit just above them. Those are characters people type on purpose.

Treating the whole block as a corruption signal reports Islamic heritage text as
broken. Measured against public corpora on Hugging Face on 9 September 2026, the
naive rule flagged **35.6%** of one heritage OCR corpus and **5.5%** of another. Every
hit was a Quranic quotation mark or an honorific. With the block classified properly,
both read **0%**.

So the classification is derived rather than tabulated: a contextual shaping artefact
is exactly a codepoint Unicode names `... ISOLATED/INITIAL/MEDIAL/FINAL FORM`.
Anything else in the block is deliberate. New Unicode additions classify themselves.

## Two checks

**Stored corruption** — Arabic presentation forms that were already written to disk.

**Source risk** (Python files) — the reshape+bidi recipe feeding a renderer that
already shapes, which corrupts at *render* time before anything is stored:

```
cogs/bear_track.py:715:16: pre-shaped text passed to matplotlib  [RENDERS REVERSED]
    return _bidi_get_display(_arabic_reshaper.reshape(text))
    matplotlib >= 3.11 shapes text with libraqm and applies bidi itself, so this
    reorders an already-reordered string and the text renders reversed
```

The recipe appears in **3,168 indexed Python files on GitHub** (measured 2026-09-08),
and flagging all of them would be worthless, because whether it is a bug depends
entirely on what draws the text:

| renderer | shapes and reorders? | verdict |
|---|---|---|
| matplotlib >= 3.11 | yes | pre-shaping **reverses** the text |
| Pillow built with Raqm | yes | pre-shaping **reverses** the text |
| Pillow without Raqm | no | pre-shaping is **required** |
| ReportLab, fpdf | no | pre-shaping is **required** |
| `print()` to a terminal | terminal-dependent | not this tool's call |

So the source check stays silent unless a shaping renderer is imported *and*
something in the file actually draws with it. It also follows import aliases, follows
the recipe when `reshape()` and `get_display()` are on separate lines, names the
renderer that actually draws rather than the first one imported, and ignores a
pre-shaping helper that a script never calls.

Validated against six real repositories found by code search: it flags the three that
are genuinely broken, and stays silent on a ReportLab project, a dead helper in an
unrelated benchmark, and a script that only prints to a terminal. Pass `--no-source`
to turn it off.

## `--fix`, and why it exists here but not for stored text

The same package refuses to repair one kind of damage and offers to repair the other,
which sounds inconsistent until you look at what each one is.

**Stored corruption is not safely repairable.** Undoing it round-trips exactly until the
text contains a lam-alef ligature, and then <span dir="rtl">السلام</span> comes back as
<span dir="rtl">السالم</span>: a real word, a different word, one that survives a
proofread. So the stored check reports and never rewrites.

**Source is the opposite.** Where the whole recipe sits inside one expression, deleting
it is exactly correct:

```diff
- return get_display(arabic_reshaper.reshape(text))
+ return text
```

`arabic-lint . --fix` makes that edit in place, then refuses to write if the result
would not parse.

It does **not** touch the split form:

```python
reshaped  = arabic_reshaper.reshape(segment)
processed = get_display(reshaped)        # NOT auto-fixable
```

Rewriting the second line to `processed = reshaped` would remove the reordering and
leave the shaping applied. That is still wrong, and wrong in a way that looks fixed.
Both lines have to go, and which other code reads `reshaped` is not knowable from that
expression. So it says so and leaves the file alone.

> **Why this exists:** the recipe below appears in **~1,160 indexed files on
> GitHub** (measured 2026-09-04; the figure drifts as GitHub reindexes), and on
> matplotlib 3.11 it now renders Arabic *backwards* with no error at all.
> [The Arabic fix everyone recommends is now the bug](https://syamjith-nk.github.io/arabic-reshape-bidi-is-now-the-bug/) — the measurements, and
> what happened when it was filed against Pillow and matplotlib.

## Check it yourself

Do not take the claim on trust — `verify_mpl311.py` re-measures it on your machine:

```bash
pip install matplotlib arabic-reshaper python-bidi
python3 verify_mpl311.py
```

It renders each test string twice, once as typed and once through the workaround,
and reports the mean absolute pixel difference against a same-string control. On
matplotlib 3.11.0 with Pillow 12.2.0 (Raqm enabled) all five strings render
differently, against a control of exactly `0.000`.

The pixel numbers depend on your font and size, so treat them as a signal rather
than a constant. The evidence that needs no renderer at all is the codepoint
count the script also prints:

```
emirates    8 →  7 codepoints · 7 in the Arabic Presentation Forms block  ← lam-alef decomposed
```

`الإمارات` loses a codepoint because lam-alef is one codepoint that decomposes
into two, and every output character lands in a block that typed Arabic never
contains.

## What it actually detects

The most widely copied recipe for "making Arabic work" in Python is:

```python
text = get_display(arabic_reshaper.reshape(text))
```

Those two calls do the job a text engine is supposed to do: substitute each letter
for its contextual *presentation form*, and reorder the string into visual order.
If your renderer already does complex text layout — Pillow with Raqm, matplotlib,
any browser — the work happens twice and the output is wrong.

The real damage is when that string gets **written back**: to a config, an export,
a translation file. Now the corruption is at rest and every downstream reader
inherits it. It renders as clean-looking Arabic, so nobody who does not read the
script will ever notice.

The signature is unambiguous: Arabic **presentation form** codepoints in stored
text — all of Forms-B (U+FE70–U+FEFF), plus the positional forms in Forms-A
(U+FB50–U+FDFF). Those exist for legacy-encoding compatibility; correctly
authored modern Arabic never contains them.

The Arabic **word ligatures** ﷲ ﷺ ﷻ ﷽ (U+FDF0–U+FDFD) are deliberately
excluded — people type those on purpose, and flagging them would mark correct
religious and formal text as corrupt.

Forms-A matters more than it looks. Persian and Urdu share most of their
alphabet with Arabic, so most reshaped Persian already trips Forms-B — but it
was being *under-counted*, and words built only from Persian-specific letters
(گچ, چپ, گپ, پژ, پی, and a bare Farsi yeh) have no Forms-B mapping at all and
were missed outright.

## Why it reports instead of fixing

You would think you could just undo it: `NFKC` maps every presentation form back
to its base letter, and reversing undoes the visual reordering. That round-trips
exactly — **until the span contains a lam-alef ligature**.

A lam-alef ligature (`لا`, `لأ`, `لإ`, `لآ`) is *one* codepoint standing for *two*
letters. NFKC expands it in logical order while the text around it is still in
visual order, so the pair comes out reversed relative to its neighbours:

| original | naive "fix" | |
|---|---|---|
| `الإمارات` | `اإلمارات` | not a word |
| `السلام` | `السالم` | a **real but different** word |

That second row is the whole reason this tool exists rather than a `sed` command.
The output is still pronounceable Arabic, so it survives a proofread — and the
definite article followed by alef is one of the most common sequences in the
language, so this is not a corner case.

`arabic-lint` shows you the candidate recovery and tells you when it is unsafe.
It never rewrites your files.

## Verification

- **10/10 tests**, no runtime dependency on `arabic_reshaper` or `python-bidi`
  (fixtures are recorded from a real run of both).
- Block boundaries were **measured, not assumed**: over a wide Arabic sample,
  `arabic_reshaper` 3.0.0 emits 53 distinct codepoints from Presentation Forms-B
  and never emits U+FEFF.
- **Validated against 3,826 real files** — the false positives that scan found are
  now regression tests:
  - **U+FEFF** sits inside Forms-B but is the byte order mark. Excluded.
  - **Presentation Forms-A is deliberately not a signal.** The reshaper emits
    exactly one codepoint from it (U+FDF2, the Allah ligature), and that character
    — like `ﷺ` U+FDFA, `ﷻ` U+FDFB and `﷽` U+FDFD — is used *intentionally* in
    ordinary Arabic writing. Treating the block as corruption flags correct
    religious and formal text.

## Known limits

- A document whose only Arabic is a standalone Allah ligature is missed. That is
  the deliberate trade above; any corrupted phrase around it still trips Forms-B.
- The recovery direction assumes bidi was applied. Text that was reshaped but
  *not* reordered recovers reversed. The tool shows you the candidate so you can
  see which case you have; it does not guess.
- It detects corruption that is *already stored*. It cannot tell you whether your
  rendering pipeline is about to create some — for that, check
  `PIL.features.check("raqm")` at runtime in the environment doing the rendering.

## Related

Part of a series measuring where Arabic silently breaks in software.
See also [`arabic-tts-frontend`](https://pypi.org/project/arabic-tts-frontend/) —
numerals, dates and currency converted to spoken Arabic before synthesis.
