Metadata-Version: 2.4
Name: indic-error-taxonomy
Version: 0.2.0
Summary: Script-level error taxonomy for Indic OCR/VLM document transcription output.
License-Expression: MIT
Project-URL: Homepage, https://github.com/Amish098/indic-error-taxonomy
Project-URL: Repository, https://github.com/Amish098/indic-error-taxonomy
Project-URL: Issues, https://github.com/Amish098/indic-error-taxonomy/issues
Keywords: ocr,indic,devanagari,vision-language-model,error-analysis,document-understanding,unicode
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Natural Language :: Bengali
Classifier: Natural Language :: Hindi
Classifier: Natural Language :: Marathi
Classifier: Natural Language :: Nepali
Classifier: Natural Language :: Panjabi
Classifier: Natural Language :: Tamil
Classifier: Natural Language :: Telugu
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rapidfuzz>=3.9
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: hypothesis>=6; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: regex>=2024.0; extra == "dev"
Dynamic: license-file

# indic-error-taxonomy

**What kind of mistake did the OCR make?** — not just how many.

Character Error Rate tells you a system is 12% wrong. It does not tell you
whether that 12% is dropped vowel signs (a post-corrector could fix most of
it) or the wrong script entirely (nothing can fix it — re-run the page).
Those need opposite decisions and CER cannot distinguish them.

This library compares a reference transcription against a system's output and
names every difference, across **nine Indic scripts**.

```python
from indic_error_taxonomy import analyze

result = analyze(reference, hypothesis, "hi")
print(result.report())
```

```
WORD TIER — how much of the page the model attempted
  reference words        13
  word_correct                6   46.2%
  word_misread                3   23.1%
  word_deleted                4   30.8%
  attempted (correct+misread) 69.2%

CHARACTER TIER — what went wrong where it was attempting
  (high-confidence only)
  matra_missing               1
  matra_wrong                 1
  (31 further errors below high confidence — inspect before use)

LINE TIER — layout / reading order
  matched lines          3
  out of order           1
  kendall tau            +0.333
```

---

## Install

```bash
pip install indic-error-taxonomy
```

One runtime dependency (`rapidfuzz`). No models, no downloads, no GPU.

Supported languages: `hi` `mr` `ne` (Devanagari) · `bn` `as` (Bengali) ·
`ta` (Tamil) · `te` (Telugu) · `kn` (Kannada) · `ml` (Malayalam) ·
`gu` (Gujarati) · `pa` (Gurmukhi) · `or` (Odia)

---

## Three tiers, and the order to read them in

| tier | question | when it matters |
|---|---|---|
| **line** | did it read the page in the right order? | multi-column pages, tables, footnotes |
| **word** | is this the right word, or a different one? | always — see below |
| **character** | what went wrong inside a word it attempted? | the detailed diagnosis |

**Read them in that order.** A page read perfectly but emitted in the wrong
sequence looks like catastrophic failure at the word and character tiers; only
the line tier says otherwise. And roughly **one in six** character errors sits
inside a word the system never actually attempted — the character label there
describes where the aligner anchored, not a real mistake. The word tier is
what tells you which is which.

```python
result.lines      # ReadingOrderResult — layout order
result.words      # WordLevelResult    — 5 outcomes per word
result.chars      # CategorizedResult  — 25 reportable categories, per edit
result.attempted  # share of words the system demonstrably read
```

---

## The tool tells you which of its own labels to trust

Not every category is equally reliable, and the library says so per instance
rather than making you find out by hand.

```python
for e in result.chars.errors:
    print(e.category, e.confidence, e.why)
```

```
matra_wrong      high   specific rule, uncontested, well-aligned
conjunct_broken  low    category agrees with a human only 20% of the time
                        (n=15); could also be other_deletion
base_confusion   low    the model wrote a different word here
                        ('பின்,' -> 'என்ன,')
```

```python
result.summary()                      # high-confidence counts only
result.chars.summary_by_confidence()  # the full breakdown
```

Confidence combines four measured signals: how often that category agrees
with a human annotator, whether competing rules could have claimed the same
edit, alignment coherence, and whether the surrounding word was attempted.

---

## The 25 character categories

They are a grid — **which part of the writing system** × **what happened to it**:

| | dropped | added | swapped | moved |
|---|---|---|---|---|
| **letter** | `base_lost` | `base_spurious` | `base_confusion` | — |
| **vowel sign** | `matra_missing` | `matra_spurious` | `matra_wrong` | `matra_misplaced` |
| **modifier** | `modifier_lost` | `modifier_spurious` | `modifier_wrong` | — |
| **conjunct join** | `conjunct_broken` | `conjunct_spurious` | n/a | — |
| **nukta** | `nukta_lost` | `nukta_spurious` | n/a | — |
| **punctuation** | `punctuation_lost` | `punctuation_spurious` | `punctuation_confusion` | — |
| **digit** | — | — | `digit_script_mix` | — |

Plus five that sit outside the grid, which is itself informative:
`base_mark_confusion` (crosses two rows), `script_confusion` and
`script_switch_failure` (whole-text), `hallucination` (multiple units), and
`other_substitution` / `other_deletion` / `other_insertion` (unclassified —
not findings).

`CATEGORIES` holds a 26th name, `reading_order_error`. It never fires: plain
Levenshtein has no transposition operation, so it cannot emit the
anagram-shaped block the rule looks for. Transposition is measured at the
line tier instead. The name is kept so the constant stays stable for callers
that iterate it.

```python
from indic_error_taxonomy.definitions import describe
print(describe("modifier_lost"))
```

Every category carries a plain description, the firing rule, a worked example,
the normative standard, and **what it is easily confused with**. Definitions
are executable: a test asserts every documented example really produces the
category it illustrates.

**Note: `matra` means a dependent vowel sign here**, not "any mark". An anusvara
(ं) is a `modifier`. Colloquial Hindi/Marathi uses मात्रा for both; this
library does not. Six of eight labelling errors in our validation were this.

---

## Grounded in Unicode, not in hand-typed tables

Categories that depend on a linguistic definition take it from the standard,
and tests enforce it on every run:

- **Character classes** are pinned to `Indic_Syllabic_Category`. Adding this
  check found 24 misclassified characters, including the VOCALIC RR/LL letters
  in four languages and Bengali khanda ta.
- **Conjunct clusters** follow UAX #29 rule GB9c / `Indic_Conjunct_Break`.
  This is why `conjunct_broken` never fires for Tamil, Kannada or Gurmukhi —
  Unicode assigns those scripts no conjunct linker.

---

## Validation

| check | result |
|---|---|
| Human agreement (194 blind labels) | **κ = 0.784** [0.722, 0.843], raw 82.0% |
| Synthetic injection, 3,000 known errors | 99.9% recall / 99.9% precision |
| …with 8 extra simultaneous errors | 99.6% |
| Held-out text from outside the corpus | identical per-category results |
| Edit-distance invariant | Σ edit sizes == Levenshtein distance, 2000/2000 |
| Localisation | 98.9% of errors land on an actionable span |
| Test suite | **519 tests** |

Injection sites are chosen from Unicode character *names*, independently of
the library's own tables, so the two can disagree — which is the point.

### Known limitations, stated plainly

- **`conjunct_broken` is not reliable.** 20% human agreement (n=15); ablation
  shows it competing with four other categories; it straddles akshara
  boundaries 10× more than average. Reported, never high-confidence, don't
  build on it.
- **`base_mark_confusion` is ambiguous** (58%, n=24). Partly the writing
  system's fault — a vowel sign is drawn attached to its consonant.
- **`hallucination`'s definition changed** after validation; re-validate
  before quoting it.
- **Single annotator**, no human–human ceiling, so κ is a lower bound of
  unknown tightness. Four languages have zero hand-checked errors.
- **The line tier is the least validated** — unit-tested, but not checked at
  corpus scale on real multi-column scans.
- **Word segmentation is whitespace-based.** A missing space merges two words.
- Three deliberate divergences from `Indic_Syllabic_Category` (Gujarati Khoja
  transliteration nuktas), documented and pinned by test.

---

## Also useful

```python
from indic_error_taxonomy import categorize_errors, align_words
from indic_error_taxonomy.reading_order import analyze_reading_order

categorize_errors(ref, hyp, "hi")      # character tier alone
align_words(ref, hyp, "hi")            # word tier alone
analyze_reading_order(ref, hyp)        # line tier alone
```

Errors carry a content-addressed `error_key`, so hand labels survive future
changes to the rules — label once, keep forever.

---

## Development

```bash
pip install -e ".[dev]"
pytest indic_error_taxonomy/tests -q
```

## License

MIT.
