Metadata-Version: 2.4
Name: medwer
Version: 1.0.0
Summary: Reproducible, model-free medical-ASR evaluation: pinned Whisper normalizer + phrase-aware medical-WER + fixed term list.
License: MIT
Project-URL: Repository, https://github.com/Nordis-Tech/medwer
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: whisper-normalizer==0.1.12
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# MedWER

MedWER is a **reproducible, model-free evaluation protocol for medical ASR**: a
pinned deterministic text normalizer, a phrase-aware **medical-WER**, and a
fixed, license-clean medical term list.

It is a protocol and a tool, not a new metric concept. Entity-weighted WER variants are
well studied; what they typically depend on is an eval-time NER model or cloud API,
which makes the denominator irreproducible. MedWER's denominator is a **fixed, typed
term list** — the same strings for everyone, forever, no model in the loop.

## Why

- **Deterministic.** Same inputs, same number, on any machine. No NER model, no API,
  no versioned black box in the metric.
- **Pinned.** Normalization is Whisper's `EnglishTextNormalizer` at an exact pinned
  version, locked by committed golden fixtures the test suite reproduces in CI — an
  upstream behavior change fails the build instead of silently moving numbers.
- **License-clean.** Every term-list source permits open commercial redistribution
  (see [NOTICE](https://github.com/Nordis-Tech/medwer/blob/main/NOTICE)); the list ships in the wheel.
- **Phrase-honest.** Multi-token terms ("insulin glargine") are matched greedily
  longest-first and scored as single atomic units, so a missed phrase counts as one
  miss, not free partial credit. The same applies to spelled-letter entries ("c h f")
  in a custom list; the bundled list carries none.

## Install

```
pip install medwer
```

## CLI

```
medwer score --refs refs.jsonl --hyps hyps.jsonl [--terms path/to/terms.txt]
```

Both files: one JSON object per line, `{"id": ..., "text": ...}`. Hypotheses join to
references by `id`; the two id sets must match exactly, and any mismatch is an error.
Output is a single JSON object (rates are fractions, `null` when the corresponding
reference-word denominator is zero):

```
{"utts": 2620, "ref_words": 52576, "med_ref_words": 4310, "wer": 0.042, "medical_wer": 0.31}
```

## Python API

```python
from medwer import Normalizer, Scorer

scorer = Scorer()                      # bundled 19,373-entry medical term list
result = scorer.corpus(refs, hyps)     # {"wer": ..., "medical_wer": ..., ...}

norm = Normalizer()                    # stock Whisper normalizer, number folding on
norm("Give five hundred milligrams.")  # -> "give 500 milligrams"
```

## The protocol

1. **Normalize** both sides with Whisper's `EnglishTextNormalizer`, unmodified —
   lowercasing, punctuation/diacritic removal, contraction and GB→US spelling
   standardization, and number-word folding ("five hundred" → "500"), so digit/word
   surface differences never score as errors. References are expected in written
   clinical form ("500 mg"). `Normalizer(fold_numbers=False)` (CLI:
   `--no-fold-numbers`) disables the folding step for corpora whose digit/word
   surface must be scored strictly.
2. **WER**: token-level Levenshtein over whitespace tokens.
3. **Medical-WER**: filter both sides to term-list content (greedy longest-match,
   phrases atomic), then WER on what remains.
4. **Corpus aggregation**: micro-average (total edits / total reference words);
   samples whose normalized reference is empty are skipped.

Scope: normalization is Latin-script (ASCII, Latin accents, NFKD-compatibility
forms). Unit words are not folded ("milligrams" and "mg" are distinct tokens), so
the written-form unit convention of the references is part of the protocol surface.

## The term list

`medwer/data/medical_terms.txt` — 19,373 entries (11,742 multi-word) across drugs,
diagnoses, symptoms, and injury mechanisms, stored in the protocol
normalizer's output space (every entry is a fixed point of `Normalizer()`).
Sources: Health Canada Drug Product Database + Canadian Clinical Drug Data Set
(Open Government Licence – Canada) and ICD-10-CM (US public domain). Every entry
derives from one of those sources; none is drafted or model-generated. Full
provenance in [NOTICE](https://github.com/Nordis-Tech/medwer/blob/main/NOTICE).

## Regenerating fixtures

`tools/gen_fixtures.py` rewrites the golden fixtures; run it only when the pinned
`whisper-normalizer` version is bumped or the fixture corpora change.

```
python tools/gen_fixtures.py
```

Every entry of the bundled term list is a fixed point of the protocol normalizer,
since an entry the normalizer would rewrite can never match normalized text. A
custom list (`--terms`) must hold the same property under the scorer's own
normalizer; `Scorer` validates it at load and rejects the list otherwise. The test
suite asserts it for the bundled list.

## License

MIT (code). Term-list data: see [NOTICE](https://github.com/Nordis-Tech/medwer/blob/main/NOTICE) for per-source terms.
