Metadata-Version: 2.4
Name: deafbench
Version: 0.2.2
Summary: Accessibility-focused evaluation for AI captions and ASR systems.
License-Expression: Apache-2.0
Project-URL: Homepage, https://488315.github.io/products/deafbench/
Project-URL: Documentation, https://github.com/488315/DeafBench#readme
Project-URL: Source, https://github.com/488315/DeafBench
Project-URL: Changelog, https://github.com/488315/DeafBench/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/488315/DeafBench/issues
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jiwer<4.0.0,>=3.0.0
Provides-Extra: audit
Requires-Dist: cryptography<49.0,>=48.0; extra == "audit"
Requires-Dist: fpdf2<3.0,>=2.8.5; extra == "audit"
Requires-Dist: matplotlib<4.0,>=3.10; extra == "audit"
Requires-Dist: nemo_toolkit[asr]<3,>=2.4; extra == "audit"
Requires-Dist: numba<0.67,>=0.65; extra == "audit"
Requires-Dist: scipy<2.0,>=1.15; extra == "audit"
Requires-Dist: tqdm<5.0,>=4.67; extra == "audit"
Requires-Dist: torchaudio<3.0,>=2.8; extra == "audit"
Requires-Dist: transformers[torch]<6.0.0,>=5.13.0; extra == "audit"
Provides-Extra: benchmark
Requires-Dist: numpy>=1.26; extra == "benchmark"
Requires-Dist: soundfile>=0.12; extra == "benchmark"
Requires-Dist: WhisperSpeech>=0.8.9; extra == "benchmark"
Requires-Dist: webdataset<2.0.0,>=1.0.2; extra == "benchmark"
Provides-Extra: local-models
Requires-Dist: faster-whisper<2.0.0,>=1.2.1; extra == "local-models"
Provides-Extra: zero-custody-pilot
Requires-Dist: cryptography<49.0,>=48.0; extra == "zero-custody-pilot"
Provides-Extra: open-asr-evaluator
Requires-Dist: datasets==3.6.0; extra == "open-asr-evaluator"
Requires-Dist: kaldialign==0.12.0; extra == "open-asr-evaluator"
Requires-Dist: num2words==0.5.14; extra == "open-asr-evaluator"
Requires-Dist: regex==2026.7.19; extra == "open-asr-evaluator"
Provides-Extra: real-speech-dev
Requires-Dist: datasets==3.6.0; python_version < "3.14" and extra == "real-speech-dev"
Requires-Dist: numpy>=1.26; extra == "real-speech-dev"
Requires-Dist: scipy<2.0,>=1.15; extra == "real-speech-dev"
Requires-Dist: soundfile<1.0,>=0.13; extra == "real-speech-dev"
Provides-Extra: qwen-asr
Requires-Dist: scipy<2.0,>=1.15; extra == "qwen-asr"
Requires-Dist: transformers[torch]<6.0.0,>=5.13.0; extra == "qwen-asr"
Provides-Extra: parakeet-asr
Requires-Dist: nemo_toolkit[asr]<3,>=2.4; extra == "parakeet-asr"
Requires-Dist: numba<0.62,>=0.61; extra == "parakeet-asr"
Provides-Extra: granite-asr
Requires-Dist: scipy<2.0,>=1.15; extra == "granite-asr"
Requires-Dist: torchaudio<3.0,>=2.8; extra == "granite-asr"
Requires-Dist: transformers[torch]<6.0.0,>=5.13.0; extra == "granite-asr"
Provides-Extra: granite-nar-asr
Requires-Dist: accelerate<2.0,>=1.10; extra == "granite-nar-asr"
Requires-Dist: flash-attn==2.8.3; platform_system == "Linux" and extra == "granite-nar-asr"
Requires-Dist: soundfile<1.0,>=0.13; extra == "granite-nar-asr"
Requires-Dist: torch==2.9.1; extra == "granite-nar-asr"
Requires-Dist: torchcodec==0.9.1; extra == "granite-nar-asr"
Requires-Dist: torchaudio==2.9.1; extra == "granite-nar-asr"
Requires-Dist: transformers<6.0.0,>=5.5.3; extra == "granite-nar-asr"
Provides-Extra: ark-asr
Requires-Dist: librosa<1.0,>=0.11; extra == "ark-asr"
Requires-Dist: soundfile<1.0,>=0.13; extra == "ark-asr"
Requires-Dist: transformers[torch]<6.0.0,>=5.5.0; extra == "ark-asr"
Provides-Extra: ark-onnx-asr
Requires-Dist: librosa<1.0,>=0.11; extra == "ark-onnx-asr"
Requires-Dist: onnxruntime<2.0,>=1.23; extra == "ark-onnx-asr"
Requires-Dist: soundfile<1.0,>=0.13; extra == "ark-onnx-asr"
Requires-Dist: transformers[torch]<6.0.0,>=5.5.0; extra == "ark-onnx-asr"
Provides-Extra: recorder
Requires-Dist: numpy>=1.26; extra == "recorder"
Requires-Dist: sounddevice>=0.5.1; extra == "recorder"
Provides-Extra: test
Requires-Dist: cryptography<49.0,>=48.0; extra == "test"
Requires-Dist: fpdf2<3.0,>=2.8.5; extra == "test"
Requires-Dist: kaldialign==0.12.0; extra == "test"
Requires-Dist: matplotlib<4.0,>=3.10; extra == "test"
Requires-Dist: numpy>=1.26; extra == "test"
Requires-Dist: PyYAML<7.0,>=6.0; extra == "test"
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pytest-cov>=5.0; extra == "test"
Requires-Dist: ruff<0.13,>=0.12; extra == "test"
Requires-Dist: scipy<2.0,>=1.15; extra == "test"
Requires-Dist: soundfile<1.0,>=0.13; extra == "test"
Requires-Dist: tqdm<5.0,>=4.67; extra == "test"
Dynamic: license-file

# DeafBench

[![CI](https://github.com/488315/DeafBench/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/488315/DeafBench/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/deafbench.svg)](https://pypi.org/project/deafbench/)
[![Python](https://img.shields.io/pypi/pyversions/deafbench.svg)](https://pypi.org/project/deafbench/)
[![License](https://img.shields.io/github/license/488315/DeafBench.svg)](LICENSE)

[Website](https://488315.github.io/products/deafbench/) ·
[Latest release](https://github.com/488315/DeafBench/releases/latest) ·
[PyPI](https://pypi.org/project/deafbench/) ·
[Hugging Face benchmark metadata](https://huggingface.co/datasets/kvjones0243/deafbench-synthetic-v2)

I wrote DeafBench because I am Deaf, I use cochlear implants, and the normal
ASR score does not always describe whether captions are useful to me. Word
error rate treats every edit as part of the same total, but a caption can have
a low WER and still get the time, medication amount, username, Wi-Fi name,
confirmation code, speaker, or sound event wrong. Those are not small mistakes
when the caption is the information I have to act on.

## What DeafBench measures

My background is in IT, so I built this like an audit instead of a demo.
DeafBench keeps the references, model revisions, decoding settings, evaluator
revision, and artifact hashes with the result. It reports WER, but it also
reports strict lexical and typed canonical recall for critical information,
non-speech information, speaker attribution, latency, and the actual
substitution, insertion, and deletion counts. The typed evaluator only accepts
the harmless representation changes allowed for that entity type; it does not
turn a different username, code, time, or Wi-Fi name into a pass.

The project now has two separate jobs. The synthetic track measures whether an
ASR system preserves accessibility-critical information. The Hugging Face
compatibility track uses the pinned Open ASR Leaderboard datasets, normalizer,
preprocessing, WER calculation, and seven-dataset macro-average. I do not mix
those scores because they answer different questions.

## One current result

The reproduced Zipformer baseline scored **5.23% public seven-set macro WER**
with the pinned official-compatible local workflow. This is evidence that the
runner reproduces the public contract; it is not a verified leaderboard win,
does not include private sets, and uses a CC-BY-NC-4.0 checkpoint that cannot be
the commercial foundation without separate permission.

## Install and run in 30 seconds

Install the current PyPI release, download the two public example files from
the matching `v0.2.1` tag, and compare them:

```bash
python -m pip install "deafbench==0.2.1"
curl -LO https://raw.githubusercontent.com/488315/DeafBench/v0.2.1/examples/references.jsonl
curl -LO https://raw.githubusercontent.com/488315/DeafBench/v0.2.1/examples/model-a.jsonl
deafbench compare references.jsonl model-a.jsonl
```

The published v0.2.1 command reports its legacy WER label together with
critical-information recall, edit counts, non-speech information, speaker
attribution, and latency. The current `main` branch adds separately named
orthographic and normalized WER/CER fields for the next release; installing
v0.2.1 from PyPI does not expose those new fields. Read the
[`ASR evaluation methodology`](docs/asr-evaluation-methodology.md) for the
normalization and aggregation contract, or continue to the separately labeled
[English accessibility benchmark](#english-accessibility-benchmark) and
[source-checkout workflows for `main`](#detailed-installation-and-workflows).

### Customer audit on `main`

The customer audit is designed around one case folder containing `audio/` and a
human-approved `references.csv`. The first run asks for a required case name and
local authorization details, then keeps its reusable state under `.deafbench/`.
Later runs reuse that setup automatically. Sample-level work is retained only for
the latest successful run; a successful rerun removes older `.deafbench/runs/`
directories. The local authorization record still carries the planned 14-day
case deletion date.

From a source checkout today:

```bash
python -m pip install -e ".[audit]"
deafbench audit ./customer-case
```

After a release containing this workflow is published, the supported install is:

```bash
python -m pip install "deafbench[audit]"
deafbench audit ./customer-case
```

A successful audit writes `audit-report/index.html`, `audit-report/report.pdf`,
and aggregate signed evidence, then opens the HTML report when a graphical
browser is available. The HTML report groups every failed sample by real-world
failure category, shows REF/HYP word alignment, and gives deterministic
investigation guidance. Use `deafbench review ./customer-case` to add customer
context or a consequence-based severity adjustment without replacing the
original DeafBench severity.

The complete three-model audit uses NVIDIA Parakeet through NeMo. NVIDIA lists
Linux as the supported operating system for the pinned Parakeet model, so the
supported Windows path for the complete three-model run is WSL2. The command,
case layout, HTML/PDF output, and review workflow are otherwise the same.

## The goal

My goal is to build an ASR system that beats the Hugging Face Open ASR
Leaderboard while still doing better on the information that matters to Deaf
and hard-of-hearing users. I will only say DeafBench beat the leaderboard after
a separate candidate is evaluated at a declared milestone and Hugging Face
verifies the result. Until then, the result above is the local public
compatibility baseline to beat, not a product claim. The exact upstream
revisions, commands, and evidence are in
[`experiments/open-asr/README.md`](experiments/open-asr/README.md).

## Models that work with DeafBench

The models below have working adapters in this repository. Nine newer adapters
have recorded local observations for a 25-sample synthetic-v2 run and a two-row
public real-speech smoke run. Their byte-stable metadata manifests are under
`experiments/model-results`, but the sample-level predictions and run artifacts
are not published in this checkout, so these values cannot be independently
recomputed from the repository alone. A smoke observation shows that the pinned
adapter executed in the recorded environment; it does not prove model quality
or a leaderboard score. OpenAI Whisper `turbo` remains legacy report evidence
only and does not have one of these newer manifests.

| DeafBench model name | Pinned model | Current evidence | License lane |
| --- | --- | --- | --- |
| `whisper` | OpenAI Whisper `turbo` | Core v1 and non-speech v1 reports | Runtime model; review upstream terms |
| `whisper-at` | Whisper-AT `medium.en` | Synthetic-v2, real-speech smoke, and non-speech-v1 | Commercial candidate, BSD-2-Clause |
| `faster-whisper` | `Systran/faster-whisper-small.en` | Frozen Core v1 baseline | Runtime model; review upstream terms |
| `distil-whisper` | `Systran/faster-distil-whisper-large-v3` | Synthetic-v2 plus real-speech smoke | Commercial candidate, MIT |
| `qwen3-asr-0.6b` | `Qwen/Qwen3-ASR-0.6B-hf` | Synthetic-v2 plus real-speech smoke | Commercial candidate, Apache-2.0 |
| `qwen3-asr-1.7b` | `Qwen/Qwen3-ASR-1.7B-hf` | Synthetic-v2 plus real-speech smoke | Commercial candidate, Apache-2.0 |
| `parakeet-tdt-0.6b-v2` | `nvidia/parakeet-tdt-0.6b-v2` | Synthetic-v2 plus real-speech smoke | Commercial candidate, CC-BY-4.0 attribution required |
| `granite-speech-4.1-2b` | `ibm-granite/granite-speech-4.1-2b` | Synthetic-v2 plus real-speech smoke | Commercial candidate, Apache-2.0 |
| `granite-speech-4.1-2b-nar` | `ibm-granite/granite-speech-4.1-2b-nar` | Synthetic-v2 plus real-speech smoke | Commercial candidate, Apache-2.0; audited remote code |
| `ark-asr-0.6b` | `AutoArk-AI/ARK-ASR-0.6B` | Synthetic-v2 plus real-speech smoke | Commercial candidate, Apache-2.0; audited isolated remote code |
| `ark-asr-0.6b-int8-onnx` | `AutoArk-AI/ark-asr-0.6b-int8-onnx` | Synthetic-v2 plus real-speech smoke | Commercial candidate, Apache-2.0; audited isolated remote code |

The machine-readable registry at
[`deafbench/model-registry.json`](deafbench/model-registry.json) pins revisions,
runtimes, license classifications, attribution requirements, expected download
sizes, and measured peak VRAM. That registry is operational metadata, not legal
advice. Model weights are third-party software and are not owned by DeafBench.

## English accessibility benchmark

Synthetic-v2 measures English accessibility-critical information on 25
generated samples. Lower WER and peak VRAM are better; higher critical recall
and local RTFx are better. This table is a separate DeafBench accessibility
evaluation and is not a Hugging Face Open ASR Leaderboard result.

| Model | WER ↓ | Strict critical recall ↑ | Canonical critical recall ↑ | Local RTFx ↑ | Peak VRAM ↓ |
| --- | ---: | ---: | ---: | ---: | ---: |
| Distil-Whisper large-v3 | 23.8% | 66.1% | 91.9% | 0.80 | CPU |
| Whisper-AT `medium.en` | 26.2% | 67.7% | 96.8% | 7.34 | 4.46 GiB |
| Qwen3-ASR 0.6B | 26.6% | 67.7% | 90.3% | 10.47 | 1.52 GiB |
| Qwen3-ASR 1.7B | 21.0% | 67.7% | 91.9% | 9.89 | 3.86 GiB |
| Parakeet TDT 0.6B v2 | 22.7% | 64.5% | 91.9% | 71.23 | 4.67 GiB |
| Granite Speech 4.1 2B | 18.9% | 69.4% | 91.9% | 12.91 | 4.35 GiB |
| Granite Speech 4.1 2B NAR | 40.6% | 64.5% | 87.1% | 45.07 | 4.26 GiB |
| ARK-ASR 0.6B | 30.1% | 66.1% | 90.3% | 14.85 | 2.20 GiB |
| ARK-ASR 0.6B INT8 ONNX | 26.6% | 66.1% | 90.3% | 2.40 | CPU |

### Two-row public real-speech smoke results

| Model | WER | Local RTFx | Peak VRAM |
| --- | ---: | ---: | ---: |
| Distil-Whisper large-v3 | 1.73% | 3.27 | CPU |
| Whisper-AT `medium.en` | 1.73% | 11.19 | 4.46 GiB |
| Qwen3-ASR 0.6B | 2.31% | 12.27 | 1.72 GiB |
| Qwen3-ASR 1.7B | 1.16% | 12.43 | 4.05 GiB |
| Parakeet TDT 0.6B v2 | 2.31% | 91.85 | 4.67 GiB |
| Granite Speech 4.1 2B | 3.47% | 7.43 | 4.42 GiB |
| Granite Speech 4.1 2B NAR | 2.89% | 25.15 | 4.46 GiB |
| ARK-ASR 0.6B | 2.89% | 7.49 | 2.29 GiB |
| ARK-ASR 0.6B INT8 ONNX | 2.89% | 3.01 | CPU |

The byte-stable metadata records for both tables are in
[`experiments/model-results`](experiments/model-results). They are recorded
local observations, not independently recomputable evidence, because their
sample-level artifacts are not in this checkout. GPU rows are local RTX 4070
measurements, and CPU rows are labeled separately. The two-row smoke results
are not the seven-dataset macro-average and are not Hugging Face verified.

### Earlier model evidence

| Model | Benchmark evidence | Result |
| --- | --- | --- |
| OpenAI Whisper `turbo` | Core v1, 25 samples | 23.4% WER; 88.7% legacy critical-information recall |
| OpenAI Whisper `turbo` | Non-speech v1, 12 samples | 2.0% WER; 95.0% legacy critical-information recall; 0.0% non-speech recall |
| Faster-Whisper `small.en` | Frozen Core v1 synthetic baseline, 25 samples | 26.2% WER; 69.4% strict lexical recall; 90.3% canonical semantic recall |
| Whisper-AT `medium.en` | Non-speech v1, 12 samples | 2.0% WER; 95.0% strict and canonical critical recall; 0 of 19 expected sound events matched |

The OpenAI Whisper reports are
[`benchmarks/core-v1/model-a-report.md`](benchmarks/core-v1/model-a-report.md)
and
[`benchmarks/non-speech-v1/model-a-report.md`](benchmarks/non-speech-v1/model-a-report.md).
The Faster-Whisper classification and scoring evidence is in
[`benchmarks/core-v1/faster-whisper-synthetic-analysis.md`](benchmarks/core-v1/faster-whisper-synthetic-analysis.md).
The older Whisper recall value uses the evaluator that produced those frozen
reports, so I do not label it as strict or canonical scoring.

WER does not tell the full accessibility story. DeafBench also measures
critical information loss and non-speech events that WER misses.

## Accessibility stress testing

[`accessibility-stress-v1`](benchmarks/accessibility-stress-v1/README.md) adds a
byte-frozen 24-utterance reference set for paired clean and degraded runs. It
predeclares fixed-SNR street, office, wind, breathing, keyboard, and rustling
noise;
noise-only interstitials; 8 kHz telephony; reverberation; long pauses; rate
variation; overlap; and codec degradation. The evaluator keeps WER edits,
deletion share, typed critical failures, interstitial hallucinations, caption
timing drift, and observed local load metrics separate instead of reducing the
stress run to one number.

This is synthetic stress coverage, not a Deaf or dysarthric speech dataset. I
will not use rate changes, pauses, or noise to claim demographic performance.
That evidence requires a separate authorized and consented human-speech lane
with subgroup reporting and a corpus that is appropriate for that purpose.

The executable local lane requires clean WAV files named for the selected
reference IDs. `--implemented-only` runs the six transformation families that
DeafBench can currently materialize and labels the result with the exact sample
count. It does not count the declared overlap or codec cases as completed.

```powershell
python -m deafbench stress `
  --references benchmarks/accessibility-stress-v1/references.jsonl `
  --clean-audio benchmarks/accessibility-stress-v1/audio-clean `
  --output benchmarks/accessibility-stress-v1/runs/faster-whisper/local `
  --model faster-whisper `
  --implemented-only
```

The output directory contains hash-bound preparation evidence, clean and
stressed predictions, and a local result. Generated audio, predictions, and
runs stay untracked.

---

## Detailed installation and workflows

### Optional installations and development setup

Install the recorder extra if you want to capture benchmark audio:

```powershell
python -m pip install "deafbench[recorder]"
deafbench recorder
```

Or install locally for development:

```bash
git clone https://github.com/488315/DeafBench.git
cd DeafBench
pip install -e .
```

### Customer-run accessibility audit

The founding-pilot workflow runs on the customer's authorized computer. Raw
audio, transcripts, filenames, paths, and critical-information values are not
customer-export artifacts. Install the signing dependency and inspect the
supported local actions:

```powershell
python -m pip install "deafbench[zero-custody-pilot]"
deafbench audit --help
```

Run the synthetic rehearsal before evaluating authorized, non-sensitive
audio:

```powershell
deafbench audit rehearse `
  --repo-root . `
  --output-dir .\rehearsal-export `
  --signing-key C:\secure-local-path\deafbench-signing-key.pem
```

The customer-local evaluation and aggregate export are one command:

```powershell
deafbench audit run `
  --repo-root . `
  --case-root C:\customer-controlled\deafbench-case `
  --attestation C:\customer-controlled\execution-attestation.json `
  --output-dir C:\customer-controlled\deafbench-export `
  --signing-key C:\secure-local-path\deafbench-signing-key.pem
```

The signing key and all customer artifacts stay on the customer's computer.
The export is labelled customer-executed and environment-dependent; it is not
a certification or a Hugging Face leaderboard result.

### Automated benchmark workflow

Install the default synthetic-audio runtime, then run a complete benchmark:

```powershell
python -m pip install "deafbench[benchmark]"

# Both model backends require ffmpeg on PATH.
# Core v1 with OpenAI Whisper
python -m pip install -U openai-whisper
deafbench benchmark core-v1 --model whisper
```

Whisper-AT uses a pinned upstream commit whose installer imports the removed
`pkg_resources` module. DeafBench keeps that source and its runtime requirements
unchanged, verifies their hashes, and applies a packaged build-only patch that
uses `pathlib` to read `requirements.txt` and requires setuptools 83 or newer.
Install it from a DeafBench checkout with Python 3.11:

```powershell
python -m pip install ".[test]"
python -m deafbench.whisper_at_compat
python -c "import whisper_at"
```

Python 3.11 is required for this pinned Whisper-AT runtime because its exact
`tiktoken==0.3.3` dependency does not publish wheels for every newer supported
DeafBench interpreter. This restriction preserves the upstream dependency pin
instead of silently changing model behavior. The patch manifest records the
upstream commit and every before/after source hash in
`deafbench/whisper_at_compat/manifest.json`.

The `benchmark` extra installs WhisperSpeech and the audio dependencies used to
build synthetic scenes. OpenAI Whisper is a separate inference backend, so
install it before its benchmark command. The extra does not install inference
backends.

Two additional local models use the Faster-Whisper runtime. For existing human
audio, install the local-model extra. Synthetic runs also need the benchmark
extra that supplies WhisperSpeech and its runtime dependencies:

```powershell
# Existing human audio
python -m pip install "deafbench[local-models]"

# Synthetic audio from an editable repository checkout
python -m pip install -e ".[benchmark,local-models]"

# CPU-friendly INT8 baseline; downloads small.en on first use.
python -m deafbench benchmark core-v1 --model faster-whisper --audio-source synthetic --repo-root .

# Distilled comparison; downloads the pinned CTranslate2 repository on first use.
deafbench benchmark synthetic-v2 --model distil-whisper --audio-source synthetic --repo-root .
```

Both default to CPU INT8 so they work without an NVIDIA GPU. Faster-Whisper
uses `small.en`; Distil-Whisper pins
`Systran/faster-distil-whisper-large-v3` with previous-text conditioning
disabled. The Faster-Whisper runtime decodes audio through PyAV, so these two
models do not need a separate system FFmpeg installation.

With the default `--audio-source auto` policy, DeafBench selects one complete
source for the whole run:

```text
complete audio/ set   -> human run
incomplete audio/ set -> complete audio-synthetic/ run
never mix sources
```

Use `--audio-source human` or `--audio-source synthetic` to require a specific
source. Human mode fails if `audio/` is incomplete. Synthetic mode generates or
reuses a complete synthetic set before inference begins.

WhisperSpeech supplies the speech signal. DeafBench supplies ambience,
environmental-event timing, and final mixing. The `default-v1` scene profile
uses seed `42` unless overridden. Scene planning is reproducible for the same
inputs, but DeafBench does not promise byte-identical TTS output across runtime,
model, or hardware versions.

Each successful run writes traceable, source-aware run artifacts. Synthetic
generation additionally writes its reusable manifest:

```text
benchmarks/<dataset>/runs/<model>/<audio-source>/predictions.jsonl
benchmarks/<dataset>/runs/<model>/<audio-source>/report.md
benchmarks/<dataset>/runs/<model>/<audio-source>/run.json
benchmarks/<dataset>/audio-synthetic/manifest.jsonl  # synthetic only
```

`run.json` records the resolved source, model identity, paths, sample count, and
benchmark version. Synthetic runs also record the scene profile, seed, and TTS
engine/version. Run directories include both model and source so human and
synthetic results cannot overwrite one another.

Reports keep critical-information scoring in two separate views. Strict lexical
recall measures the expected surface form; canonical semantic recall applies
only the normalization allowed by an entity's explicit type, such as TIME or
DIGIT_SEQUENCE. Reports also include per-sample WER and aggregate substitution,
insertion, and deletion counts. Conventional transcription output now names
orthographic and normalized WER and CER separately and records the normalization
policy. The exact aggregation, normalization, RTFx, and leaderboard boundaries
are documented in
[`docs/asr-evaluation-methodology.md`](docs/asr-evaluation-methodology.md).

### Usage

**1. Compare predictions against reference captions:**

```bash
deafbench compare examples/references.jsonl examples/model-a.jsonl
```

Output:
```text
DeafBench v0.1

Samples: 3

Orthographic WER            33.3%
Normalized WER              28.6%
Orthographic CER            21.6%
Normalized CER              20.2%
Normalization policy      deafbench-asr-normalization-v1
Strict Critical Information      50.0%
Canonical Critical Information   50.0%
WER edits (S/I/D)         3/0/4
Non-Speech Information      50.0%
Speaker Attribution        100.0%
Median Latency               1.1s

[!] 3 critical-information failures detected
```

**2. Generate a Markdown evaluation report:**

```bash
deafbench report examples/references.jsonl examples/model-a.jsonl --output report.md
```

### OpenAI Whisper transcription

Install OpenAI Whisper and generate `model-a.jsonl` for Core v1:

```powershell
python -m pip install -U openai-whisper
python tools\transcribe_whisper.py --dataset core-v1
```

Run the same helper for non-speech v1:

```powershell
python tools\transcribe_whisper.py --dataset non-speech-v1
```

The helper uses Whisper `turbo` in English and writes predictions into the selected benchmark directory.

### Whisper-AT Model B

Model B uses [Whisper-AT](https://github.com/YuanGongND/whisper-at) to keep speech recognition and audio-event tagging in one run. DeafBench stores the ASR transcript in `text`, mapped benchmark sound events in `sounds`, and the original Whisper-AT AudioSet labels in `audio_tags`. Keeping sound labels out of `text` means environmental-sound scoring does not change the speech WER.

Install Whisper-AT with the exact revision and dependency versions in the
[automated benchmark workflow](#automated-benchmark-workflow) above. Do not
replace that command with an unpinned PyPI installation when reproducing the
recorded evidence. Whisper-AT also requires `ffmpeg` on `PATH`.

Generate Model B predictions for both current benchmarks:

```powershell
python tools\transcribe_whisper_at.py --dataset core-v1
python tools\transcribe_whisper_at.py --dataset non-speech-v1
```

The runner defaults to Whisper-AT `medium.en`. Model A uses Whisper `turbo`, so this comparison measures the complete captioning systems rather than isolating only the audio-tagging layer.

Generate the reports after Model B finishes:

```powershell
deafbench report benchmarks\core-v1\references.jsonl benchmarks\core-v1\model-b.jsonl --output benchmarks\core-v1\model-b-report.md
deafbench report benchmarks\non-speech-v1\references.jsonl benchmarks\non-speech-v1\model-b.jsonl --output benchmarks\non-speech-v1\model-b-report.md
```

Measured Model B numbers should only be added to this README after those runs are completed.

### Non-speech v1 recording workflow

`non-speech-v1` stays separate from Core v1. Each reference has one or more `sounds` labels. The GUI shows those labels before recording. When you press **Stop**, the recorder synthesizes each sound and appends it after the speech in label order.

```powershell
python -m pip install "deafbench[recorder]"
deafbench recorder --dataset non-speech-v1
```

`deafbench recorder` can run outside a source checkout. It defaults to Core v1 and seeds `benchmarks\<dataset>\references.jsonl` in the current directory when the bundled benchmark is not already there. Existing references are left alone, and recordings go to `benchmarks\<dataset>\audio`.

For example:

```json
"sounds": ["[phone rings]", "[knock]"]
```

produces:

```text
recorded speech → short gap → phone ring → short gap → knock
```

Transcribe and score it with:

```powershell
python tools\transcribe_whisper.py --dataset non-speech-v1
deafbench report benchmarks\non-speech-v1\references.jsonl benchmarks\non-speech-v1\model-a.jsonl --output benchmarks\non-speech-v1\model-a-report.md
```

Supported generated events are `[alarm]`, `[door closes]`, `[phone rings]`, `[knock]`, `[error notification]`, and `[siren]`.

---

## Input JSONL Schema

`references.jsonl`
```json
{
  "id": "sample-001",
  "text": "John Doe needs 25 milligrams on Friday.",
  "critical": ["John Doe", "25 milligrams", "Friday"],
  "sounds": ["[alarm]"],
  "speaker": "Speaker 1"
}
```

`predictions.jsonl`
```json
{
  "id": "sample-001",
  "text": "Guy needs 20 milligrams on Friday.",
  "latency_ms": 820,
  "speaker": "Speaker 1"
}
```

---

## Contributing and security

Start with [`CONTRIBUTING.md`](CONTRIBUTING.md) before changing an evaluator,
adapter, benchmark, or customer-run audit boundary. Use
[`SECURITY.md`](SECURITY.md) and GitHub's
[private security-advisory interface](https://github.com/488315/DeafBench/security/advisories/new)
for suspected vulnerabilities; do not put customer audio, transcripts,
credentials, or other sensitive content in a public issue.

---

## Citation

GitHub can generate a citation from [`CITATION.cff`](CITATION.cff). The file
identifies the released software version, author, license, source repository,
and project website without implying a paper, DOI, or independent
certification that does not exist.

---

## License

[Apache License 2.0](LICENSE)
