Metadata-Version: 2.4
Name: anytimemonitor
Version: 0.10.0
Summary: Adaptive listening-test controller for TTS-MOS evaluation: anytime-valid confidence sequences, per-pair verdicts, two-lever (items/listeners) budget advice
Author: Jonathan Chevelu, Nelly Barbot
License: MIT
Project-URL: Homepage, https://expression.gitlabpages.inria.fr/replication/2026-doing-mos-well/
Project-URL: Source, https://github.com/jchevelu/AnytimeMonitor
Project-URL: Repository, https://gitlab.inria.fr/expression/tools/anytimemonitor
Project-URL: Archive, https://doi.org/10.5281/zenodo.21326813
Keywords: speech-synthesis,MOS,listening-test,anytime-valid,confidence-sequence,evaluation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.1
Requires-Dist: scipy>=1.11
Requires-Dist: matplotlib>=3.8
Provides-Extra: ui
Requires-Dist: streamlit>=1.50; extra == "ui"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# AnytimeMonitor — adaptive controller for TTS-MOS listening tests

AnytimeMonitor watches a MOS listening test *while it runs* and tells
the test admin, for every pair of systems, whether the data already
supports a verdict — and if not, which lever (more items, or more
listeners) closes the pair fastest.

The statistical core is an **anytime-valid confidence sequence**
(Howard, Ramdas, McAuliffe & Sekhon, 2021) on per-item paired score
differences. Anytime validity means the coverage guarantee holds at
*any* data-dependent stopping time: the admin can look at the verdicts
as often as they like, stop whenever a pair resolves, and the reported
confidence interval is still valid. No peeking penalty bookkeeping, no
pre-registered sample size required.

This is a research prototype (v0.9). It accompanies a position paper
on budget allocation in TTS evaluation (under submission, *Computer
Speech & Language* Special Issue on the evaluation of speech and
speech synthesis).

## What the admin sees

For each pair of systems (A, B), one of six verdicts:

| Verdict | Meaning |
| --- | --- |
| `STOP` | The anytime-valid CI excludes 0 (0 ∉ CI): the pair is separated. |
| `STOP_EQUIV` | Strict TOST equivalence: CI ⊂ ±δ_equiv *and* the half-width Ψ_eff ≤ δ_equiv/2. The pair is statistically equivalent within tolerance. |
| `CONTINUE +N` | Neither verdict yet. The tool predicts how many more **items** (+N) or **listeners** (+L) would close the pair, and which terminal (STOP or STOP_EQUIV) it is heading for. The headline is the cheaper option. |
| `WAIT` | The listener×system variance component cannot be estimated yet (fewer than 3 listeners with ≥ 2 paired items). Structural: keep collecting before trusting any verdict. |
| `CORPUS_LIMIT` | The item budget is exhausted *and* the listener lever can no longer close the pair. Terminal. |
| `LISTENER_LIMIT` | The listener budget is exhausted *and* the item lever can no longer close the pair. Terminal. |

> **Verdict naming.** `STOP` is the separation verdict the companion paper
> calls `STOP_SEP`. `STOP` is the current public label and will be **renamed to
> `STOP_SEP` in a future release** (to pair symmetrically with `STOP_EQUIV`);
> code matching on `"STOP"` should expect that change.

Verdicts are recomputed from scratch at every call — the controller is
queried on demand (a handful of times over a test lifecycle), not in a
real-time loop. A `STOP` is **not sticky**: if you keep collecting
past it, a borderline pair can revert to `CONTINUE`. This is honest
anytime-CI behaviour (the guarantee is on coverage, not on the
monotonicity of the observed difference); in practice the admin acts
on the first `STOP`.

## Install

```bash
pip install -e .          # from the repository root (pyproject.toml)
pip install -e ".[ui]"    # + Streamlit demo UI
pip install -e ".[test]"  # + pytest; then run `pytest`
```

Requires Python ≥ 3.10. Pinned versions used for development are in
`requirements.txt`.

## Quick start

### Programmatic

```python
import pandas as pd
from anytimemonitor import AnytimeMonitor

ctrl = AnytimeMonitor(mde_threshold=0.30)   # δ_sep, in MOS units

# Feed ratings as they arrive — long format, one row per judgment
ctrl.add_ratings(pd.DataFrame({
    "listener_id": ...,   # who rated
    "item":        ...,   # which sentence / stimulus
    "system":      ...,   # which TTS system
    "rating":      ...,   # the MOS score
}))

print(ctrl.report())                  # all pairs
dec = ctrl.decision("sysA", "sysB")   # one pair, structured
print(dec.verdict, dec.ci_lo, dec.ci_hi, dec.n_more_items)
```

### CLI demo

Feed a ratings CSV to the controller batch by batch and watch the per-pair
verdicts tighten. With no arguments it runs on a bundled synthetic sample:

```bash
python -m anytimemonitor.demo                          # bundled sample
python -m anytimemonitor.demo --input my_ratings.csv   # your own long-format CSV
```

### Streamlit demo UI

```bash
streamlit run anytimemonitor/ui/app.py
```

Upload a ratings CSV (or pick a bundled synthetic example from
`sample_uploads/`), watch the controller replay the stream
batch-by-batch, and drag the time-machine slider to see the verdict
and two-lever advice at any past timestep. See `ui/README.md`.

## How it works

For a pair (A, B), the controller builds per-item paired differences
d_j = mean rating of item j under A minus under B, and maintains a
sub-Gaussian normal-mixture confidence sequence on their mean
(Howard et al. 2021). Two ingredients adapt this to listening-test
data, where ratings are clustered by listener as well as by item:

- **Bivariate cluster bootstrap SE.** Resampling *items* gives the
  i.i.d.-part SE that the anytime boundary multiplies; resampling
  *listeners* gives the listener-pool floor ("plancher") — the
  precision ceiling that no amount of extra items can break. The two
  enter the half-width as a sum of squares, the item side under the
  anytime multiplier at level α₁ and the floor under its own critical
  value z_{1−α₂/2}, with α₁ + α₂ = α. When the data is too sparse to bootstrap, the
  controller falls back to the confidence sequence's internal running
  variance proxy.
- **Variance decomposition layer.** Method-of-moments estimates of
  the system×item and residual components (per system, with a
  Half-Cauchy Bayesian regulariser that prevents the
  truncation-at-zero bias of raw MoM), plus a per-pair estimate of
  the listener×system component. These feed the two-lever
  prediction: +N items shrinks the interaction term, +L listeners
  lowers the floor.

The price of anytime validity is a wider interval than a fixed-n
test: the item side carries the sequence's √(log t) multiplier and the
floor carries a quantile rather than a single standard deviation. The
gap is modest once the floor dominates and several-fold at small item
counts, where the multiplier is at its largest. On top of that comes
deliberate conservatism early on: below `avci_t_min_ebernstein` items the variance proxy falls back
to the worst-case Hoeffding bound, so the first batches produce wide
intervals. The controller is therefore most useful from a few dozen
rated items per system onward.

The internal architecture is documented in
[`Architecture.md`](Architecture.md) (and
[`ui/Architecture.md`](ui/Architecture.md) for the demo UI); the deeper design
notes for the confidence sequence are in `AVCI_PLAN.md` (development
repository only, not shipped in the public snapshot).

## Key parameters

All knobs are constructor arguments of `AnytimeMonitor`; the
defaults are deliberately conservative.

| Parameter | Default | Role |
| --- | --- | --- |
| `mde_threshold` (δ_sep) | (required) | The smallest effect worth *resolving* (MOS): seeds `delta_equiv` and the reported MDE. It does **not** gate `STOP` — separation fires on `0 ∉ CI` regardless. |
| `delta_equiv` | `None` (= `mde_threshold`) | δ_equiv — equivalence margin for `STOP_EQUIV`. Set it below `mde_threshold` to create an indifference zone. |
| `alpha` | 0.05 | Nominal level of the confidence sequence. |
| `power` | 0.80 | Only sets the reported-MDE scale factor (`z_{α/2} + z_power`); does not affect stopping. |
| `n_max_corpus`, `l_max_listeners` | `None` | Item / listener budgets; enable the `*_LIMIT` terminal verdicts. |
| `avci_n_planned` | `240` (§7 census median budget) | Planned item budget; the controller derives η = 1/√(N_planned) (Howard's a-priori rule). |
| `avci_eta` | `None` (→ 1/√`avci_n_planned`) | Set the Howard 2021 mixing parameter η by hand instead of via `avci_n_planned` — η is fixed a priori, never tuned to data. **Pass one or the other, not both** (raises otherwise). |
| `avci_lissys_floor_r0` | `0.36` (measured r_lisSys band frontier) | A-priori guard on the listener floor: bounds it below by √(2·σ²₀/L) with σ²₀ = 1.5·r₀·σ̂²_eval (the ×1.5 is an MC-calibrated validity margin). Protects coverage when a small pool under-estimates σ²_lisSys. `0` disables. |
| `avci_lissys_floor_sigma2` | `None` | Absolute σ²₀ override for the guard (expert path, no extra margin). **Exclusive with `avci_lissys_floor_r0`** (raises otherwise). The per-pair `Decision.lissys_prior_exceeded` flags when data-driven floor estimates exceed the declared guard (declared margin exhausted; consider raising r₀). |
| `avci_gamma_alpha_factor` | 1.0 | Multiplier `c` on the listener floor (√c on the plancher); `1.0` assumes σ²_lisSys ≤ σ²_int. Bypassed when the bootstrap plancher is available (the normal path). |
| `avci_proxy` | `"ebernstein"` | Variance proxy: running empirical variance (`ebernstein`) or worst-case `hoeffding`. |
| `avci_d_bounds` | `(-4, 4)` | Range of the per-item paired *difference* (not the raw rating scale): ±(rating max − min), so the default spans a rating range of 4. The normal path bootstraps the SE from the data and **ignores this** (it feeds only the sparse-data fallback and a reported diagnostic); widen it for a larger rating range if you drive `AnytimeCI` directly on sparse data. |
| `avci_t_min_ebernstein` | 10 | Below this item count the `ebernstein` proxy falls back to `hoeffding` (fallback path only). |
| `avci_ebernstein_floor_frac` | 0.01 | Relative floor on the empirical-variance proxy, as a fraction of the `hoeffding` bound (fallback path only). |
| `avci_bootstrap_B` | 200 | Bootstrap resamples per decision call. |
| `avci_bootstrap_seed` | 0 | PRNG seed for the bivariate cluster bootstrap. |
| `sigma2_ga_aggregator` | `"median"` | Which bootstrap-CI point anchors the listener floor: `median` / `mean` / `q95` (conservative) / `None`. |
| `cold_start_ga_ratio` | 2.7 | Conservative BTTF-anchored prior ratio (σ²_lisSys/σ²_int) used before the component is identifiable. |
| `avci_wait_min_listeners_kge2` | 3 | Structural WAIT gate: minimum listeners with ≥ 2 paired items. |
| `avci_wait_max_ci_ratio` | 5.0 | WAIT gate: max tolerated q95/median on the σ²_lisSys bootstrap CI before abstaining. |

> **Legacy naming.** Parameters and `Decision` fields carrying `ga` /
> `gamma_alpha` (e.g. `cold_start_ga_ratio`, `sigma2_ga_*`) name the
> listener×system variance the paper writes `σ²_lisSys`; the identifiers will be
> **renamed to `lissys` in a future release** (an API change, hence deferred).

## Input data requirements

- Long format, columns `listener_id`, `item`, `system`, `rating`
  (extra columns are ignored).
- Ratings on any bounded scale. The controller works on per-item paired
  *differences*, whose extremes are ±(rating max − min); the default
  `avci_d_bounds = (-4, 4)` spans a rating range of 4. On the normal path the
  standard error is bootstrapped from the data, so the scale is handled
  automatically and this default is safe on any scale; `avci_d_bounds` only
  enters the sparse-data fallback (when the cluster bootstrap cannot run).
- The design must let listeners rate **multiple items per system**
  (K ≥ 2 somewhere): a design where each listener rates a single item
  — or each item is rated by a single listener — cannot identify the
  listener×system component, and the controller will honestly report
  `WAIT` forever rather than emit verdicts blind to that term.

## Repository layout

```text
anytimemonitor/
├── __init__.py           ← public API: AnytimeMonitor, Decision,
│                           VarianceEstimate, estimate_components
├── controller.py         ← the AnytimeMonitor controller
├── avci.py               ← AnytimeCI — Howard 2021 sub-Gaussian
│                           normal-mixture confidence sequence
├── variance.py           ← MoM variance components + Half-Cauchy
│                           regulariser
├── demo.py               ← CLI example (batch-fed on a bundled sample)
├── ui/                   ← Streamlit demo (app.py; see ui/README.md + ui/Architecture.md)
├── sample_uploads/       ← synthetic example CSVs for the UI
├── Architecture.md       ← internal architecture (controller + statistics)
├── CHANGELOG.md          ← what changed between versions, and what it moves
├── LICENSE               ← MIT
└── requirements.txt      ← pinned development stack
```

The validation and simulation harnesses (synthetic-data generator,
controller-vs-oracle simulator, false-positive-rate check) are not part
of the installed package; they ship with the companion paper's
reproduction code.

## Status and caveats

- **Research prototype.** The API may still change. A `pytest` smoke-test
  suite (`anytimemonitor/tests/`) checks the package runs end to end; the
  controller has additionally been validated by simulation
  (coverage under H₀, controller-vs-oracle gap decomposition) and
  exercised on Blizzard Challenge data; it has not yet been plugged
  into a live test platform.
- **Planning aid first.** The anytime-valid CI is a valid inference
  under optional stopping, so it can be reported as such — but the
  two-lever predictions (+N items / +L listeners) are noisy
  projections from running variance estimates, not guarantees.
- A full-scale validation campaign (multi-seed, multi-corpus
  false-positive and power sweeps) is planned alongside the companion
  paper; the numbers shipped here come from smaller smoke-scale runs.
- **Known bias in the per-system `σ²_int` estimate.** `variance.py`'s
  per-system path under-estimates `σ²_int`, by 25–100 % in Monte-Carlo
  checks: the column centring removes system *i*'s own share of the item
  means (a `1−1/S` shrinkage) and the subtracted `σ̂²_eval/K̄` then
  over-corrects. A per-system EMS re-derivation is pending; the fix is
  `v·S/(S−1) − σ̂²_eval/K̄`. **Where it bites:** the estimate is exposed on
  the public API and feeds the demo presets. It does *not* reach the
  controller's stopping decisions at the default settings — those use the
  per-pair bootstrap, and the only in-controller consumer is the cold-start
  prior, unreachable at the default — nor any number in the companion
  paper. Treat `estimate_components(..., per_system=True)` output as
  indicative until the re-derivation lands.

## License

MIT — see [LICENSE](LICENSE).

## Citation

A companion paper is under submission. Until it is out, please cite
this repository directly.
