Metadata-Version: 2.4
Name: anytimemonitor
Version: 0.9.1
Summary: Adaptive listening-test controller for TTS-MOS evaluation: anytime-valid confidence sequences, per-pair verdicts, two-lever (items/listeners) budget advice
Author: Jonathan Chevelu, Nelly Barbot
License: MIT
Project-URL: Homepage, https://expression.gitlabpages.inria.fr/replication/2026-doing-mos-well/
Project-URL: Source, https://github.com/jchevelu/AnytimeMonitor
Project-URL: Repository, https://gitlab.inria.fr/expression/tools/anytimemonitor
Project-URL: Archive, https://doi.org/10.5281/zenodo.21326813
Keywords: speech-synthesis,MOS,listening-test,anytime-valid,confidence-sequence,evaluation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.1
Requires-Dist: scipy>=1.11
Requires-Dist: matplotlib>=3.8
Provides-Extra: ui
Requires-Dist: streamlit>=1.50; extra == "ui"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# AnytimeMonitor — adaptive controller for TTS-MOS listening tests

AnytimeMonitor watches a MOS listening test *while it runs* and tells
the test admin, for every pair of systems, whether the data already
supports a verdict — and if not, which lever (more items, or more
listeners) closes the pair fastest.

The statistical core is an **anytime-valid confidence sequence**
(Howard, Ramdas, McAuliffe & Sekhon, 2021) on per-item paired score
differences. Anytime validity means the coverage guarantee holds at
*any* data-dependent stopping time: the admin can look at the verdicts
as often as they like, stop whenever a pair resolves, and the reported
confidence interval is still valid. No peeking penalty bookkeeping, no
pre-registered sample size required.

This is a research prototype (v0.9). It accompanies a position paper
on budget allocation in TTS evaluation (under submission, *Computer
Speech & Language* Special Issue on the evaluation of speech and
speech synthesis).

## What the admin sees

For each pair of systems (A, B), one of six verdicts:

| Verdict | Meaning |
| --- | --- |
| `STOP` | The anytime-valid CI excludes 0 (0 ∉ CI): the pair is separated. |
| `STOP_EQUIV` | Strict TOST equivalence: CI ⊂ ±δ_equiv *and* the half-width Ψ_eff ≤ δ_equiv/2. The pair is statistically equivalent within tolerance. |
| `CONTINUE +N` | Neither verdict yet. The tool predicts how many more **items** (+N) or **listeners** (+L) would close the pair, and which terminal (STOP or STOP_EQUIV) it is heading for. The headline is the cheaper option. |
| `WAIT` | The listener×system variance component cannot be estimated yet (fewer than 3 listeners with ≥ 2 paired items). Structural: keep collecting before trusting any verdict. |
| `CORPUS_LIMIT` | The item budget is exhausted *and* the listener lever can no longer close the pair. Terminal. |
| `LISTENER_LIMIT` | The listener budget is exhausted *and* the item lever can no longer close the pair. Terminal. |

> **Verdict naming.** `STOP` is the separation verdict the companion paper
> calls `STOP_SEP`. `STOP` is the current public label and will be **renamed to
> `STOP_SEP` in a future release** (to pair symmetrically with `STOP_EQUIV`);
> code matching on `"STOP"` should expect that change.

Verdicts are recomputed from scratch at every call — the controller is
queried on demand (a handful of times over a test lifecycle), not in a
real-time loop. A `STOP` is **not sticky**: if you keep collecting
past it, a borderline pair can revert to `CONTINUE`. This is honest
anytime-CI behaviour (the guarantee is on coverage, not on the
monotonicity of the observed difference); in practice the admin acts
on the first `STOP`.

## Install

```bash
pip install -e .          # from the repository root (pyproject.toml)
pip install -e ".[ui]"    # + Streamlit demo UI
pip install -e ".[test]"  # + pytest; then run `pytest`
```

Requires Python ≥ 3.10. Pinned versions used for development are in
`requirements.txt`.

## Quick start

### Programmatic

```python
import pandas as pd
from anytimemonitor import AnytimeMonitor

ctrl = AnytimeMonitor(mde_threshold=0.30)   # δ_sep, in MOS units

# Feed ratings as they arrive — long format, one row per judgment
ctrl.add_ratings(pd.DataFrame({
    "listener_id": ...,   # who rated
    "item":        ...,   # which sentence / stimulus
    "system":      ...,   # which TTS system
    "rating":      ...,   # the MOS score
}))

print(ctrl.report())                  # all pairs
dec = ctrl.decision("sysA", "sysB")   # one pair, structured
print(dec.verdict, dec.ci_lo, dec.ci_hi, dec.n_more_items)
```

### CLI demo

Feed a ratings CSV to the controller batch by batch and watch the per-pair
verdicts tighten. With no arguments it runs on a bundled synthetic sample:

```bash
python -m anytimemonitor.demo                          # bundled sample
python -m anytimemonitor.demo --input my_ratings.csv   # your own long-format CSV
```

### Streamlit demo UI

```bash
streamlit run anytimemonitor/ui/app.py
```

Upload a ratings CSV (or pick a bundled synthetic example from
`sample_uploads/`), watch the controller replay the stream
batch-by-batch, and drag the time-machine slider to see the verdict
and two-lever advice at any past timestep. See `ui/README.md`.

## How it works

For a pair (A, B), the controller builds per-item paired differences
d_j = mean rating of item j under A minus under B, and maintains a
sub-Gaussian normal-mixture confidence sequence on their mean
(Howard et al. 2021). Two ingredients adapt this to listening-test
data, where ratings are clustered by listener as well as by item:

- **Bivariate cluster bootstrap SE.** Resampling *items* gives the
  i.i.d.-part SE that the anytime boundary multiplies; resampling
  *listeners* gives the listener-pool floor ("plancher") — the
  precision ceiling that no amount of extra items can break, added in
  quadrature. When the data is too sparse to bootstrap, the
  controller falls back to the confidence sequence's internal running
  variance proxy.
- **Variance decomposition layer.** Method-of-moments estimates of
  the system×item and residual components (per system, with a
  Half-Cauchy Bayesian regulariser that prevents the
  truncation-at-zero bias of raw MoM), plus a per-pair estimate of
  the listener×system component. These feed the two-lever
  prediction: +N items shrinks the interaction term, +L listeners
  lowers the floor.

The price of anytime validity is a modestly wider interval than a
fixed-n test (a √log log factor), and deliberate conservatism early
on: below `avci_t_min_ebernstein` items the variance proxy falls back
to the worst-case Hoeffding bound, so the first batches produce wide
intervals. The controller is therefore most useful from a few dozen
rated items per system onward.

The internal architecture is documented in
[`Architecture.md`](Architecture.md) (and
[`ui/Architecture.md`](ui/Architecture.md) for the demo UI); the deeper design
notes for the confidence sequence are in [`AVCI_PLAN.md`](AVCI_PLAN.md).

## Key parameters

All knobs are constructor arguments of `AnytimeMonitor`; the
defaults are deliberately conservative.

| Parameter | Default | Role |
| --- | --- | --- |
| `mde_threshold` (δ_sep) | (required) | The smallest effect worth *resolving* (MOS): seeds `delta_equiv` and the reported MDE. It does **not** gate `STOP` — separation fires on `0 ∉ CI` regardless. |
| `delta_equiv` | `None` (= `mde_threshold`) | δ_equiv — equivalence margin for `STOP_EQUIV`. Set it below `mde_threshold` to create an indifference zone. |
| `alpha` | 0.05 | Nominal level of the confidence sequence. |
| `power` | 0.80 | Only sets the reported-MDE scale factor (`z_{α/2} + z_power`); does not affect stopping. |
| `n_max_corpus`, `l_max_listeners` | `None` | Item / listener budgets; enable the `*_LIMIT` terminal verdicts. |
| `avci_n_planned` | `240` (§7 census median budget) | Planned item budget; the controller derives η = 1/√(N_planned) (Howard's a-priori rule). |
| `avci_eta` | `None` (→ 1/√`avci_n_planned`) | Set the Howard 2021 mixing parameter η by hand instead of via `avci_n_planned` — η is fixed a priori, never tuned to data. **Pass one or the other, not both** (raises otherwise). |
| `avci_lissys_floor_r0` | `0.36` (measured r_lisSys band frontier) | A-priori guard on the listener floor: bounds it below by √(2·σ²₀/L) with σ²₀ = 1.5·r₀·σ̂²_eval (the ×1.5 is an MC-calibrated validity margin). Protects coverage when a small pool under-estimates σ²_lisSys. `0` disables. |
| `avci_lissys_floor_sigma2` | `None` | Absolute σ²₀ override for the guard (expert path, no extra margin). **Exclusive with `avci_lissys_floor_r0`** (raises otherwise). The per-pair `Decision.lissys_prior_exceeded` flags when data-driven floor estimates exceed the declared guard (declared margin exhausted; consider raising r₀). |
| `avci_gamma_alpha_factor` | 1.0 | Multiplier `c` on the listener floor (√c on the plancher); `1.0` assumes σ²_lisSys ≤ σ²_int. Bypassed when the bootstrap plancher is available (the normal path). |
| `avci_proxy` | `"ebernstein"` | Variance proxy: running empirical variance (`ebernstein`) or worst-case `hoeffding`. |
| `avci_d_bounds` | `(-4, 4)` | Range of the per-item paired *difference* (not the raw rating scale): ±(rating max − min), so the default spans a rating range of 4. The normal path bootstraps the SE from the data and **ignores this** (it feeds only the sparse-data fallback and a reported diagnostic); widen it for a larger rating range if you drive `AnytimeCI` directly on sparse data. |
| `avci_t_min_ebernstein` | 10 | Below this item count the `ebernstein` proxy falls back to `hoeffding` (fallback path only). |
| `avci_ebernstein_floor_frac` | 0.01 | Relative floor on the empirical-variance proxy, as a fraction of the `hoeffding` bound (fallback path only). |
| `avci_bootstrap_B` | 200 | Bootstrap resamples per decision call. |
| `avci_bootstrap_seed` | 0 | PRNG seed for the bivariate cluster bootstrap. |
| `sigma2_ga_aggregator` | `"median"` | Which bootstrap-CI point anchors the listener floor: `median` / `mean` / `q95` (conservative) / `None`. |
| `cold_start_ga_ratio` | 2.7 | Conservative BTTF-anchored prior ratio (σ²_lisSys/σ²_int) used before the component is identifiable. |
| `avci_wait_min_listeners_kge2` | 3 | Structural WAIT gate: minimum listeners with ≥ 2 paired items. |
| `avci_wait_max_ci_ratio` | 5.0 | WAIT gate: max tolerated q95/median on the σ²_lisSys bootstrap CI before abstaining. |

> **Legacy naming.** Parameters and `Decision` fields carrying `ga` /
> `gamma_alpha` (e.g. `cold_start_ga_ratio`, `sigma2_ga_*`) name the
> listener×system variance the paper writes `σ²_lisSys`; the identifiers will be
> **renamed to `lissys` in a future release** (an API change, hence deferred).

## Input data requirements

- Long format, columns `listener_id`, `item`, `system`, `rating`
  (extra columns are ignored).
- Ratings on any bounded scale. The controller works on per-item paired
  *differences*, whose extremes are ±(rating max − min); the default
  `avci_d_bounds = (-4, 4)` spans a rating range of 4. On the normal path the
  standard error is bootstrapped from the data, so the scale is handled
  automatically and this default is safe on any scale; `avci_d_bounds` only
  enters the sparse-data fallback (when the cluster bootstrap cannot run).
- The design must let listeners rate **multiple items per system**
  (K ≥ 2 somewhere): a design where each listener rates a single item
  — or each item is rated by a single listener — cannot identify the
  listener×system component, and the controller will honestly report
  `WAIT` forever rather than emit verdicts blind to that term.

## Repository layout

```text
anytimemonitor/
├── __init__.py           ← public API: AnytimeMonitor, Decision,
│                           VarianceEstimate, estimate_components
├── controller.py         ← the AnytimeMonitor controller
├── avci.py               ← AnytimeCI — Howard 2021 sub-Gaussian
│                           normal-mixture confidence sequence
├── variance.py           ← MoM variance components + Half-Cauchy
│                           regulariser
├── demo.py               ← CLI example (batch-fed on a bundled sample)
├── ui/                   ← Streamlit demo (app.py; see ui/README.md + ui/Architecture.md)
├── sample_uploads/       ← synthetic example CSVs for the UI
├── Architecture.md       ← internal architecture (controller + statistics)
├── AVCI_PLAN.md          ← deeper design notes for the confidence sequence
├── TODO.md               ← development roadmap
├── archive/              ← retired code, notes and frozen runs
├── LICENSE               ← MIT
└── requirements.txt      ← pinned development stack
```

The validation and simulation harnesses (synthetic-data generator,
controller-vs-oracle simulator, false-positive-rate check) are not part
of the installed package; they ship with the companion paper's
reproduction code.

## Status and caveats

- **Research prototype.** The API may still change. A `pytest` smoke-test
  suite (`anytimemonitor/tests/`) checks the package runs end to end; the
  controller has additionally been validated by simulation
  (coverage under H₀, controller-vs-oracle gap decomposition) and
  exercised on Blizzard Challenge data; it has not yet been plugged
  into a live test platform.
- **Planning aid first.** The anytime-valid CI is a valid inference
  under optional stopping, so it can be reported as such — but the
  two-lever predictions (+N items / +L listeners) are noisy
  projections from running variance estimates, not guarantees.
- A full-scale validation campaign (multi-seed, multi-corpus
  false-positive and power sweeps) is planned alongside the companion
  paper; the numbers shipped here come from smaller smoke-scale runs.

## License

MIT — see [LICENSE](LICENSE).

## Citation

A companion paper is under submission. Until it is out, please cite
this repository directly.
