Metadata-Version: 2.4
Name: vernier
Version: 0.5.1
Classifier: Development Status :: 1 - Planning
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=15
Requires-Dist: rfdetr==1.6.5.post0 ; extra == 'real-models'
Requires-Dist: platformdirs>=4 ; extra == 'real-models'
Requires-Dist: torch>=2.4 ; extra == 'real-models'
Requires-Dist: transformers>=5.1 ; extra == 'real-models'
Requires-Dist: huggingface-hub>=0.27 ; extra == 'real-models'
Requires-Dist: pillow>=10 ; extra == 'real-models'
Requires-Dist: timm>=1.0 ; extra == 'real-models'
Requires-Dist: torchmetrics>=1.5 ; extra == 'real-models'
Requires-Dist: polars>=1.0 ; extra == 'tables'
Requires-Dist: torch>=2.4 ; extra == 'torch'
Requires-Dist: plotly>=6.0 ; extra == 'viz'
Provides-Extra: real-models
Provides-Extra: tables
Provides-Extra: torch
Provides-Extra: viz
License-File: LICENSE-APACHE
License-File: LICENSE-MIT
Summary: High-performance, parity-preserving COCO-style evaluation
Keywords: evaluation,metrics,computer-vision,detection,coco,object-detection
Author: The vernier authors
License: MIT OR Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://github.com/NoeFontana/vernier#readme
Project-URL: Homepage, https://github.com/NoeFontana/vernier
Project-URL: Issues, https://github.com/NoeFontana/vernier/issues
Project-URL: Repository, https://github.com/NoeFontana/vernier

# vernier

[![PyPI](https://img.shields.io/pypi/v/vernier.svg)](https://pypi.org/project/vernier/)
[![Python](https://img.shields.io/pypi/pyversions/vernier.svg)](https://pypi.org/project/vernier/)
[![Crates.io](https://img.shields.io/crates/v/vernier.svg)](https://crates.io/crates/vernier)
[![License](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue.svg)](#license)

**Fast, auditable evaluation for 2D vision models.** Detection, instance and
panoptic segmentation, semantic segmentation, keypoints, and LVIS, all in one
package with a Rust core, a Python API, and a standalone CLI.

- **Bit-exact** with `pycocotools==2.0.11`, `panopticapi`, `lvis-api` and
  `boundary-iou-api` in strict mode. Every upstream quirk has a documented
  disposition ([quirks survey](docs/engineering/pycocotools-quirks.md)).
- **Drop-in** for `pycocotools.cocoeval.COCOeval`: change one import, or none.
  Same constructor, same `params` mutations, same `eval` / `evalImgs` /
  `ious` / `stats` read back off the instance ([what carries
  over](docs/migrate/from-pycocotools.md#params-what-the-shim-honors-and-what-it-rejects)).
- **4–22× faster** than faster-coco-eval and pycocotools at equal CPU budget,
  measured on real detector output rather than jittered ground truth
  ([benchmarks](#performance)).
- **Built for real pipelines**: training-loop evaluation, multi-rank
  gathering, per-image tables, error decomposition, calibration, scenario
  slicing.

## Install

```sh
pip install vernier               # Python ≥ 3.10, abi3 wheels
cargo binstall vernier-cli        # standalone `vernier` binary, no Python needed
cargo add vernier                 # Rust library
```

## Quickstart: the COCO API you already know

`vernier.COCOeval` has the same constructor, the same
`evaluate() / accumulate() / summarize()` sequence, and the same `.stats`
as pycocotools. It defaults to `parity_mode="strict"`, so the output is
bit-identical (see [Status & validation](#status--validation) for what
that is measured against).

```python
from pycocotools.coco import COCO
from vernier import COCOeval  # was: from pycocotools.cocoeval import COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.loadRes("detections.json")

E = COCOeval(coco_gt, coco_dt, iouType="bbox")  # "segm" | "keypoints" | "boundary"
E.evaluate()
E.accumulate()
E.summarize()
```

**Code you can't edit** (mmdetection, detectron2, ultralytics, …): patch
the symbol before anything imports `pycocotools.cocoeval`.

```python
import vernier

unpatch = vernier.patch_pycocotools()  # pycocotools.cocoeval.COCOeval -> vernier
run_existing_eval()
unpatch()
```

The patch is explicit, reversible, and never happens on import. A context
manager (`vernier.adapters.patched_pycocotools`) and a one-fixture pytest
recipe are in the [pycocotools migration guide](docs/migrate/from-pycocotools.md#pytest-integration).

TorchMetrics' `MeanAveragePrecision` runs under the patch unchanged,
including `class_metrics=True` and `extended_summary=True`
(`tests/python/test_compat_torchmetrics.py` asserts the patched and
unpatched results are equal). Two `params` fields are the exception and
raise rather than diverge quietly: `imgIds` subsetting, and `areaRng`
(custom area ranges are a native-`Evaluator` feature, ADR-0040). The
[migration guide](docs/migrate/from-pycocotools.md#params-what-the-shim-honors-and-what-it-rejects)
has the field-by-field table.

## Recommended: the native API

The shim exists for migration. New code should use the native `Evaluator`:
immutable configuration, typed results, no pycocotools dependency, and
access to everything below.

```python
from pathlib import Path
from vernier.instance import Bbox, CocoDataset, Evaluator

gt = CocoDataset.from_json(Path("instances_val2017.json").read_bytes())
dt = Path("detections.json").read_bytes()

evaluator = Evaluator(iou=Bbox(), parity_mode="strict")
summary = evaluator.evaluate(gt, dt, num_threads=8)

print("\n".join(summary.pretty_lines()))  # the familiar 12-line table
ap = summary.stats[0]
```

> The native `Evaluator` defaults to `parity_mode="corrected"`, which applies
> the [itemized fixes](docs/engineering/pycocotools-quirks.md) to upstream
> bugs. Pass `"strict"` when your numbers must match published pycocotools
> results.

**Inside a training loop**, evaluate on a background worker and feed it
tensors directly (torch, JAX, CuPy, NumPy via DLPack, zero-copy):

```python
with evaluator.background(gt) as bg:
    for images, targets in val_loader:
        preds = model(images)
        bg.submit([{"image_id": int(t["image_id"]), **p} for t, p in zip(targets, preds)])
    summary = bg.finalize()
```

**Per-image and per-class diagnostics** as Polars DataFrames
(`pip install "vernier[tables]"`):

```python
result = evaluator.evaluate(gt, dt, tables="all")
result.per_class
```

### Paradigms

Pick the submodule that matches your model's output. They have different
data models and matching rules, so they are separate evaluators rather than
one class with a mode switch ([why](docs/explanation/three-paradigms.md)).

| Submodule | Input | Metrics |
| --- | --- | --- |
| `vernier.instance` | Scored detections: boxes, masks, keypoints | AP / AR (bbox, segm, boundary, OKS), LVIS federated AP |
| `vernier.panoptic` | Panoptic PNGs + `segments_info` | PQ / SQ / RQ, boundary PQ |
| `vernier.semantic` | Class-id label maps | mIoU, FWIoU, pixel accuracy, mean accuracy |

### Beyond the COCO API

| Need | Feature | Guide |
| --- | --- | --- |
| Evaluate without blocking training | `Evaluator.background(...)` | [how-to](docs/how-to/background-evaluator.md) |
| Evaluate across DDP ranks | `evaluate_to_partial` / `from_partials` | [how-to](docs/how-to/distributed-eval.md) |
| Find which images/classes regressed | `tables="all"` | [how-to](docs/how-to/result-tables.md) |
| Explain an AP gap | TIDE error decomposition, oLRP | [tutorial](docs/tutorials/debugging-with-tide.md) |
| Check score calibration | ECE / MCE / reliability (`calibration=True`) | [how-to](docs/how-to/calibration.md) |
| Metrics per weather, time of day, … | Manifest slicing, `vernier aggregate` (mPC / rPC) | [how-to](docs/how-to/scenario-slicing.md) |
| Non-standard IoU / recall / area grids | `iou_thresholds=`, `recall_thresholds=`, `area_ranges=` | [how-to](docs/how-to/custom-evaluation-grids.md) |

## CLI

A static binary for CI gates and robotics replay pipelines. Output is
byte-deterministic (sorted keys, no timestamps), so artifacts diff cleanly.

```sh
vernier eval --gt gt.json --dt dt.json --iou-type bbox                  # pycocotools-identical stdout
vernier eval --gt gt.json --dt dt.json --iou-type segm --emit json=result.json --threads 8
```

Exit codes: `0` success, `1` evaluation error, `2` invalid arguments.
Full reference: [`crates/vernier-cli`](crates/vernier-cli/README.md).

## Status & validation

Every row is checked by a parity harness that runs the reference
implementation and vernier on the same inputs. "Bit-exact" means
`parity_mode="strict"`, against the reference **as published on PyPI**
— vernier's own output is identical on x86-64 and ARM. (A pycocotools
you recompile from source can round bbox IoU 1-2 ULP differently on
ARM; see [ADR-0056](docs/adr/0056-pin-no-fp-contraction-for-bbox-iou.md)
and the [migration guide](docs/migrate/from-pycocotools.md#bit-for-bit-and-against-which-build).)

| Metric | Reference | Parity | Notes |
| --- | --- | --- | --- |
| bbox / segm / keypoints AP | `pycocotools==2.0.11` | bit-exact | |
| Boundary AP | `boundary-iou-api` | bit-exact | |
| LVIS federated AP | `lvis-api` 0.5.3 | bit-exact | full v1 val, bbox |
| Panoptic PQ, boundary PQ | `panopticapi` (single-core path) | bit-exact | Cityscapes panoptic deferred |
| Semantic mIoU / FWIoU / pAcc / mAcc | `mmseg.IoUMetric` v1.2.2 (vendored) | bit-exact on class marginals | [ADR-0036](docs/adr/0036-vendor-mmsegmentation-ioumetric.md) proposed; ADE20K-scale check pending |
| oLRP | clean-room NumPy oracle | ≤ 1e-9 | panoptic not supported |
| Calibration (ECE / MCE) | clean-room NumPy oracle | bit-exact | detection family only |
| TIDE thresholds (segm, boundary) | none | corrected only | [ADR-0022](docs/adr/0022-tide-thresholds.md) proposed |

Parity model: [ADR-0002](docs/adr/0002-three-tier-parity-model.md).
Library-by-library comparison: [`docs/comparison.md`](docs/comparison.md).

## Performance

Median wall time at a one-CPU budget (CPU/wall is 1.00 in every cell).
COCO val2017 throughout, except the LVIS row, which is LVIS v1 val.
Speedup is the other library's time divided by vernier's.

The **DT source** column is load-bearing. A *jittered* workload perturbs the
ground truth to synthesise detections, which inherits the GT's per-image
class distribution — not the distribution a detector actually produces. The
bbox and segm rows are measured on real RF-DETR predictions on val2017
(513,808 and 430,516 detections) and the keypoints row on real ViTPose
output; the remaining rows still run against synthetic DT, and are
labelled as such rather than presented as equivalent.

<!-- Hand-mirrored: the bbox/segm/keypoints rows from
     docs/engineering/benchmarking/2026-09-rfdetr-real-predictions.md, the rest
     from docs/benchmarks.md (generated by tools/render_benchmarks.py).
     After a bench round, re-run the renderer, then update these numbers. -->

| Workload | DT source | vernier | vs pycocotools | vs faster-coco-eval | vs hotcoco |
| --- | --- | ---: | ---: | ---: | ---: |
| bbox AP | RF-DETR Nano | 993 ms | 22.4× | 5.4× | 2.3× |
| segm AP | RF-DETR Seg-Nano | 2.38 s | 9.4× | 4.2× | 2.0× |
| keypoints AP | ViTPose-B | 129 ms | 18.6× | 6.1× | 1.6× |
| boundary AP | jittered GT | 3.15 s | 19.8× ¹ | 16.9× | — |
| Panoptic PQ | perfect DT | 10.5 s | 3.3× ² | — | — |
| Semantic mIoU | perfect DT | 2.87 s | 14.1× ³ | — | — |
| LVIS v1 bbox AP | jittered GT | 2.50 s | 81.0× ⁴ | — | 1.6× |

¹ boundary-iou-api · ² panopticapi · ³ mmsegmentation · ⁴ lvis-api, with 13.6× lower peak memory (1.11 vs 15.08 GiB)

A second real detector on the same kernel and GT gives a different point
estimate: RF-DETR Seg-Nano's bbox output (16% fewer detections, higher AP)
measures 15.4× / 4.0× / 1.9×. Treat a single-workload speedup as a point
estimate, not a property of the library — the
[full snapshot](docs/engineering/benchmarking/2026-09-rfdetr-real-predictions.md)
gives per-stage breakdowns, including the one cell where hotcoco's evaluate
kernel is faster than vernier's.

Peak RSS on the real bbox cell is 395 MiB, against 997 MiB (hotcoco),
1383 MiB (faster-coco-eval) and 1429 MiB (pycocotools) — a 2.5–3.6×
advantage. Every implementation's working set grows with real detection
density, but not by the same factor: against hotcoco the gap widens from
1.4× on the synthetic cell to 2.5× here, because hotcoco's per-detection
Python-object overhead scales with detection count while vernier's binary
ingest does not.

<details>
<summary>Thread scaling (<code>num_threads</code>, 8-vCPU host)</summary>

faster-coco-eval ≥ 1.8 and hotcoco are multi-threaded by default, so each
column compares equal CPU budgets.

| Workload | 1 | 2 | 4 | 8 | vs hotcoco @ 8 | vs faster-coco-eval @ 8 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| bbox | 306 ms | 221 ms | 146 ms | 131 ms | 2.8× | 11.4× |
| segm | 926 ms | 536 ms | 304 ms | 239 ms | 2.3× | 14.6× |
| boundary | 3.18 s | 1.68 s | 866 ms | 711 ms | — | 23.8× |
| keypoints | 139 ms | 90 ms | 60 ms | 52 ms | 3.1× | 14.7× |
| bbox, Objects365 (1.06M dets) † | 6.6 s | — | — | 3.2 s | 2.6× | OOM at ~30 GiB |

Strict-mode results are bit-identical across thread counts.
† Single-rep dev run; pycocotools takes ~6 min per rep at this size. The
faster-coco-eval OOM is carried forward from the previous round rather
than re-measured — reproducing a 30 GiB OOM destabilises the host
mid-round.

</details>

AMD EPYC-Milan KVM VPS (4 cores x 2 threads = 8 logical CPUs), harness
mode `release` (N=10 measurement reps + 2 warmup, randomised impl order,
5% relative-IQR gate), against `hotcoco==1.0.1`,
`faster-coco-eval==1.8.0` and `pycocotools==2.0.11`. Full methodology,
every baseline pin and the complete results:
[`docs/benchmarks.md`](docs/benchmarks.md) for the synthetic-DT rows,
[the RF-DETR snapshot](docs/engineering/benchmarking/2026-09-rfdetr-real-predictions.md)
for the real-prediction ones.

## Documentation

- [Tutorials](docs/tutorials/): first evaluation, training-loop integration
- [Migration guides](docs/migrate/): pycocotools, faster-coco-eval,
  panopticapi, lvis-api, mmsegmentation
- [How-to guides](docs/how-to/) · [Reference](docs/reference/) ·
  [Design decisions (ADRs)](docs/adr/)

## Contributing

```sh
just lint && just test && just audit
```

See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the ADR workflow, vendoring
policy, and code style.

## License

Licensed under either of [Apache-2.0](LICENSE-APACHE) or [MIT](LICENSE-MIT)
at your option. Test-only reference implementations used for parity checks
are listed in [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). They are not
shipped in wheels or binaries.

