Metadata-Version: 2.4
Name: annotations-evaluate
Version: 0.1.0
Summary: Span annotation metrics: nested standard report and a SemEval-2013 compatible flat mode
Keywords: evaluation,nested-tagging,metrics,ner,nlp,semeval
Author: Nicole Dresselhaus
Author-email: Nicole Dresselhaus <nicole.dresselhaus@hu-berlin.de>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11, <4.0
Project-URL: Repository, https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate
Project-URL: Issues, https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate/-/issues
Description-Content-Type: text/markdown

# annotations-evaluate

Python library for evaluating span annotations — built for **nested and
overlapping** annotations as they occur in historical corpora (BeNASch/HLR-GT)
and in LLM-assisted tagging. It provides a nested report by default and an
opt-in, SemEval-conformant flat mode for comparability with published NER
numbers.

Part of the `annotations-*` family: the annotation pipeline lives in
[`annotations4all`](https://scm.cms.hu-berlin.de/annotations4all/annotations4all),
evaluation orchestration in
[`annotations-orchestration`](https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration);
run projects combine both. This library is the family's home for metrics.

## Why a separate library

- The previous evaluation setup relied on a vendored `nervaluate` snapshot
  (identical to PyPI `nervaluate` 0.2.0) which **miscounts as soon as
  predictions are not exactly equal to a gold span** — also on flat data: one
  prediction is scored once per overlapping gold span, and a differently
  labelled prediction "consumes" a gold span so that the actually matching
  prediction is counted as spurious.
- Upstream `nervaluate` 1.2.x supports nesting but thinks flat: no level, no
  parent–child structure, no crossing, and undecodable model output shows up as
  an ordinary error.
- What a state means is research work — it belongs in a versioned spec in this
  repository, not in a vendored snapshot.

Background, measurements and the decisions E1–E12 are documented in the
project's internal notes.

## Features

- **Nested report (default)**: precision/recall/F1 per nesting level and per
  element type; structural precision/recall/F1 over parent–child edges (catches
  "spans correct, nesting wrong"); recall bucketed by span length and depth;
  decode rate including a breakdown by parser warning code (`unparsed` is its
  own state).
- **Flat mode (opt-in)**: `strict`, `exact`, `partial`, `ent_type` following
  SemEval-2013, validated against the reference example in the `nervaluate`
  documentation and against `nervaluate` 1.2.1.
- **Deterministic matching**: per label, order-independent, each span matched at
  most once.
- **Half-open spans** (`[start, end)`), consistent with the family convention.
- **Re-scoring** (`annotations-rescore`): re-scores stored predictions without
  calling a model — nested and/or flat, with an old-versus-new table against the
  metrics recorded by earlier runs. The completeness of the input (read,
  ignored, without predictions) is part of the report, and an incomplete report
  is marked as such.
- **Record adapters** (`document_from_record`, `spans_from_ground_truth`,
  `spans_from_predictions`, `decode_from_predictions`): records held in memory
  (harness `prodigy`/`tags` shapes) become documents of spec §2.6 directly — the
  same translation `annotations-rescore` applies per line. `predictions` may be
  the plain span list or the mapping with `answers`/`unparsed` (§2.6), so a
  re-scored run reports a decode rate (§6.2.4) when it was recorded instead of
  `n/a`.

## Usage

```python
from annotations_evaluate import evaluate

documents = [
    {
        "document_id": "doc-1",
        "text": "Der Karl Schmidt liest in Wien.",
        "gold": [
            {"label": "PERSON", "start": 4, "end": 16},
            {"label": "CITY", "start": 26, "end": 30},
        ],
        "prediction": {
            "spans": [
                {"label": "PERSON", "start": 4, "end": 16},
                {"label": "CITY", "start": 26, "end": 29},
            ],
            "answers": 1,
            "unparsed": [],
        },
    }
]

nested = evaluate(documents)  # nested report — the default mode
print(nested.overall.counts)
# StateCounts(correct=1, part_boundary=1, correct_label_other_bounds=0, spurious=0, missed=0, duplicates=0)
print(nested.overall.f1)  # 0.75
print(nested.element["CITY"].counts)
# StateCounts(correct=0, part_boundary=1, correct_label_other_bounds=0, spurious=0, missed=0, duplicates=0)
print(nested.decode.answers, nested.decode.decode_rate)  # 1 1.0 — the answer was decoded

flat = evaluate(documents, mode="flat")  # SemEval-2013 scenarios (opt-in)
print(flat.scenarios["strict"].f1, flat.scenarios["partial"].f1, flat.overlap_threshold)
# 0.5 0.75 1.0
```

Documents are the records of spec §2.6: `gold` and `prediction.spans` are half-open
spans (`[start, end)`), `answers`/`unparsed` carry the decode metadata of the run, so
the report can name a decode rate instead of `n/a`.

Recorded runs are re-scored without a model call:

```bash
annotations-rescore results/runs/<run_id> --mode both \
    --legacy-metrics results/runs/<run_id>/metrics.json
```

## Specification

The evaluation semantics are normative and versioned; terms, span relations,
deterministic matching, the state catalogue and both metric modes are defined in
[docs/evaluation-semantics.md](https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate/-/blob/main/docs/evaluation-semantics.md)
(Spec 1.1, English).
It is the implementation contract and the reference for every published number.

## Scope

- No scheduler and no model calls — that is `annotations-orchestration`'s and
  the run project's job.
- No prompt or parser logic — that lives in `annotations4all`.
- No runtime dependencies: standard library only.

## Installation

```bash
pip install annotations-evaluate
```

The package is published on PyPI as `annotations-evaluate`; the version follows
[Semantic Versioning](https://semver.org/spec/v2.0.0.html), the metrics follow the
spec version of `docs/evaluation-semantics.md`.

## Development

```bash
uv sync
uv run ruff check .
uv run ruff format --check .
uv run pytest
```

`.agent-memories/` is gitignored (family convention). `README.md` carries the
repository metadata front matter and is excluded from release archives; the PyPI
readme is this file.

## Citation

If you use this software in academic work, please cite it as:

> Dresselhaus, Nicole. (2026). *annotations-evaluate* (Version 0.1.0)
> [Software]. Humboldt-Universität zu Berlin.
> <https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate>

DOI: `10.5281/zenodo.22795383`

The concept DOI `10.5281/zenodo.22795382` always resolves to the latest
version.

Machine-readable metadata is in
[`CITATION.cff`](https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate/-/blob/main/CITATION.cff).

## License

MIT.
