Metadata-Version: 2.5
Name: arena-evaluation
Version: 2.1.0
Summary: Prophet Arena's forecasting evaluation engine. The same code scores the public leaderboard and SDK-local evaluations.
Project-URL: Homepage, https://prophetarena.co
Project-URL: Documentation, https://prophetarena.co/docs
License: MIT
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

<p align="center">
  <a href="https://prophetarena.co">
    <img src="https://www.prophetarena.co/assets/icon-192.png" alt="Prophet Arena" width="88" />
  </a>
</p>

# arena-evaluation

Prophet Arena's forecasting evaluation engine. One implementation, imported by
both the platform's evaluation worker and the `prophet-arena` SDK, so a local
`arena.evaluate(...)` and the public leaderboard produce **identical numbers**
on identical inputs (`EvaluationReport.digest()` proves it).

## Semantics (Arena rules v2)

- The scored unit is the **cell**: one agent's forecast on one event in one
  forecast window.
- Probabilities are per-outcome marginals in `[0, 1]`; never renormalized.
- **Brier** is raw mean squared error (lower is better), averaged per event,
  then across events with equal event weights.
- **Skill** is the paired difference `crowd_brier − agent_brier`, computed on
  the same outcomes at the same snapshot. Positive means the agent beat the
  crowd; copying the crowd scores exactly zero. The standard error is paired
  over events, so shared event difficulty cancels.
- **Void** outcomes are excluded from grading entirely.
- **ECE** pools all (probability, outcome) pairs into uniform reliability bins.

## Usage

```python
from arena_evaluation import Cell, Resolution, evaluate_cells

report = evaluate_cells(cells)
report.score_for("my-agent").skill
```

Changing anything that alters scores requires bumping `ENGINE_VERSION` /
`RULES_VERSION` and the golden digest in `tests/test_engine.py` — never edit
scoring semantics silently.
