Metadata-Version: 2.5
Name: jevkit-calibrate
Version: 0.1.0
Summary: Verify TypeSafe Jev's calibration on your own data. Reliability diagrams, ECE, Brier, and confidence thresholds derived from labeled outcomes.
Project-URL: Homepage, https://github.com/pjdurden/jevkit-py
Project-URL: Issues, https://github.com/pjdurden/jevkit-py/issues
Author: Prajjwal Chittori
License-Expression: MIT
Keywords: calibration,ece,jev,reliability,system-one,thresholds,typesafe
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: jevkit-core>=0.2.0
Description-Content-Type: text/markdown

# jevkit-calibrate

Calibration is Jev's central claim: the probabilities are meant to track real
frequencies, so that among answers given 0.8, about 80% are right. TypeSafe
measures this across groups of predictions and says plainly that it does not
guarantee any individual answer.

This package checks the claim on **your** data, which is the part nobody else
can do for you, and turns the result into a threshold you can defend.

> Unofficial and unaffiliated with TypeSafe.

```bash
pip install jevkit-calibrate
```

Pure standard library. No numpy, no pandas, no plotting stack. A calibration
check that needs a build toolchain is a calibration check that does not get run.

## Measure

```python
from jevkit_core import read_records
from jevkit_calibrate import calibrate, observations_from_records

observations = observations_from_records(read_records("labeled.jevl"))
report = calibrate(observations)

print(report.summary())
print(report.diagram())
```

```
observations: 4000
accuracy:     0.7485
mean claimed: 0.7469  (underconfident by 0.0016)
ECE:          0.0094  (over 10 bins)
MCE:          0.0167
Brier:        0.1681
log loss:     0.5043
```

`ECE` is the average gap between claimed and observed, weighted by how many
observations fall in each bin. `MCE` is the worst gap in any one bin, which is
what catches a healthy-looking ECE hiding one badly wrong region. `Brier` and
`log loss` are proper scoring rules: they reward being calibrated **and**
decisive, so a model that always says 0.5 scores badly even though it is
perfectly calibrated.

## Pick a threshold

TypeSafe's confidence page recommends three bands and says where you draw them
depends on your data. This draws them from the data.

```python
from jevkit_calibrate import recommend_for_accuracy

point = recommend_for_accuracy(observations, target_accuracy=0.95)
if point is None:
    print("no threshold reaches 95% on this data")
else:
    print(f"threshold {point.threshold:.2f}: "
          f"covers {point.coverage:.1%} at {point.accuracy:.1%}, "
          f"{point.errors} wrong answers acted on")
```

`None` is a real answer, not a failure. It means this question cannot be
automated at that bar, and the honest move is to change the question rather
than lower the threshold.

## CLI

```bash
jevkit-calibrate labeled.jevl
jevkit-calibrate labeled.jevl --target-accuracy 0.95
jevkit-calibrate labeled.jevl --min-coverage 0.80
jevkit-calibrate labeled.jevl --max-ece 0.05     # CI gate
jevkit-calibrate labeled.jevl --format json
```

## Which quantity gets calibrated

By default, the probability mass on the chosen outcome, which is what "when it
says 0.8, is it right 80% of the time" means.

`--use-confidence` calibrates the API's `confidence` statistic instead. That is
a different question, and the one to ask when you want to know whether your
*routing* threshold sits in the right place. Nouls carry no confidence, so they
fall back to probability either way.

## License

MIT
