Metadata-Version: 2.4
Name: idaniloju
Version: 0.1.1
Summary: Conformal prediction from scratch: the exact finite-sample guarantee, what drift does to it, and the online repair
Author-email: Kenny Obidele <obidelek19@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/Kenny0bi/idaniloju
Project-URL: Issues, https://github.com/Kenny0bi/idaniloju/issues
Keywords: conformal-prediction,uncertainty-quantification,prediction-intervals,distribution-free,drift
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.24

# idaniloju

Conformal prediction from scratch: the exact finite-sample guarantee,
proof by experiment, the deployment that breaks it, and the online
repair. The tenth and last project of this series, and it closes the
loop: the drift that kills the guarantee here is detected by
[iyipada](https://github.com/Kenny0bi/iyipada), my drift library,
installed from PyPI like any other user would.

```
pip install idaniloju
```

*Idaniloju* is Yoruba for assurance. Conformal prediction is the cleanest
promise in machine learning: wrap ANY point predictor, and its intervals
will contain the truth 90% of the time, guaranteed at finite sample size
with no distributional assumptions at all. The entire method is one
counting argument, and this repository walks it from the theorem to the
hurricane that broke it.

## The guarantee is a law, not a hope

![The law](https://raw.githubusercontent.com/Kenny0bi/idaniloju/main/assets/law.svg)

600 fresh train/calibrate/test splits of the same data. Conditional
coverage is not just ">= 90% on average": it has a known exact
distribution, Beta(n+1-l, l) over the calibration draw (Vovk 2012). The
600-run empirical mean lands on the law's mean to the fourth digit
(0.9001 vs 0.9001), and the histogram sits inside the density like it
was poured there. Also visible: 47% of individual runs are below 0.90,
which is exactly what the law says should happen, and exactly the fine
print people miss.

![The animation](https://raw.githubusercontent.com/Kenny0bi/idaniloju/main/assets/ranks.gif)

The whole proof: an exchangeable point's score has uniform rank among
the n+1 scores, so it falls at or below the ceil((n+1)(1-alpha))-th
calibration score with probability at least 1-alpha. Counting, nothing
else. (Source: [assets/manim_ranks.py](https://raw.githubusercontent.com/Kenny0bi/idaniloju/main/assets/manim_ranks.py), video in
[assets/ranks.mp4](https://raw.githubusercontent.com/Kenny0bi/idaniloju/main/assets/ranks.mp4).)

## Then you deploy it, and the fine print detonates

Train on early 2011 of the Washington DC bike-sharing panel, calibrate
on late 2011, then serve every hour of 2012 in order:

![The break](https://raw.githubusercontent.com/Kenny0bi/idaniloju/main/assets/break.svg)

2012 ridership grew about 65% over 2011. Exchangeability with the
calibration set is simply false, and the guarantee follows it down:
78% delivered against 90% promised for the year, and when Hurricane
Sandy closes the city on October 29 (22 rides all day against a normal
7,500), the rolling window bottoms out at 42%.

Three things about that figure that I care about:

- **iyipada alarms in the first week of 2012** (PSI 0.15 on the model's
  conformity scores, p = 0.005), eleven months before Sandy. The drift
  was structural, not meteorological.
- **The shuffled control behaves**: run the same machinery on a
  shuffled (exchangeable) version of the stream and coverage holds at
  90.3% with zero alarm weeks out of 51. The alarm is not trigger-happy;
  the guarantee did not break by accident.
- **Adaptive conformal repairs it online.** ACI (Gibbs & Candes 2021)
  moves the miscoverage dial with the observed errors,
  alpha_t+1 = alpha_t + gamma(alpha - err_t), and ends the year at
  89.97% against the 90% target, hurricane included, without ever
  refitting the model.

## What the repair costs

![The price](https://raw.githubusercontent.com/Kenny0bi/idaniloju/main/assets/price.svg)

Nothing is free: the adaptive intervals are about 5% wider in the median
hour, and for 36 hours around Sandy the dial demands more assurance than
4,395 calibration points can certify, so the honest interval is
unbounded. I consider that a feature: a system that says "this week I
can promise nothing" is telling the truth, and the fixed system that
kept quoting tidy intervals through a hurricane was not.

![The anatomy](https://raw.githubusercontent.com/Kenny0bi/idaniloju/main/assets/anatomy.svg)

## Using it

```python
from idaniloju import Ridge, SplitConformal, AdaptiveConformal, coverage

sc = SplitConformal(Ridge(lam=5.0), alpha=0.1)   # any model with fit/predict
sc.fit(X_train, y_train, X_cal, y_cal)
lo, hi = sc.predict(X_new)

aci = AdaptiveConformal(sc, gamma=0.01)           # for streams
lo, hi, err = aci.step(x_t, y_t)
```

`SplitConformal` takes any object with fit/predict (a scale model makes
the intervals difficulty-aware without touching the guarantee), and
`conformal_quantile` is the exact order statistic the theorem is proved
for, not an interpolated quantile: at small n and small alpha it
returns an infinite interval instead of a fake finite one.

## Honest limits

- The guarantee is **marginal**: 90% over everything, not 90% for every
  hour of the day or every kind of input. Group-conditional coverage
  needs group-wise calibration this library does not implement.
- ACI trades a broken assumption for a feedback loop: it needs y_t
  observed reasonably soon after predicting. With delayed labels the
  dial lags exactly that delay.
- The base model here is a ridge regression on purpose (any model
  works). Sharper models give narrower intervals; nothing about the
  coverage story changes.
- The 65% growth and the hurricane make this dataset a vivid drift
  story, which is why I chose it; datasets with gentler drift break more
  slowly and less photogenically.

## Reproduce it

```bash
python -m venv .venv && .venv/bin/pip install numpy pytest iyipada
bash data/get_data.sh                        # or let the loader fetch it

.venv/bin/python -m pytest tests/ -q         # 6 contracts
.venv/bin/python benchmarks/bench_guarantee.py   # the Beta law, 600 refits
.venv/bin/python benchmarks/bench_stream.py      # the 2012 deployment
.venv/bin/python assets/make_visuals.py          # the four figures
```

## Layout

- [idaniloju/conformal.py](https://github.com/Kenny0bi/idaniloju/blob/main/idaniloju/conformal.py) the method: exact
  quantile, split conformal, difficulty scaling, ACI
- [idaniloju/models.py](https://github.com/Kenny0bi/idaniloju/blob/main/idaniloju/models.py) a closed-form ridge, the
  deliberately boring predictor
- [idaniloju/bike.py](https://github.com/Kenny0bi/idaniloju/blob/main/idaniloju/bike.py) the panel, with the year column
  deliberately withheld from the features (deployment does not get told
  the world changed)
- [benchmarks/](https://github.com/Kenny0bi/idaniloju/tree/main/benchmarks) the law and the deployment, everything into
  JSON the figures and tests read

## Papers

- Vovk, Gammerman, Shafer (2005), *Algorithmic Learning in a Random
  World*. The theory.
- Lei et al. (2018), *Distribution-Free Predictive Inference for
  Regression*, JASA. The split conformal recipe used here.
- Vovk (2012), *Conditional validity of inductive conformal predictors*.
  The Beta law the first figure draws.
- Gibbs & Candes (2021), *Adaptive Conformal Inference Under
  Distribution Shift*. The repair.
