Metadata-Version: 2.4
Name: validity-screen
Version: 0.1.0
Summary: Validity screening protocol for LLM confidence signals
Author: Jon-Paul Cacioli
License: MIT
Project-URL: Homepage, https://github.com/synthiumjp/validity-scaling-llm
Project-URL: Documentation, https://github.com/synthiumjp/validity-scaling-llm/tree/master/screen
Project-URL: Repository, https://github.com/synthiumjp/validity-scaling-llm
Project-URL: Issues, https://github.com/synthiumjp/validity-scaling-llm/issues
Keywords: llm,confidence,validity,metacognition,calibration,selective-prediction,screening,psychometrics,evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20
Requires-Dist: scipy>=1.7
Dynamic: license-file

# validity-screen

**Check whether an LLM's confidence signal carries information before you build on it.**

[![PyPI](https://img.shields.io/pypi/v/validity-screen)](https://pypi.org/project/validity-screen/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Implements the screening protocol from:

> Cacioli, J. P. (2026). *Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals.* arXiv.

## Install

```bash
pip install validity-screen
```

## Quick start (Python)

```python
import numpy as np
from validity_screen import screen

# Your data: item-level correctness and confidence
correct    = np.array([True, True, False, True, False, True, True, False])
confidence = np.array([True, True, True,  True, False, True, False, False])

result = screen(correct, confidence, model_name="My Model")

print(result.tier)          # 'Valid', 'Indeterminate', or 'Invalid'
print(result.vrs_table())   # Complete reporting table
```

## Quick start (command line)

```bash
# From a CSV with 'correct' and 'confidence' columns
validity-screen run --data my_data.csv --model-name "GPT-5.4"

# From separate files
validity-screen run --correct correct.txt --confidence confidence.txt

# Continuous confidence? Auto-binarised at median
validity-screen run --data my_data.csv --confidence-col prob --threshold 0.5

# JSON output for pipelines
validity-screen run --data my_data.csv --json
```

## What it does

Before computing calibration metrics (ECE), metacognitive sensitivity (meta-d', AUROC), or selective prediction accuracy, this protocol checks whether the confidence signal carries item-level information about correctness. If it doesn't, those downstream metrics are fitting noise.

Five values from a 2x2 contingency table. Three possible outcomes.

| Tier | Meaning | Action |
|------|---------|--------|
| **Valid** | Confidence tracks correctness | Proceed with downstream metrics |
| **Indeterminate** | Near threshold, uncertain | Compute but flag; consider more items |
| **Invalid** | Confidence does not discriminate | Do not interpret AUROC, ECE, selective prediction |

## Indices

| Index | What it detects | Invalid threshold |
|-------|-----------------|-------------------|
| **L** | Blanket confidence on errors | >= 0.95 |
| **Fp** | Over-withdrawal of correct items | >= 0.50 |
| **RBS** | Inverted monitoring direction | > 0 (CI excludes zero) |
| **TRIN** | Fixed responding | >= 0.95 (warning only) |
| **r** | Item-level sensitivity | Reported, not thresholded |

## Batch screening

```python
from validity_screen import screen_batch, summary_table

models = {
    "GPT-5.4":  {"correct": correct_gpt,  "confidence": conf_gpt},
    "Claude":   {"correct": correct_claude, "confidence": conf_claude},
    "Gemini":   {"correct": correct_gemini, "confidence": conf_gemini},
}

results = screen_batch(models, benchmark_name="MMLU")
print(summary_table(results))
```

## Continuous confidence

```python
from validity_screen import screen, binarise

# Binarise at a fixed threshold
confidence_binary = binarise(confidence_continuous, threshold=50)

# Or at the sample median
confidence_binary = binarise(confidence_continuous, method='median')

result = screen(correct, confidence_binary)
```

## Requirements

- Python >= 3.8
- NumPy >= 1.20
- SciPy >= 1.7

## Citation

```bibtex
@article{cacioli2026screen,
  title={Screen Before You Interpret: A Portable Validity Protocol for
         Benchmark-Based LLM Confidence Signals},
  author={Cacioli, Jon-Paul},
  journal={arXiv preprint},
  year={2026}
}

@article{cacioli2026validity,
  title={Before You Interpret the Profile: Validity Scaling for
         LLM Metacognitive Self-Report},
  author={Cacioli, Jon-Paul},
  journal={arXiv preprint},
  year={2026}
}
```

## License

MIT
