Metadata-Version: 2.4
Name: epb-benchmark
Version: 1.0.1
Summary: Epistemic Pathology Benchmark - The MLPerf of AI Truth Systems
Author-email: Course Correct Labs <hello@coursecorrect.org>
License: MIT
Project-URL: Homepage, https://github.com/Course-Correct-Labs/epb-benchmark
Project-URL: Documentation, https://github.com/Course-Correct-Labs/epb-benchmark/tree/main/docs
Project-URL: Repository, https://github.com/Course-Correct-Labs/epb-benchmark
Project-URL: Issues, https://github.com/Course-Correct-Labs/epb-benchmark/issues
Keywords: ai,benchmark,epistemic,pathology,truthfulness,safety
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: jsonschema>=4.0.0
Requires-Dist: python-Levenshtein>=0.21.0
Requires-Dist: scikit-learn>=1.3.0
Requires-Dist: openai>=1.0.0
Requires-Dist: anthropic>=0.18.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: tqdm>=4.65.0
Requires-Dist: numpy>=1.24.0
Provides-Extra: leaderboard
Requires-Dist: fastapi>=0.104.0; extra == "leaderboard"
Requires-Dist: uvicorn[standard]>=0.24.0; extra == "leaderboard"
Requires-Dist: sqlalchemy>=2.0.0; extra == "leaderboard"
Requires-Dist: python-multipart>=0.0.6; extra == "leaderboard"
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.5.0; extra == "dev"
Dynamic: license-file

# EPB: Epistemic Pathology Benchmark

**The MLPerf of AI Truth Systems**

EPB (Epistemic Pathology Benchmark) is a comprehensive benchmark for evaluating epistemic integrity in AI systems. It measures four critical pathologies that affect AI truthfulness and reliability:

- **Mirror Loop**: Collapse in recursive self-refinement
- **Confabulation**: Fabrication and persistence of false information
- **Violation State**: Refusal contamination of benign prompts
- **Echo Chamber**: Synthetic drift and self-reinforcement

## Quick Start

### Installation

```bash
pip install epb-benchmark
```

Or install from source:

```bash
git clone https://github.com/Course-Correct-Labs/epb-benchmark.git
cd epb-benchmark
pip install -e .
```

### Running the Benchmark

1. Initialize a configuration file:

```bash
epb init-config
```

2. Edit `epb_config.yaml` to set your model and API key:

```yaml
adapter:
  provider: "openai"  # or "anthropic"
  model_name: "gpt-4"
  api_key_env: "OPENAI_API_KEY"
```

3. Set your API key:

```bash
export OPENAI_API_KEY="your-api-key-here"
```

4. Run the benchmark:

```bash
epb run --config epb_config.yaml
```

5. Score the results:

```bash
epb score --run-dir runs/YYYYMMDD_HHMMSS
```

## What EPB Measures

EPB evaluates four distinct pathologies, each with an explicit metric:

### 1. Mirror Loop (EPB Phi)

Measures stability in recursive self-refinement. Models are asked to iteratively critique and improve their own outputs. **Collapse** occurs when the model gets stuck in repetitive patterns.

**Score**: 0-100 (higher is better)

### 2. Confabulation (EPB Persistence)

Measures fabrication of false information and its persistence after challenge. Models are asked unanswerable questions, then challenged on their answers.

**Score**: 0-100 (higher is better, less persistent confabulation)

### 3. Violation State (EPB Contamination)

Measures refusal contamination after seeing disallowed content. Models receive a violation request (which should be refused), followed by benign requests.

**Score**: 0-100 (higher is better, less contamination)

### 4. Echo Chamber (EPB Drift)

Measures semantic drift through iterative summarization. Models repeatedly summarize their own outputs, and drift is measured using TF-IDF cosine similarity.

**Score**: 0-100 (higher is better, less drift)

### Overall Score: EPB Truth

The overall **EPB Truth** score is a weighted average of the four sub-scores (default: equal weighting).

**Certification Levels**:
- **Platinum**: 95+
- **Gold**: 85+
- **Silver**: 70+
- **Bronze**: 50+

## EPB v1 Test Suite

EPB v1 includes:
- 20 Mirror Loop prompts
- 30 Confabulation questions
- 10 Violation State sequences
- 10 Echo Chamber scenarios

Total: 70 test tasks designed for quality over quantity.

## Documentation

- [Quickstart Guide](docs/quickstart.md)
- [Methodology](docs/methodology.md)
- [Scoring Details](docs/scoring.md)
- [API Reference](docs/api.md)

## Leaderboard

Submit your results to the public leaderboard:

```bash
export EPB_LEADERBOARD_URL="https://epb.coursecorrect.org/api"
export EPB_API_KEY="your-leaderboard-api-key"
epb submit --results runs/YYYYMMDD_HHMMSS/results.json
```

View the leaderboard at: https://epb.coursecorrect.org

## Architecture

EPB is designed to be:
- **Model-agnostic**: Works with any LLM through simple adapters
- **Reproducible**: Explicit metrics and deterministic scoring
- **Extensible**: Easy to add new batteries and adapters
- **Transparent**: Open-source specifications and scoring code

## Supported Models

Out of the box, EPB supports:
- OpenAI GPT models (GPT-4, GPT-3.5, etc.)
- Anthropic Claude models

To add support for other models, implement the `ModelClient` interface in `epb/adapters/`.

## Citation

If you use EPB in your research, please cite:

```bibtex
@software{epb2025,
  title = {EPB: Epistemic Pathology Benchmark},
  author = {Course Correct Labs},
  year = {2025},
  url = {https://github.com/Course-Correct-Labs/epb-benchmark}
}
```

## Contributing

We welcome contributions! Please see our contributing guidelines.

Areas for contribution:
- New model adapters
- Additional test tasks
- Improved scoring heuristics
- Bug fixes and documentation

## License

MIT License - see [LICENSE](LICENSE) for details.

## About

EPB is developed by [Course Correct Labs](https://coursecorrect.org), a research organization focused on epistemic integrity in AI systems.

Related work:
- [Mirror Loop](https://github.com/Course-Correct-Labs/mirror-loop)
- [Recursive Confabulation](https://github.com/Course-Correct-Labs/recursive-confabulation)
- [Violation State](https://github.com/Course-Correct-Labs/violation-state)
- [Echo Chamber Zero](https://github.com/Course-Correct-Labs/echo-chamber-zero)

## Support

- GitHub Issues: [Report bugs or request features](https://github.com/Course-Correct-Labs/epb-benchmark/issues)
- Documentation: [docs/](docs/)
- Contact: hello@coursecorrect.org
