Metadata-Version: 2.1
Name: medeval-framework
Version: 0.1.6
Summary: An open-source Python framework for rigorously benchmarking medical LLMs for accuracy, hallucination rates, and clinical safety (ECE).
Home-page: https://github.com/your-org/medeval-framework
Author: medeval contributors
License: Apache License 2.0
Keywords: medical llm evaluation benchmarking calibration ece hallucination sickle-cell
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Healthcare Industry
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.0
Provides-Extra: all
Requires-Dist: transformers>=4.35.0; extra == "all"
Requires-Dist: evaluate>=0.4.0; extra == "all"
Requires-Dist: datasets>=2.14.0; extra == "all"
Requires-Dist: torch>=2.0.0; extra == "all"
Requires-Dist: bert-score>=0.3.13; extra == "all"
Requires-Dist: peft>=0.5.0; extra == "all"
Requires-Dist: pytest>=7.4.0; extra == "all"
Requires-Dist: pytest-cov>=4.1.0; extra == "all"
Requires-Dist: ruff>=0.1.0; extra == "all"
Requires-Dist: mypy>=1.0.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Provides-Extra: nlp
Requires-Dist: transformers>=4.35.0; extra == "nlp"
Requires-Dist: evaluate>=0.4.0; extra == "nlp"
Requires-Dist: datasets>=2.14.0; extra == "nlp"
Requires-Dist: torch>=2.0.0; extra == "nlp"
Requires-Dist: bert-score>=0.3.13; extra == "nlp"
Requires-Dist: peft>=0.5.0; extra == "nlp"

# medeval-framework

[![PyPI version](https://img.shields.io/pypi/v/medeval-framework.svg)](https://pypi.org/project/medeval-framework/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python Support](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)](https://www.python.org/)
[![CI Pipeline](https://github.com/TeslaInch/medeval-framework/actions/workflows/ci.yml/badge.svg)](https://github.com/TeslaInch/medeval-framework/actions/workflows/ci.yml)

A rigorous, open-source Python evaluation framework designed to benchmark medical Large Language Models (LLMs) for clinical accuracy, hallucination rates, model calibration, and safety.

---

## 📖 Table of Contents

- [Key Features](#-key-features)
- [Installation](#-installation)
  - [Install via Pip (Recommended)](#1-install-via-pip-recommended)
  - [Install from Source](#2-install-from-source)
- [Quickstart](#-quickstart)
  - [Command Line Interface (CLI)](#1-command-line-interface-cli)
  - [Python Orchestration API](#2-python-orchestration-api)
- [Repository Structure](#-repository-structure)
- [Development & Testing](#-development--testing)
- [License](#-license)

---

## 🌟 Key Features

- **Multi-Dataset Benchmarks**: Out-of-the-box loaders for standardized medical datasets (MedQA, PubMedQA).
- **Dual Clinical Safety Audit**:
  - *Deterministic Checker*: Fast regex scanning for explicit clinical contraindications in **Sickle Cell Disease** and **Cardiology**.
  - *Semantic Safety Net*: NLI-based hazard verification (`SemanticSafetyChecker`) to flag context-dependent medical hazards.
- **Unified Model & PEFT Connectors**: Modular drivers to query API models (OpenAI) or execute local PyTorch/Transformers weights (Hugging Face) and PEFT/LoRA adapters smoothly.
- **NLP & Semantic Accuracy Engines**: Exact Match comparison and BERTScore semantic similarity scoring (`SemanticSimilarityScorer`).
- **NLI Hallucination Detection**: Cross-encoder Natural Language Inference (`NLIHallucinationDetector`) evaluating predictions (`hypothesis`) against authoritative clinical facts (`ground_truth`).
- **Advanced Calibration Suite**: Vectorized calculation of **Expected Calibration Error (ECE)**, **Maximum Calibration Error (MCE)**, and **Brier Score**.
- **CLI & Report Generator**: Command-line `medeval` interface and structured JSON report exporter for auditability.

---

## ⚙️ Installation

### 1. Install via Pip (Recommended)

`medeval-framework` is available on PyPI. You can install the core framework or include optional ML/NLP extras:

```bash
# Core installation (numpy-only, lightweight)
pip install medeval-framework

# Full ML & NLP stack (Transformers, PyTorch, evaluate, datasets, bert_score, peft)
pip install medeval-framework[nlp]

# Complete installation including development and testing tools
pip install medeval-framework[all]
```

### 2. Install from Source

For development or contributing:

```bash
# Clone the repository
git clone https://github.com/TeslaInch/medeval-framework.git
cd medeval-framework

# Install editable package with all extras
pip install -e ".[all]"
```

---

## 🚀 Quickstart

### 1. Command Line Interface (CLI)

Run evaluations directly from your terminal using the `medeval` command:

```bash
# Get full usage help and flag options
medeval --help

# Run evaluation on MedQA using OpenAI GPT-4o with Sickle Cell safety audit (default)
export OPENAI_API_KEY="your-api-key"
medeval --model gpt-4o --dataset medqa --limit 20 --output report.json

# Run evaluation on Hugging Face model or PEFT adapter with explicit safety selection
# Options for --safety: sickle_cell (default), none
medeval \
  --model "microsoft/Phi-3.5-mini-instruct" \
  --dataset medqa \
  --safety sickle_cell \
  --device "cuda:0" \
  --limit 50 \
  --output base_model_report.json
```

#### CLI Options Reference
| Flag | Description | Default |
| :--- | :--- | :--- |
| `--model` *(required)* | Model identifier (HuggingFace repo, PEFT adapter, `gpt-4o`, `openai:...`, or `mock-...`). | - |
| `--dataset` *(required)* | Benchmark dataset: `medqa` or `pubmedqa`. | - |
| `--output` *(required)* | File path where JSON report is saved. | - |
| `--safety` | Safety checker to run: `sickle_cell` (SCD contraindications) or `none`. | `sickle_cell` |
| `--device` | PyTorch device index (`cpu`, `cuda:0`, etc.). | `cpu` |
| `--limit` | Maximum number of samples to process. | All |
| `--no-hallucination` | Disable NLI hallucination check to increase speed. | Enabled |
| `--no-semantic-similarity` | Disable BERTScore semantic scoring to increase speed. | Enabled |
| `--trust-remote-code` | Enable `trust_remote_code=True` for Hugging Face models. | `False` |

---

### 2. Python Orchestration API

Create customized evaluation pipelines tailored to specific clinical domains. You can configure individual safety checkers (`SickleCellSafetyChecker`, `CardiologySafetyChecker`, `SemanticSafetyChecker`) or combine them into a composite `SafetySuite`:

```python
from medeval.benchmark import BenchmarkLoader
from medeval.models.huggingface import HuggingFaceConnector
from medeval.runner import BenchmarkRunner
from medeval.safety import (
    SickleCellSafetyChecker,
    CardiologySafetyChecker,
    SemanticSafetyChecker,
    SafetySuite,
)
from medeval.report import export_report_to_json

# 1. Load benchmark dataset
loader = BenchmarkLoader(split="test", max_samples=10)
samples = loader.load_medqa()

# 2. Instantiate Model Connector (Hugging Face base or PEFT adapter)
model = HuggingFaceConnector(model_name="microsoft/Phi-3.5-mini-instruct", device="cuda:0")

# 3. Configure Safety Checker
# Option A: Domain-specific checker (e.g. Sickle Cell Disease only)
safety_checker = SickleCellSafetyChecker()

# Option B: Multi-domain SafetySuite (Sickle Cell + Cardiology + NLI Semantic Net)
# safety_suite = SafetySuite([
#     SickleCellSafetyChecker(),
#     CardiologySafetyChecker(),
#     SemanticSafetyChecker(device=0)
# ])

# 4. Initialize and execute BenchmarkRunner
runner = BenchmarkRunner(model=model, safety_checker=safety_checker, ignore_errors=True)
report = runner.run(samples)

# 5. Export structured JSON report
export_report_to_json(report, "evaluation_report.json")
```

---

## 📁 Repository Structure

```
medeval/
├── medeval/
│   ├── models/               # Model Connectors (Base, HF, PEFT, OpenAI, Mock)
│   ├── safety/               # Safety Checkers (SickleCell, Cardiology, Semantic, SafetySuite)
│   ├── accuracy.py           # Scorers (Exact Match, BERTScore F1)
│   ├── benchmark.py          # Benchmark Loaders (MedQA, PubMedQA)
│   ├── calibration.py        # Calibration Suite (ECE, MCE, Brier Score)
│   ├── hallucination.py      # NLI Cross-Encoder Hallucination Engine
│   ├── report.py             # Metric aggregation & JSON serialization
│   ├── runner.py             # BenchmarkRunner pipeline orchestrator
│   └── structures.py         # Data contracts (MedicalEvalSample & EvaluationReport)
├── tests/                    # 169 Unit & Integration Tests
├── pyproject.toml            # Ruff & Mypy configurations
├── setup.py                  # PyPI Packaging configuration
└── requirements.txt          # Package dependencies
```

---

## 🧪 Development & Testing

Ensure style alignment and type safety before submitting pull requests:

```bash
# Run full pytest test suite
pytest

# Run Ruff style & linting check
ruff check .

# Run Mypy static type verification
mypy medeval/
```

---

## 📄 License

This project is licensed under the **Apache License 2.0**. See the [LICENSE](LICENSE) file for details.
