Metadata-Version: 2.4
Name: CEval
Version: 1.1.1
Summary: A framework for evaluating counterfactual explanations.
Author-email: Betül Bayrak <betul.bayrak@ntnu.no>
License: MIT
Project-URL: Homepage, https://github.com/b-bayrak/CEval
Project-URL: Documentation, https://github.com/b-bayrak/CEval#readme
Project-URL: Paper, https://ieeexplore.ieee.org/document/10550910
Project-URL: Bug Tracker, https://github.com/b-bayrak/CEval/issues
Keywords: explainable AI,XAI,counterfactual explanations,machine learning,evaluation,interpretability
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Intended Audience :: Science/Research
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: scikit-learn>=1.1
Requires-Dist: scipy>=1.9
Requires-Dist: gower>=0.1.2
Requires-Dist: category-encoders>=2.6
Dynamic: license-file

# CEval — Counterfactual Explanation Evaluator

[![PyPI version](https://img.shields.io/pypi/v/CEval.svg)](https://pypi.org/project/CEval/)
[![Python](https://img.shields.io/pypi/pyversions/CEval.svg)](https://pypi.org/project/CEval/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Paper](https://img.shields.io/badge/IEEE-Access%202024-blue)](https://ieeexplore.ieee.org/document/10550910)

**CEval** is a lightweight Python package for evaluating the quality of counterfactual explanations produced by post-hoc XAI (Explainable AI) methods.  It computes **14 established metrics** with a single call and works with diverse architectures.

> **Paper:** Bayrak, B., & Bach, K. (2024). *Evaluation of Instance-Based Explanations: An In-Depth Analysis of Counterfactual Evaluation Metrics, Challenges, and the CEval Toolkit.* IEEE Access. [doi:10.1109/ACCESS.2024.3410540](https://ieeexplore.ieee.org/document/10550910)

---

## Why CEval?

When you build or compare counterfactual explainers, you need more than one number to judge quality.  CEval lets you measure all key dimensions (validity, proximity, sparsity, diversity, feasibility, and more) in a single unified framework, across different explainers and datasets.

```python
from ceval import CEval

evaluator = CEval(samples=test_df, label="income", data=train_df, model=clf)
evaluator.add_explainer("DiCE",  dice_cfs,  "generated-cf")
evaluator.add_explainer("DICE+", dicep_cfs, "generated-cf")

print(evaluator.comparison_table)
```

---

## Installation

```bash
pip install CEval
```

**Requirements:** Python ≥ 3.9, pandas, numpy, scikit-learn, scipy, gower, category-encoders

---

## Metrics

| Metric | Description | Needs model | Needs data |
|---|---|:---:|:---:|
| `validity` | Fraction of CFs that actually flip the classifier's prediction | ✓ | ✓ |
| `proximity` | Average feature-space distance between instance and its CF | | |
| `proximity_gower` | Proximity using the Gower mixed-type distance | | ✓ |
| `sparsity` | Average fraction of features changed | | |
| `count` | Average number of CFs per instance | | |
| `diversity` | Determinant-based spread of the CF set | | |
| `diversity_lcc` | Diversity weighted by label-class coverage | | |
| `yNN` | Label consistency of the CF's k nearest neighbours | ✓ | ✓ |
| `feasibility` | Average kNN distance of CFs to the training set | | ✓ |
| `kNLN_dist` | Distance of CF to nearest same-class neighbour | | ✓ |
| `relative_dist` | dist(x, CF) / dist(x, NUN) | | ✓ |
| `redundancy` | Average number of unnecessary feature changes | ✓ | ✓ |
| `plausibility` | dist(CF, NLN) / dist(NLN, NUN(NLN)) | ✓ | ✓ |
| `constraint_violation` | Fraction of CFs that break user constraints | | |

Not every metric applies to every explanation type, CEval handles this automatically and fills non-applicable cells with `"-"`.

---

## Quick Start

```python
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from ceval import CEval

# 1. Prepare your data
train_df = ...   # pd.DataFrame with features + label column
test_df  = ...   # pd.DataFrame with features + label column
clf      = RandomForestClassifier().fit(train_df.drop("label", axis=1),
                                        train_df["label"])

# 2. Generate counterfactuals with your favourite explainer
#    (DiCE, PertCF, DICE, NICE, etc.)
counterfactuals = ...   # pd.DataFrame, same columns as test_df

# 3. Evaluate
evaluator = CEval(
    samples    = test_df,        # instances to explain
    label      = "label",        # target column name
    data       = train_df,       # background dataset (unlocks more metrics)
    model      = clf,            # fitted classifier (unlocks more metrics)
    k_nn       = 5,              # neighbours for kNN-based metrics
    constraints= ["age"],        # features that must not change (optional)
)

evaluator.add_explainer(
    name            = "MyExplainer",
    explanations    = counterfactuals,
    exp_type        = "generated-cf",   # "generated-cf" | "existed-cf" |
                                        # "generated-factual" | "existed-factual"
    mode            = "1to1",           # "1to1" | "1toN"
)

print(evaluator.comparison_table)
```

### Explanation types

| `exp_type` | When to use |
|---|---|
| `"generated-cf"` | Counterfactuals *synthesised* by an algorithm (e.g. DiCE, PertCF) |
| `"existed-cf"` | Counterfactuals retrieved from the *training set* |
| `"generated-factual"` | Factual explanations generated by an algorithm |
| `"existed-factual"` | Factual explanations retrieved from the training set |

### Explanation modes

| `mode` | DataFrame shape | When to use |
|---|---|---|
| `"1to1"` | Same number of rows as `samples` | One explanation per instance |
| `"1toN"` | Any number of rows + an `"instance"` column | Multiple explanations per instance |

---

## Model Compatibility

CEval works with **any** classifier, not just scikit-learn.  
Use the built-in wrappers from `ceval.wrappers` to adapt your model:

| Framework | Wrapper class | Import |
|---|---|---|
| scikit-learn | *(none needed)* | pass model directly |
| XGBoost | `XGBoostWrapper` | `from ceval.wrappers import XGBoostWrapper` |
| LightGBM | `LightGBMWrapper` | `from ceval.wrappers import LightGBMWrapper` |
| CatBoost | `CatBoostWrapper` | `from ceval.wrappers import CatBoostWrapper` |
| PyTorch | `TorchWrapper` | `from ceval.wrappers import TorchWrapper` |
| Keras / TensorFlow | `KerasWrapper` | `from ceval.wrappers import KerasWrapper` |
| Anything else | `GenericWrapper` | `from ceval.wrappers import GenericWrapper` |

```python
# PyTorch
from ceval.wrappers import TorchWrapper
model = TorchWrapper(my_net, num_classes=2, device="cuda")

# XGBoost  (works with XGBClassifier and native Booster)
from ceval.wrappers import XGBoostWrapper
model = XGBoostWrapper(xgb_clf)

# Keras / TensorFlow
from ceval.wrappers import KerasWrapper
model = KerasWrapper(keras_model, num_classes=3)

# Anything else — supply two callables
from ceval.wrappers import GenericWrapper
model = GenericWrapper(
    predict_fn       = lambda X: my_model.infer(X).argmax(axis=1),
    predict_proba_fn = lambda X: my_model.infer(X),
)

# Then use as normal
evaluator = CEval(samples=test_df, label="income", data=train_df, model=model)
```

If you pass an incompatible model without a wrapper, CEval raises a clear `TypeError` that tells you exactly which wrapper to use.

---

See [`examples/demo_adult_income.py`](examples/demo_adult_income.py) for a complete working demo that:

- Loads the **Adult Income** dataset
- Trains a **Random Forest** classifier
- Generates counterfactuals with **DiCE**
- Evaluates them in both **1-to-1** and **1-to-N** mode
- Prints a full comparison table

```bash
python examples/demo_adult_income.py
```

Expected output:

```
                     DiCE (1-to-1)  DiCE (1-to-N)
validity                      0.90          0.867
proximity_gower               0.11          0.152
sparsity                      0.32          0.347
yNN                           0.68          0.713
feasibility                  48.21         183.44
redundancy                    0.80          0.733
constraint_violation          0.50          0.233
...
```

---

## Comparing Multiple Explainers

```python
evaluator = CEval(samples=test_df, label="label", data=train_df, model=clf)

evaluator.add_explainer("DiCE",   dice_cfs,   "generated-cf", mode="1toN")
evaluator.add_explainer("PertCF", pertcf_cfs, "generated-cf", mode="1toN")
evaluator.add_explainer("NICE",   nice_cfs,   "existed-cf",   mode="1to1")

# Side-by-side comparison
print(evaluator.comparison_table.T)
```

---

## API Reference

### `CEval:`
`CEval(samples, label, ...)`

| Parameter | Type | Default | Description |
|---|---|---|---|
| `samples` | `pd.DataFrame` | *required* | Instances to be explained (includes label column) |
| `label` | `str` | *required* | Name of the target column |
| `data` | `pd.DataFrame` | `None` | Full background dataset; unlocks distribution-based metrics |
| `model` | sklearn estimator | `None` | Fitted classifier; unlocks prediction-based metrics |
| `k_nn` | `int` | `5` | Neighbours for kNN metrics |
| `encoder` | `str` | `None` | Category-encoder name for categoricals (default: OrdinalEncoder) |
| `distance` | `str` | `None` | scipy distance metric for proximity; `None` uses built-in mixed metric |
| `constraints` | `list[str]` | `None` | Feature names that must not change in valid CFs |

### `evaluator.add_explainer:`

`evaluator.add_explainer(name, explanations, exp_type, mode="1to1")`

Registers an explainer and computes all applicable metrics.  Results are appended to `evaluator.comparison_table`.

### `evaluator.comparison_table:`

A `pd.DataFrame` with one row per explainer and one column per metric.  Non-applicable metrics show `"-"`.

---

## Citation

If you use CEval in your research, please cite:

```bibtex
@article{bayrak2024ceval,
  title   = {Evaluation of Instance-Based Explanations: An In-Depth Analysis of Counterfactual Evaluation Metrics, Challenges, and the CEval Toolkit},
  author  = {Bayrak, Bet{\"u}l and Bach, Kerstin},
  journal = {IEEE Access},
  year    = {2024},
  doi     = {10.1109/ACCESS.2024.3410540}
}
```

---

## Related Work

This package is part of a broader research effort on counterfactual explanation methods:

- **PerCE** — Personalised Counterfactual Explanations ([IEEE](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=11271748))
- **PertCF** — Perturbation-based Counterfactual Explainer ([Paper](https://folk.idi.ntnu.no/kerstinb/paper/2023-SGAI-PertCF-BayrakBach.pdf) | [Code](https://github.com/b-bayrak/PertCF-Explainer))

---

## License

MIT © [Betül Bayrak](mailto:betul.bayrak@ntnu.no)
