Metadata-Version: 2.5
Name: cafe-core
Version: 0.0.1
Summary: CAFE — a design-of-experiments evaluation engine for compound AI systems
Project-URL: Homepage, https://cafe-ai.de
Project-URL: Documentation, https://fabian-lu.github.io/Cafe
Project-URL: Repository, https://github.com/fabian-lu/Cafe
Project-URL: Issues, https://github.com/fabian-lu/Cafe/issues
Author: Christoph Weisser, Thomas Kneib, Alexander Silbersdorff
Author-email: Fabian Lukassen <fabian.lukassen@uni-goettingen.de>
License: Apache-2.0
License-File: LICENSE
Keywords: compound-ai,design-of-experiments,evaluation,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Provides-Extra: all
Requires-Dist: datasets>=2.0; extra == 'all'
Requires-Dist: ipywidgets>=8.0; extra == 'all'
Requires-Dist: jupyterlab>=4.0; extra == 'all'
Requires-Dist: litellm>=1.50; extra == 'all'
Requires-Dist: matplotlib>=3.7; extra == 'all'
Requires-Dist: numpy>=1.24; extra == 'all'
Requires-Dist: pandas>=2.0; extra == 'all'
Requires-Dist: scipy>=1.10; extra == 'all'
Requires-Dist: statsmodels>=0.14; extra == 'all'
Requires-Dist: tqdm>=4.66; extra == 'all'
Provides-Extra: datasets
Requires-Dist: datasets>=2.0; extra == 'datasets'
Provides-Extra: dev
Requires-Dist: krippendorff>=0.6; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-jupyter>=0.24; extra == 'docs'
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.26; extra == 'docs'
Provides-Extra: llm
Requires-Dist: litellm>=1.50; extra == 'llm'
Requires-Dist: tqdm>=4.66; extra == 'llm'
Provides-Extra: notebooks
Requires-Dist: ipywidgets>=8.0; extra == 'notebooks'
Requires-Dist: jupyterlab>=4.0; extra == 'notebooks'
Requires-Dist: pandas>=2.0; extra == 'notebooks'
Requires-Dist: tqdm>=4.66; extra == 'notebooks'
Provides-Extra: stats
Requires-Dist: matplotlib>=3.7; extra == 'stats'
Requires-Dist: numpy>=1.24; extra == 'stats'
Requires-Dist: pandas>=2.0; extra == 'stats'
Requires-Dist: scipy>=1.10; extra == 'stats'
Requires-Dist: statsmodels>=0.14; extra == 'stats'
Description-Content-Type: text/markdown

# cafe-core

The CAFE evaluation engine: define a compound AI system as a black box, declare
the factors to vary, and run a design-of-experiments study over it — headless,
no web stack, no database required.

CAFE generates full and **fractional** factorial designs, runs each configuration
with replication (resumable checkpoints), scores answers with a configurable
**LLM judge** and/or human raters, and attributes quality to the factors with
**mixed-effects models** matched to the rubric's scale (linear · cumulative-link ·
logistic) plus **inter-rater reliability** (Krippendorff's α). See the `examples/`
notebooks for the full walkthrough.

## Install (dev)

```bash
uv venv && uv pip install -e "packages/cafe-core[dev]"
```

## Try it (no API keys)

```bash
cafe run example            # run the bundled toy 2-factor study
cafe run example --smoke    # preflight: 1 input, 1 rep, cost/time estimate
cafe validate example       # expand the design without running
```

## Library use

```python
import cafe

async def my_system(config, item):
    # your compound system: routing, RAG, cascade, agent — anything.
    return f"answer for {item!r} using {config['model']}"

study = cafe.Study(
    name="my-study",
    system=my_system,
    factors=[
        cafe.Factor("model", ["small", "large"]),
        cafe.Factor("prompt", ["plain", "cot"]),
    ],
    dataset=["question 1", "question 2"],
    replications=3,
)

results = study.run(checkpoint_path=".cafe/my-study.jsonl")  # resumable
print(results.summary())
for obs in results:
    print(obs.config, obs.output, obs.elapsed_s)
```

`study.run()` returns a `Results` object you hold like any value. The optional
`checkpoint_path` makes a long run crash-safe: re-running resumes instead of
restarting.
