Metadata-Version: 2.4
Name: qapitol-evals-kit
Version: 0.1.0
Summary: Local BYOK LLM and RAG evaluation metrics (Qapitol Evals Kit)
Project-URL: Homepage, https://github.com/QapitolAI/qapitol-evals-kit
Project-URL: Documentation, https://github.com/QapitolAI/qapitol-evals-kit#readme
Project-URL: Repository, https://github.com/QapitolAI/qapitol-evals-kit
Project-URL: Issues, https://github.com/QapitolAI/qapitol-evals-kit/issues
Author: Qapitol
License: MIT
License-File: LICENSE
Keywords: evaluation,llm,metrics,rag
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: pydantic>=2.0
Requires-Dist: typing-extensions>=4.5
Provides-Extra: all
Requires-Dist: anthropic>=0.18; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.18; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Description-Content-Type: text/markdown

# qapitol-evals-kit

Local **BYOK** LLM and RAG evaluation metrics from [Qapitol](https://github.com/QapitolAI). Your data never leaves your machine.

## Install

**PyPI** (after [publish setup](docs/RELEASING.md)):

```bash
pip install qapitol-evals-kit==0.1.0
pip install "qapitol-evals-kit[all]==0.1.0"   # OpenAI + Anthropic judges
```

**From source** (development):

```bash
pip install -e ".[dev,all]"
```

**From GitHub** (fallback):

```bash
pip install "qapitol-evals-kit @ git+https://github.com/QapitolAI/qapitol-evals-kit@v0.1.0"
pip install "qapitol-evals-kit @ git+https://github.com/QapitolAI/qapitol-evals-kit.git@main"
```

Repo: [github.com/QapitolAI/qapitol-evals-kit](https://github.com/QapitolAI/qapitol-evals-kit) · Releases: [v0.1.0](https://github.com/QapitolAI/qapitol-evals-kit/releases/tag/v0.1.0) · PyPI: [qapitol-evals-kit](https://pypi.org/project/qapitol-evals-kit/)

## Quick start (no API key)

```python
from qapitol.evals import ExactMatchEvaluator

score = ExactMatchEvaluator().evaluate({
    "output": "Paris",
    "expected": "Paris",
})
print(score.score, score.label)  # 1.0 match
```

## Quick start (LLM judge)

```bash
export OPENAI_API_KEY=sk-...
```

```python
from qapitol.evals import CoherenceEvaluator
from qapitol.evals.llm import LLM

llm = LLM(provider="openai", model="gpt-4o-mini")
score = CoherenceEvaluator(llm).evaluate({
    "input": "What is RAG?",
    "output": "RAG retrieves context then generates an answer.",
})
print(score.score, score.label, score.explanation)
```

## CLI

```bash
qapitol-evals doctor
qapitol-evals run --metric coherence --input "What is AI?" --output "AI is ..."
```

## Metrics (v0.1.0)

| Type | Evaluators |
|------|------------|
| Code | `ExactMatchEvaluator`, `CustomAccuracyEvaluator` |
| LLM | Coherence, Relevance, Correctness, Hallucination, Toxicity |
| RAG | Faithfulness, Answer Relevancy |

## Test

```bash
ruff check src tests
pytest tests/ -v
```

For agents working on this repo, see [`agent.md`](agent.md).

## License

MIT
