Metadata-Version: 2.4
Name: evalflow
Version: 0.1.7
Summary: pytest for LLMs - catch prompt regressions before they reach production
Project-URL: Homepage, https://github.com/emartai/evalflow
Project-URL: Documentation, https://emartai.mintlify.app/
Project-URL: Repository, https://github.com/emartai/evalflow
Project-URL: Issues, https://github.com/emartai/evalflow/issues
Project-URL: Changelog, https://github.com/emartai/evalflow/blob/main/CHANGELOG.md
License: MIT
Keywords: ai,ci,evaluation,llm,prompts,quality,testing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: anthropic>=0.28.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: openai>=1.30.0
Requires-Dist: pydantic>=2.6.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rich>=13.7.0
Requires-Dist: typer>=0.12.0
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: hatchling>=1.24.0; extra == 'dev'
Requires-Dist: httpx>=0.27.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=2.7.0; extra == 'embeddings'
Description-Content-Type: text/markdown

```text
> evalflow
```

pytest for LLMs

[![PyPI](https://img.shields.io/pypi/v/evalflow)](https://pypi.org/project/evalflow/)
[![Python](https://img.shields.io/pypi/pyversions/evalflow)](https://pypi.org/project/evalflow/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/emartai/evalflow/blob/main/LICENSE)
[![CI](https://github.com/emartai/evalflow/actions/workflows/ci.yml/badge.svg)](https://github.com/emartai/evalflow/actions/workflows/ci.yml)

```text
You changed one prompt.
Summarization improved.
Classification silently broke.
Nobody noticed for 4 days.

evalflow catches this in CI before it ships.
```

## Install

```bash
pip install evalflow
```

## Quick Start

```bash
evalflow init
evalflow eval
```

What you get on day one:

- local prompt and dataset files
- SQLite-backed run history in `.evalflow/`
- CI-friendly exit codes
- offline cache support for repeatable checks

## Terminal Screenshot

```text
> evalflow eval

Running 5 test cases against gpt-4o-mini...

✓ summarize_short_article    0.91
✓ classify_sentiment         1.00
✓ extract_entities           0.87
✗ answer_with_context        0.61
✓ rewrite_formal             0.93

Quality Gate: PASS
Failures: 1
Run ID: 20240315-a3f9c2d81b4e
```

## Why evalflow

Traditional unit tests do not tell you when a prompt tweak quietly degrades a task.
evalflow gives you a small local quality gate for prompt, model, and dataset changes.

Use it when you need to:

- catch regressions before merge
- compare runs locally
- keep prompt versions in YAML
- run the same gate in CI and on a laptop

## GitHub Actions Workflow

```yaml
# .github/workflows/evalflow.yml
name: LLM Quality Gate

on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"
      - "**.py"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install evalflow
      - run: evalflow eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

## Features

- pytest-style exit codes: `0=pass`, `1=fail`, `2=error`
- exact match, embedding, consistency, and LLM judge methods
- baseline snapshots catch regressions, not just low scores
- prompt registry keeps prompts versioned in YAML
- works with OpenAI, Anthropic, Groq, Gemini, and Ollama
- local SQLite storage, no account needed
- offline cache for repeated and CI-safe checks

## Command Surface

```bash
evalflow init
evalflow eval
evalflow doctor
evalflow runs
evalflow compare RUN_A RUN_B
evalflow prompt list
```

## Documentation

- Docs hub: [emartai.mintlify.app](https://emartai.mintlify.app/)
- Quickstart source: [docs/quickstart.mdx](https://github.com/emartai/evalflow/blob/main/docs/quickstart.mdx)
- CLI reference source: [docs/cli-reference.mdx](https://github.com/emartai/evalflow/blob/main/docs/cli-reference.mdx)
- CI guide source: [docs/ci-github-actions.mdx](https://github.com/emartai/evalflow/blob/main/docs/ci-github-actions.mdx)
- Provider docs: [docs/providers](https://github.com/emartai/evalflow/tree/main/docs/providers)

## Security

- evalflow reads API keys from environment variables, never config files
- `evalflow.yaml` stores env var names, not secret values
- keep `.env` and `.evalflow/` out of git
- see [docs/dev-doc/security.md](https://github.com/emartai/evalflow/blob/main/docs/dev-doc/security.md) for the full security model

## Reporting Security Issues

Please do not open public GitHub issues for security vulnerabilities.
Open a private [GitHub Security Advisory](https://github.com/emartai/evalflow/security/advisories/new).

## Examples

- [examples/openai-basic](https://github.com/emartai/evalflow/tree/main/examples/openai-basic)
- [examples/groq-ci](https://github.com/emartai/evalflow/tree/main/examples/groq-ci)
- [examples/langchain-app](https://github.com/emartai/evalflow/tree/main/examples/langchain-app)

## Development

See [CONTRIBUTING.md](https://github.com/emartai/evalflow/blob/main/CONTRIBUTING.md) for local setup, tests, smoke checks, and performance baselines.

## License

MIT
