Metadata-Version: 2.4
Name: parity-ai
Version: 0.1.16
Summary: Automatically generate evals for every AI change
Author: Antoine Nguyen
License-Expression: MIT
Project-URL: Homepage, https://github.com/antoinenguyen27/parity
Project-URL: Repository, https://github.com/antoinenguyen27/parity
Project-URL: Issues, https://github.com/antoinenguyen27/parity/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: arize-phoenix-client>=2.0.0
Requires-Dist: braintrust>=0.9.0
Requires-Dist: claude-agent-sdk>=0.1.48
Requires-Dist: click>=8.1
Requires-Dist: httpx>=0.27
Requires-Dist: langsmith>=0.7
Requires-Dist: mcp>=1.0
Requires-Dist: numpy>=1.26
Requires-Dist: openai>=1.0
Requires-Dist: pydantic>=2.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: tiktoken>=0.7
Provides-Extra: dev
Requires-Dist: build==1.3.0; extra == "dev"
Requires-Dist: pytest==9.0.2; extra == "dev"
Requires-Dist: pytest-cov==7.0.0; extra == "dev"
Requires-Dist: respx==0.22.0; extra == "dev"
Dynamic: license-file

# Parity

[![PyPI](https://img.shields.io/pypi/v/parity-ai)](https://pypi.org/project/parity-ai/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/downloads/)

Parity analyzes behavior-defining AI changes in pull requests, discovers the most relevant existing eval target, validates the real coverage gaps, and proposes native eval additions that fit the target suite.

Parity is not an eval runner. It does not create or mutate hosted evaluator infrastructure. It reuses the eval system you already have.

## What Parity Does

For each PR that changes prompts, instructions, guardrails, judges, validators, or similar behavior-defining assets, Parity:

1. Detects the behavioral change.
2. Resolves the best matching eval target and method.
3. Validates which gaps are actually uncovered.
4. Synthesizes native eval additions for that target.
5. Writes only `native_ready` evals after explicit approval.

## Support

| Path | Status | Notes |
|---|---|---|
| Promptfoo | Strong | Best fully native path. Assertions are row-local and writeback is straightforward. |
| LangSmith | Strong | Strong dataset discovery and writeback. Evaluator reuse is supported; evaluator mutation is out of scope. |
| Braintrust | Supported with limitations | Writeback works. Target discovery is weaker and evaluator recovery depends more on repo assets. |
| Arize Phoenix | Supported with limitations | Dataset read/write works. Evaluator discovery is weaker than Promptfoo and LangSmith. |
| Bootstrap mode | Built in | If no safe target is found, Parity proposes starter evals and abstains from unsafe writeback. |

More detail: [docs/platforms.md](docs/platforms.md)

## Public Commands

These are the commands most users need:

- `parity init` — scaffold `parity.yaml`, the GitHub Actions workflow, and `context/` stubs
- `parity doctor` — verify your setup and environment
- `parity run-stage 1` — detect behavioral artifact changes in a PR
- `parity run-stage 2` — analyze coverage gaps against existing evals
- `parity run-stage 3` — synthesize native eval proposals
- `parity write-evals` — write approved evals to your platform after merge
- `parity setup-mcp` — generate an MCP server config from `parity.yaml` (for local agent tooling)

Internal CI commands (`post-comment`, `resolve-run-id`, etc.) are used by the generated workflow and are not intended to be called directly.

## Quick Start

```bash
pip install parity-ai
parity init
```

Then:

1. Fill in the generated `context/` files.
2. Add GitHub secrets: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and any platform keys you use.
3. Commit `parity.yaml`, `.github/workflows/parity.yml`, and `context/`.
4. Open a PR that changes AI behavior.
5. Add the fixed approval label `parity:approve` before merging if you want Parity to write approved evals back after merge.

## Docs

- [Configuration](docs/configuration.md)
- [Architecture](docs/spec.md)
- [Platform support](docs/platforms.md)
- [Example quickstart](examples/langgraph-agentic-rag/docs/quickstart.md)
- [Maintainer guide](docs/maintainers.md)

## License

[MIT](LICENSE)
