Metadata-Version: 2.4
Name: pycharter
Version: 0.0.65
Summary: Runtime data contract enforcement for Python: one Pydantic-native contract validates API objects, Kafka streams, and ETL batches in-process — with quality scoring, a versioned contract registry, and ODCS round-trip.
Author-email: Optophi <contact@optophi.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/optophi/pycharter
Project-URL: Documentation, https://optophi.github.io/pycharter/
Project-URL: Repository, https://github.com/optophi/pycharter
Project-URL: Issues, https://github.com/optophi/pycharter/issues
Project-URL: Contributing, https://github.com/optophi/pycharter/blob/main/CONTRIBUTING.md
Keywords: data-contract,data-contracts,runtime-validation,data-quality,data-governance,json-schema,pydantic,odcs,open-data-contract-standard,kafka,streaming,etl,validation,schema-registry,schema
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Database
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3.0,>=2.0.0
Requires-Dist: jsonschema>=4.26.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: sqlalchemy<3.0,>=2.0.0
Requires-Dist: alembic<2.0,>=1.13.0
Requires-Dist: jsonata-python>=0.6.0
Requires-Dist: httpx<1.0,>=0.24.0
Requires-Dist: networkx>=3.0
Provides-Extra: server
Requires-Dist: fastapi<0.140,>=0.136.3; extra == "server"
Requires-Dist: uvicorn[standard]<1.0,>=0.24.0; extra == "server"
Requires-Dist: starlette>=1.2.1; extra == "server"
Requires-Dist: pydantic-settings<3.0,>=2.14.2; extra == "server"
Requires-Dist: python-multipart>=0.0.6; extra == "server"
Requires-Dist: PyJWT<3.0,>=2.8.0; extra == "server"
Requires-Dist: httpx<1.0,>=0.24.0; extra == "server"
Requires-Dist: aiofiles>=25.1.0; extra == "server"
Requires-Dist: prometheus-client>=0.19.0; extra == "server"
Requires-Dist: psycopg[binary]>=3.3.4; extra == "server"
Provides-Extra: pipeline
Requires-Dist: sshtunnel>=0.4.0; extra == "pipeline"
Requires-Dist: boto3>=1.26.0; extra == "pipeline"
Requires-Dist: google-cloud-storage>=3.12.0; extra == "pipeline"
Requires-Dist: azure-storage-blob>=12.0.0; extra == "pipeline"
Requires-Dist: openpyxl>=3.1.5; extra == "pipeline"
Requires-Dist: lxml>=4.9.0; extra == "pipeline"
Requires-Dist: pyarrow>=14.0.0; extra == "pipeline"
Requires-Dist: pymongo>=4.0.0; extra == "pipeline"
Requires-Dist: openlineage-python>=1.0.0; extra == "pipeline"
Provides-Extra: streaming
Requires-Dist: websockets>=12.0; extra == "streaming"
Requires-Dist: watchfiles>=0.21.0; extra == "streaming"
Requires-Dist: prometheus_client>=0.19.0; extra == "streaming"
Requires-Dist: aiohttp>=3.9.0; extra == "streaming"
Requires-Dist: aiokafka>=0.9.0; extra == "streaming"
Requires-Dist: aio-pika>=9.0.0; extra == "streaming"
Requires-Dist: redis>=5.0.0; extra == "streaming"
Requires-Dist: python-socketio>=5.11.0; extra == "streaming"
Provides-Extra: semantic
Requires-Dist: linkml>=1.7.0; extra == "semantic"
Requires-Dist: linkml-runtime>=1.7.0; extra == "semantic"
Requires-Dist: rdflib>=7.0.0; extra == "semantic"
Requires-Dist: owlrl>=6.0.0; extra == "semantic"
Requires-Dist: neo4j<7.0.0,>=6.2.0; extra == "semantic"
Provides-Extra: extraction
Requires-Dist: tree-sitter<1.0.0,>=0.25.2; extra == "extraction"
Requires-Dist: tree-sitter-python<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-javascript<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-typescript<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-go<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-rust<1.0.0,>=0.24.2; extra == "extraction"
Requires-Dist: tree-sitter-java<1.0.0,>=0.23.5; extra == "extraction"
Requires-Dist: markdown-it-py<5.0.0,>=4.2.0; extra == "extraction"
Requires-Dist: pypdf<8.0.0,>=4.0.0; extra == "extraction"
Requires-Dist: sqlglot>=25.0; extra == "extraction"
Provides-Extra: ai
Requires-Dist: anthropic>=0.34.0; extra == "ai"
Requires-Dist: openai>=2.44.0; extra == "ai"
Requires-Dist: mcp<2,>=1.0; extra == "ai"
Provides-Extra: otel
Requires-Dist: opentelemetry-api<2,>=1.20; extra == "otel"
Provides-Extra: cli
Requires-Dist: rich<16.0,>=13.0.0; extra == "cli"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.5.3; extra == "docs"
Requires-Dist: mkdocs-material[imaging]>=9.7.6; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
Requires-Dist: pymdown-extensions>=10.0.0; extra == "docs"
Requires-Dist: mike>=2.0.0; extra == "docs"
Provides-Extra: airflow
Requires-Dist: apache-airflow<4,>=3.2.2; extra == "airflow"
Provides-Extra: dagster
Requires-Dist: dagster>=1.7; extra == "dagster"
Provides-Extra: prefect
Requires-Dist: prefect<4,>=3.0; extra == "prefect"
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.29; extra == "postgres"
Requires-Dist: psycopg[binary]>=3.3.4; extra == "postgres"
Provides-Extra: all
Requires-Dist: pycharter[ai,cli,docs,extraction,otel,pipeline,postgres,semantic,server,streaming]; extra == "all"
Provides-Extra: ci
Requires-Dist: pycharter[extraction,pipeline,postgres,semantic,server,streaming]; extra == "ci"
Requires-Dist: testcontainers[postgres]>=4.0.0; extra == "ci"
Requires-Dist: testcontainers[kafka]>=4.0.0; extra == "ci"
Requires-Dist: pytest>=7.0.0; extra == "ci"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "ci"
Requires-Dist: pytest-cov>=4.0.0; extra == "ci"
Requires-Dist: pytest-timeout>=2.1.0; extra == "ci"
Requires-Dist: pytest-xdist>=3.5.0; extra == "ci"
Requires-Dist: pytest-randomly>=3.15.0; extra == "ci"
Requires-Dist: ruff>=0.15.20; extra == "ci"
Requires-Dist: mypy>=1.0.0; extra == "ci"
Requires-Dist: types-PyYAML>=6.0.0; extra == "ci"
Requires-Dist: requests>=2.32.3; extra == "ci"
Requires-Dist: pre-commit>=3.0.0; extra == "ci"
Requires-Dist: mkdocs<2,>=1.5.3; extra == "ci"
Requires-Dist: mkdocs-material[imaging]>=9.7.6; extra == "ci"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "ci"
Requires-Dist: pymdown-extensions>=10.0.0; extra == "ci"
Provides-Extra: dev
Requires-Dist: pycharter[all]; extra == "dev"
Requires-Dist: requests>=2.32.3; extra == "dev"
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.1.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.5.0; extra == "dev"
Requires-Dist: pytest-benchmark>=4.0.0; extra == "dev"
Requires-Dist: testcontainers[kafka]>=4.0.0; extra == "dev"
Requires-Dist: testcontainers[postgres]>=4.0.0; extra == "dev"
Requires-Dist: opentelemetry-sdk<2,>=1.20; extra == "dev"
Requires-Dist: ruff>=0.15.20; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0.0; extra == "dev"
Requires-Dist: pre-commit>=3.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: marimo>=0.9.0; extra == "dev"
Requires-Dist: vulture>=2.11; extra == "dev"
Requires-Dist: deptry>=0.20; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Dynamic: license-file

# PyCharter

> **PyCharter enforces data contracts where your code runs.**

Define a contract once — from a **Pydantic model** (`pycharter contract
from-model`) or versioned **YAML** — and the same artifact validates API
payloads in-process, gates Kafka consumers record by record, and checks ETL
batches at stage boundaries. Deterministic, Pydantic-native, in your process:
no warehouse round-trip, no separate CI engine deciding after the fact.
**The contract is what actually runs.**

Most contract tooling stops earlier in the lifecycle: datacontract-cli — the
ODCS reference implementation — verifies contracts in CI and batch; Soda and
Great Expectations scan the warehouse; dbt tests run after materialisation.
PyCharter is the **runtime half** of that ecosystem: it executes contracts
inside the producing service, and round-trips **ODCS** (v3.1.0 export, v3.0.x /
v3.1.x import) so the contracts you enforce at runtime slot into the tools you
already use.

**Full documentation:** [https://optophi.github.io/pycharter/](https://optophi.github.io/pycharter/)

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

**[Where PyCharter fits vs datacontract-cli, Pydantic, Pandera, GX, Soda, dbt →](docs/getting-started/comparisons.md#where-pycharter-fits)**

---

## Validate a record in 30 seconds

```bash
pip install pycharter
```

```python
from pycharter import Validator

validator = Validator.from_dict(
    {
        "type": "object",
        "version": "1.0.0",
        "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"},
        },
        "required": ["name", "age"],
    }
)

result = validator.validate({"name": "Alice", "age": 30})
print(result.is_valid)  # True

result = validator.validate({"name": "Bob", "age": "N/A"})
print(result.errors[0])  # Column 'age': expected integer, found string 'N/A'
```

That's the library at its smallest — no database, no setup. In production
you'd load the same contract from a file (`Validator.from_file("contract.yaml")`)
or a shared store. Failures come back as structured diagnostics (row, column,
expected type, offending value), and the generated model validates as fast as
a hand-written Pydantic model (~0.7 µs/record —
[measured](docs/operations/benchmarks.md)).

Prefer the shell? The same check works on a whole file:

```bash
pycharter validate contract.yaml data.jsonl   # colorized report, exit 0/1
```

---

## Why PyCharter

Teams often maintain a Pydantic model at the API, a Pandera schema in the
pipeline, a Great Expectations suite for quality, and an ODCS YAML as the
"contract" — and they drift. PyCharter unifies authoring and enforcement so
the contract is what actually runs, **where** it runs being the point:

| Tool | Where enforcement runs | Object | Dataset | Quality | Contract | Versioned |
|------|------------------------|:------:|:-------:|:-------:|:--------:|:---------:|
| Pydantic | In-process (single objects) | ● | ○ | ○ | ◐ | ◐ |
| Pandera | In-process (DataFrames) | ◐ | ● | ◐ | ◐ | ◐ |
| Great Expectations | Warehouse / batch jobs | ○ | ● | ● | ◐ | ◐ |
| Soda | Warehouse-side, scheduled scans | ○ | ● | ● | ◐ | ◐ |
| dbt tests | Warehouse, post-materialisation | ○ | ● | ◐ | ◐ | ● |
| datacontract-cli (ODCS ref. impl.) | CI / batch | ○ | ◐ | ◐ | ● | ● |
| **PyCharter** | **In-process runtime: APIs, streams, ETL** | **●** | **●** | **●** | **●** | **●** |

<sub>● full · ◐ partial · ○ none — markers reflect scope, not quality; see the
[full comparison](docs/getting-started/comparisons.md#where-pycharter-fits)</sub>

- **One definition, every surface** — model or YAML → API gate, stream gate,
  ETL steps, quality jobs, ODCS round-trip.
- **Enforces, not just authors** — rejects or coerces bad records *before*
  they land; warehouse-side engines measure what *did* land.
- **Deterministic and auditable** — versioned contracts, reproducible runs,
  thread-safe shared validators.

Choose Pydantic alone for one-service API shape checks, Pandera for
DataFrame validation in one process, GX/Soda for warehouse-centric batch
quality, datacontract-cli for ODCS authoring and CI gates — and PyCharter
when the contract must be the enforced single source of truth across
services. Full narrative and per-tool sections:
**[PyCharter vs. the alternatives](docs/getting-started/comparisons.md)**.

---

## Gate a FastAPI endpoint

```bash
pip install pycharter[server]
```

```python
from fastapi import Depends, FastAPI
from pycharter.contrib.fastapi import contract_dependency

app = FastAPI()
validate_order = contract_dependency("contracts/orders.yaml")  # built once


@app.post("/orders")
async def create_order(order: dict = Depends(validate_order)) -> dict:
    # `order` is the coerced + validated payload.
    return {"status": "accepted", "order_id": order["order_id"]}
```

Contract violations return a structured 422; the same YAML also drives your
pipelines and workers. Recipe:
[Gate a FastAPI endpoint](docs/cookbook/gate-fastapi-endpoint.md).

## Validate a Kafka topic, per record

```python
import asyncio
from pycharter import kafka


async def main():
    async for record in kafka.consume(
        topic="orders",
        bootstrap_servers="localhost:9092",
        group_id="my-service",
        contract_dir="contracts/orders",  # or contract_store=...
    ):
        process(record.payload)  # only records that passed the contract


asyncio.run(main())
```

Invalid records route to a DLQ with a structured envelope; `detect` /
`shadow` / `enforce` modes support safe rollouts, and `pycharter stream run`
runs it as a worker with Prometheus metrics. Start at
[Streaming — start here](docs/guides/streaming-overview.md).

## Gate contract evolution in CI

```bash
pycharter contract diff old.yaml new.yaml --fail-on-breaking
```

Classifies every field/type/required change as breaking or safe
(backward-compat semantics) — drop into pre-commit or CI so a contract
change can't ship without review.

---

## Concepts

| Concept | What it is | When you use it |
|--------|------------|------------------|
| **Schema** | The shape of the data (JSON Schema): types, required fields, nested objects. | When you only need structure. |
| **Data contract** | Schema + **coercion rules** (`"30"` → `30`) + **validation rules** (min/max, allowed values) + optional **metadata** (ownership, governance). | One artifact for structure, transforms, and business rules. |
| **Contract store** | A database (SQLite, PostgreSQL, …) holding versioned contracts so many apps reuse them. | Multiple services needing one source of truth. |
| **Enforcement point** | Where the contract executes: API dependency, stream consumer, ETL step flags (`coerce` / `validate` / `quality`). | Wherever bad data must be stopped. |

```
  Option A: no database   schema/contract in code or YAML → Validator → validate(data)
  Option B: with store    contract in DB → Validator(store=...)       → validate(data)
  Option C: pipelines     extract → [contract] → transform → [contract] → load
```

Start with Option A; add the store when contracts need versioned sharing; add
pipeline/stream enforcement where data moves. Deep dives:
[Concepts](docs/getting-started/concepts.md) ·
[Contract → ETL → quality journey](docs/getting-started/contract-etl-quality-journey.md) ·
[Cookbook](docs/cookbook/index.md).

---

## Installation

```bash
pip install pycharter
```

That's all you need for in-process validation. Optional extras add
capabilities — install only what you use:

| Extra | Adds | Install |
|-------|------|---------|
| `[fastapi]` | `contract_dependency` request gate | `pip install pycharter[server]` |
| `[streaming]` / `[kafka]` | Streaming validation engine / Kafka source | `pip install pycharter[streaming]` |
| `[api]` | REST API server (FastAPI + Uvicorn) | `pip install pycharter[server]` |
| `[ui]` | Pre-built Web UI (no Node.js required) | `pip install pycharter[server]` |
| `[postgres]` | PostgreSQL drivers for the ETL loader, DLQ, and contract store (asyncpg + psycopg) | `pip install pycharter[postgres]` |
| `[mcp]` | MCP server for AI agents | `pip install pycharter[ai]` |
| `[otel]` | OpenTelemetry tracing instruments | `pip install pycharter[otel]` |
| `[observability]` | Prometheus exporter + webhook sinks | `pip install pycharter[server]` |
| `[airflow]` / `[dagster]` | Orchestrator operators | `pip install pycharter[airflow]` |
| `[all]` | The common stack: api, ui, worker, pipeline, postgres, streaming, messaging, extraction, lineage, semantic-export, collab, cli, docs (not the optional `mcp` / `otel` / `observability` / `airflow` / `dagster` integrations — install those explicitly) | `pip install pycharter[all]` |

### Database setup (only for the contract store, API, or UI)

```bash
pycharter db init     # creates schema (default: sqlite:///pycharter.db)
pycharter db seed     # loads reference data
```

Skip this entirely if you only validate dicts or YAML contracts directly.
PostgreSQL/SQLite have managed Alembic migrations; see
[Store backend tiers](docs/reference/store-backend-tiers.md) and the
[Configuration Guide](docs/guides/configuration.md).

---

## ODCS and the contract ecosystem

PyCharter round-trips the
[Open Data Contract Standard](https://bitol-io.github.io/open-data-contract-standard/):
`to_odcs()` emits v3.1.0 documents that validate against the published
schema; `from_odcs()` imports v3.0.x / v3.1.x documents authored anywhere —
so contracts written with datacontract-cli enforce at runtime here, and
contracts authored here flow back into the ODCS toolchain. PyCharter is not
a rival standard; it is an **execution engine** for the standard.

```python
from pycharter.contract_io import from_odcs, to_odcs
from pycharter import Validator

contract = from_odcs(odcs_document)  # any v3.0.x / v3.1.x doc
validator = Validator(contract)  # enforce it immediately
```

How PyCharter composes with datacontract-cli, Soda, GX, and dbt:
[ODCS integration and interop](docs/guides/odcs-and-interop.md).

## Agents and MCP

PyCharter ships an MCP server (`python -m pycharter.mcp`, stdio JSON-RPC)
that serves **governed context packs** — concepts, relationships, and the
contract fields bound to them — to Claude Code, Claude Desktop, or any MCP
client, with role/workspace policy filtering. The context an agent receives
is bound to the same contracts PyCharter enforces at runtime, so what the
agent is told about your data is what your validators actually check.

```bash
pip install pycharter[ai]
export PYCHARTER_DATABASE_URL=postgresql://localhost/pycharter
python -m pycharter.mcp
```

Tools, client config, and the governance model:
[Agent integration (MCP)](docs/guides/agent-integration-mcp.md).

---

## Architecture at a glance

| Service | Input | Output | Journey stage |
|---------|-------|--------|---------------|
| **Contract Parser** | Contract files (YAML/JSON) | `ContractMetadata` | Specification → Parsing |
| **Contract Builder** | Separate artifacts or store | Consolidated contract | Storage → Consolidation |
| **Contract Store** | `ContractMetadata` | Stored metadata (DB) | Parsing → Storage |
| **Pydantic Generator** | JSON Schema | Pydantic models | Storage → Model generation |
| **JSON Schema Converter** | Pydantic models | JSON Schema | (Bidirectional) |
| **Runtime Validator** | Pydantic models + data | `ValidationResult` | Model generation → Validation |
| **ETL Pipelines** | Config files or code | `PipelineResult` | Extract → Transform → Load |
| **Quality Assurance** | Contract + data | `QualityReport` | Validation → Quality monitoring |

Each service is independent yet composable. The full tour — including the
REST API and Web UI wrappers — lives in the
[Core services reference](docs/reference/core-services.md); the layered
design philosophy is in [ARCHITECTURE.md](ARCHITECTURE.md).

## Picking the right API

The canonical path is the **`Validator` class** — create one from a contract
file, dict, directory, or store, and call `validate(record)` /
`validate_batch(records)`. It compiles the Pydantic model once, is
[thread-safe to share](docs/operations/benchmarks.md), and is what
production code should use.

| Use case | Approach | Example |
|----------|----------|---------|
| Production service, many validations | `Validator` class | `Validator.from_file("c.yaml").validate(data)` |
| Pandas DataFrame, split good/bad rows | `validate_dataframe()` | `validate_dataframe(df, "c.yaml", on_error="quarantine")` |
| Shell / CI gate on a data file | `pycharter validate` CLI | `pycharter validate c.yaml data.jsonl` |
| Batch processing | `Validator.validate_batch()` | `validator.validate_batch(rows)` |
| You already have a Pydantic model | Low-level function (deprecated at top level) | `from pycharter.runtime_validator import validate` |

ETL flows use **`Pipeline`**; quality checks use **`QualityCheck`**; contract
storage is pluggable via **`ContractStoreClient`** implementations. Built-in
coercions/validations and custom registration:
[reference](docs/reference/builtin-coercions-and-validations.md).

---

## Documentation

- **Docs site:** [https://optophi.github.io/pycharter/](https://optophi.github.io/pycharter/) — or `pycharter docs serve` locally (`pip install pycharter[docs]`)
- [Start here](docs/getting-started/start-here.md) · [Quick Start tour](docs/getting-started/quickstart.md) · [Cookbook](docs/cookbook/index.md)
- [End-to-end user guide](docs/guides/user-guide-end-to-end.md) · [Admin / governance guide](docs/guides/admin-guide-governance.md)
- [Streaming — start here](docs/guides/streaming-overview.md) · [Configuration](docs/guides/configuration.md) · [Validator benchmarks](docs/operations/benchmarks.md)

## Development and testing

```bash
python -m venv venv && source venv/bin/activate
pip install -e ".[dev]" && pre-commit install
pytest tests/unit            # fast suite; `pytest -m integration` needs Docker
./scripts/ci.sh              # full GHA-parity gate before opening a PR
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow and
[AGENTS.md](AGENTS.md) for AI-assisted contributions.

## Contributing

Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Report
security issues per [SECURITY.md](SECURITY.md); community expectations are in
[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

## License

MIT — see [LICENSE](LICENSE).

## Links

- **Repository:** [github.com/optophi/pycharter](https://github.com/optophi/pycharter)
- **Issues:** [GitHub Issues](https://github.com/optophi/pycharter/issues)
- **PyPI:** [pypi.org/project/pycharter](https://pypi.org/project/pycharter/)
