Metadata-Version: 2.4
Name: providercontract
Version: 0.1.0
Summary: Cross-provider schema contract testing for LLMs. Define once, validate everywhere — OpenAI, Anthropic, Mistral, LiteLLM and any JSON-returning model.
Project-URL: Homepage, https://github.com/maheshmakvana/llmcontract
Project-URL: Documentation, https://github.com/maheshmakvana/llmcontract#readme
Project-URL: Repository, https://github.com/maheshmakvana/llmcontract
Project-URL: Bug Tracker, https://github.com/maheshmakvana/llmcontract/issues
Project-URL: Changelog, https://github.com/maheshmakvana/llmcontract/blob/main/CHANGELOG.md
Author: providercontract contributors
License: MIT
License-File: LICENSE
Keywords: ai quality assurance,ai testing,anthropic,cohere,contract testing,cross-provider,generative ai,json schema,litellm,llm,llm contract,llm quality,llm schema,llm testing,llm validation,mistral,openai,output validation,provider agnostic,pytest,response validation,schema testing,schema validation,structured output
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: all
Requires-Dist: anthropic>=0.20.0; extra == 'all'
Requires-Dist: openai>=1.0.0; extra == 'all'
Requires-Dist: pydantic>=2.0.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.20.0; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == 'openai'
Provides-Extra: pydantic
Requires-Dist: pydantic>=2.0.0; extra == 'pydantic'
Description-Content-Type: text/markdown

# llmcontract

**Cross-provider schema contract testing for LLMs.**

Define the expected shape of an LLM response once. Validate it against OpenAI,
Anthropic, Mistral, LiteLLM, or any model that returns JSON — with a single,
consistent API.

```python
from llmcontract import Contract, field, required_fields

contract = Contract(
    required_fields("action", "confidence", "reasoning"),
    field("action",     type=str,   values=["buy", "sell", "hold"]),
    field("confidence", type=float),
    field("reasoning",  type=str,   min_len=20),
)

# Works with any provider
from llmcontract import openai_response
data = openai_response(client.chat.completions.create(...))
contract.validate(data)   # raises ContractViolation on mismatch
```

---

## Why llmcontract?

| Pain | llmcontract fix |
|---|---|
| Provider A returns `{"action": "BUY"}`, provider B returns `{"signal": "buy"}` | One contract definition catches both regressions |
| LLM adds unexpected keys that break downstream code | `no_extra_fields()` validator |
| Confidence field is sometimes a string, sometimes a float | `field_type()` catches it every time |
| Nested tool-call responses are hard to validate | `nested()` validator recurses cleanly |
| You switch providers and don't know what broke | Run the same contract against the new provider |

---

## Install

```bash
pip install llmcontract
```

No hard dependencies. Provider SDKs are optional:

```bash
pip install "llmcontract[openai]"
pip install "llmcontract[anthropic]"
pip install "llmcontract[all]"
```

---

## Quickstart

### 1. Define a contract

```python
from llmcontract import Contract, field, required_fields, no_extra_fields

trade_contract = Contract(
    required_fields("action", "confidence"),
    field("action",     type=str,   values=["buy", "sell", "hold"]),
    field("confidence", type=float),
    field("reasoning",  type=str,   min_len=10, required=False),
    no_extra_fields("action", "confidence", "reasoning"),
    name="TradeSignal",   # optional, shown in error messages
)
```

### 2. Validate an OpenAI response

```python
import openai
from llmcontract import openai_response

client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Give me a trade signal as JSON."}],
)

data = openai_response(response)
trade_contract.validate(data)
```

### 3. Validate an Anthropic response

```python
import anthropic
from llmcontract import anthropic_response

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=256,
    messages=[{"role": "user", "content": "Give me a trade signal as JSON."}],
)

data = anthropic_response(response)
trade_contract.validate(data)
```

### 4. Validate a raw JSON string

```python
from llmcontract import raw_json

text = '```json\n{"action": "buy", "confidence": 0.87}\n```'
data = raw_json(text)          # strips fences, parses JSON
trade_contract.validate(data)
```

---

## Validators

| Validator | What it checks |
|---|---|
| `required_fields(*names)` | All named fields exist |
| `field_type(name, type)` | Field is a specific Python type |
| `enum_values(name, allowed)` | Field value is in a list of allowed values |
| `min_length(name, n)` | String/list has at least n chars/items |
| `max_length(name, n)` | String/list has at most n chars/items |
| `regex_match(name, pattern)` | String field matches a regex |
| `no_extra_fields(*allowed)` | Response has no unexpected keys |
| `nested(name, contract)` | Sub-dict satisfies a child contract |
| `field(name, ...)` | All of the above in one call |

### `field()` — the Swiss Army knife

```python
field(
    "email",
    required=True,          # default True
    type=str,
    pattern=r"^[^@]+@[^@]+\.[^@]+$",
    min_len=5,
    max_len=254,
)
```

---

## Strict mode — collect all violations

By default, `validate()` raises on the **first** failure. In strict mode it
collects every failure and raises a single combined error:

```python
contract = Contract(
    required_fields("a", "b", "c"),
    strict=True,
)

try:
    contract.validate({})
except ContractViolation as exc:
    print(exc)
    # Contract [MyContract] violated with 3 error(s):
    #   • Required field missing: 'a'
    #   • Required field missing: 'b'
    #   • Required field missing: 'c'
```

---

## Extend a contract

```python
base = Contract(required_fields("id", "name"))
strict_version = base.extend(
    field("id",   type=int),
    field("name", type=str, min_len=1),
    no_extra_fields("id", "name"),
)
```

---

## Nested responses

```python
from llmcontract import Contract, nested, required_fields, field

address_contract = Contract(
    required_fields("street", "city", "country"),
    field("country", type=str, min_len=2, max_len=2),  # ISO 3166-1 alpha-2
)

person_contract = Contract(
    required_fields("name", "address"),
    field("name", type=str),
    nested("address", address_contract),
)

person_contract.validate({
    "name": "Alice",
    "address": {"street": "42 Main St", "city": "Springfield", "country": "US"},
})
```

---

## Use with pytest

```python
# tests/test_my_llm.py
import pytest
from llmcontract import Contract, required_fields, field

@pytest.fixture
def contract():
    return Contract(
        required_fields("answer", "confidence"),
        field("confidence", type=float),
    )

def test_response_shape(contract, mock_llm_response):
    data = raw_json(mock_llm_response)
    contract.validate(data)   # pytest shows ContractViolation as assertion error
```

---

## Exceptions

All exceptions inherit from `ContractViolation(AssertionError)`:

- `FieldMissingError` — required field absent
- `FieldTypeError` — wrong Python type
- `EnumViolationError` — value not in allowed list
- `LengthViolationError` — string/list too short or too long
- `PatternViolationError` — regex did not match
- `ExtraFieldError` — unexpected keys present

---

## Provider adapters

| Adapter | Usage |
|---|---|
| `openai_response(resp)` | `openai>=1.0` `ChatCompletion` objects |
| `anthropic_response(resp)` | `anthropic>=0.20` `Message` objects |
| `litellm_response(resp)` | LiteLLM `ModelResponse` (OpenAI-compatible) |
| `raw_json(text)` | Raw JSON string, strips code fences |
| `raw_dict(data)` | Pass-through for already-parsed dicts |

---

## License

MIT
