Metadata-Version: 2.5
Name: crilio
Version: 0.0.6
Summary: The CI/CD quality gate for AI. Stop shipping prompt regressions to production.
Project-URL: Homepage, https://github.com/crilio/crilio
Project-URL: Repository, https://github.com/crilio/crilio
Author: Crilio
License: AGPL-3.0-only
License-File: LICENSE
Keywords: ai-quality,ci-cd,llm,openai,prompt-regression,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Affero General Public License v3
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.9
Requires-Dist: anthropic>=0.34.0
Requires-Dist: instructor>=1.3.0
Requires-Dist: openai>=1.40.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7.0
Requires-Dist: typer[all]>=0.9.0
Description-Content-Type: text/markdown

<div align="center">
<img src="./assests/readme_logo.png" alt="Crilio" width="170" />
<h1 style="margin: -4px 0 0 0;">crilio</h1>
<p style="margin: 6px 0 0 0;"><strong>The CI/CD quality gate for AI — pytest for prompts.</strong></p>

<p style="margin: 12px 0 0 0; font-size: 14px;">If Crilio helps you, please ⭐ <a href="https://github.com/mukundzha/crilio"><strong>star this repo</strong></a> — it helps other devs find it and motivates me to keep building.</p>

<p style="margin: 6px 0 0 0; font-size: 14px;">☁️ <a href="https://tally.so/r/0QRj4j">Join the Crilio Cloud Waitlist</a> (Get 40% off team dashboards & analytics when we launch)</p>

  <p align="center" style="margin-top: 14px;">
    <a href="https://pypi.org/project/crilio/"><img src="https://img.shields.io/pypi/v/crilio?style=flat-square&label=PyPI&color=black" alt="PyPI"/></a>
    <a href="https://pypi.org/project/crilio/"><img src="https://img.shields.io/pypi/pyversions/crilio?style=flat-square&label=python&color=black" alt="Python"/></a>
    <a href="./LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-black?style=flat-square" alt="License"/></a>
  </p>
</div>

---

**Crilio stops prompt regressions from reaching production.** Version your prompts + natural-language rules in `crilio.yaml` — Crilio calls your model, judges every response with an LLM, and reports `PASS / FAIL` per rule.

---

### ✨ Features

- **LLM-as-a-Judge** — Strict Pydantic-verified verdicts (`rule_passed: bool`), temp 0. No flaky free-text parsing.
- **BYOK** — Your keys, your bill. OpenAI + Anthropic. Typically < $0.01 / test on `gpt-4o-mini`.
- **CI/CD Native** — `exit 1` blocks PRs in GitHub Actions. Locally it warns but never blocks.
- **PR Comments** — Auto-posts formatted failure details to the PR when running in Actions (`GITHUB_TOKEN`, silent fail, never blocks gate).
- **Leak Guard** — Rejects `crilio.yaml` containing `sk-...`/`api_key` — keys must be in `.env`/Secrets, never committed.
- **Local Bots** — `target: {command: "python bot.py '{{prompt}}'"}` runs any local model (Ollama/vLLM) via stdout → Judge, `$0` target.
- **Skip & List** — `skip: true` per-test to pause, `crilio ls [--tag] [--json]` to preview tests without running.
- **Diff** — `crilio diff --base main` shows prompt/rule changes between git refs (`+`/`-` per rule, not whole list).
- **Budget Guard** — `max_monthly_budget_usd` halts the run when cost exceeds cap. Delete the line or leave it blank for unlimited.
---

### 🚀 Quick Start

```bash
pip install crilio

export OPENAI_API_KEY="sk-proj-..."  # or ANTHROPIC_API_KEY — .env also works
crilio init                          # creates crilio.yaml
crilio run --dry-run                 # validate without API calls
crilio run                           # Target → Judge → gate
```

> `crilio` with no args shows status, budget, and next steps.

---

### ⚙️ Usage

| Command | Description |
|---|---|
| `crilio init [--force] [--yes]` | Create `crilio.yaml` + optional GitHub Actions workflow |
| `crilio ls [-c FILE] [--tag TAG] [--json]` | List tests — preview without running |
| `crilio diff [--base REF] [-c FILE] [--json] [--fail-on-change]` | Show prompt/rule diff between git refs |
| `crilio run [-c FILE] [-m MODEL] [--judge-model MODEL] [--verbose] [--json] [--dry-run] [--tag TAG]` | Run gate — `0` pass, `1` fail (only in CI) · `--tag` filters to `tags: [TAG]` tests |
| `crilio validate [-c FILE] [--json]` | Validate config without API calls — `0` valid, `2` invalid |
| `crilio --version` / `crilio --docs` | Version / full interactive guide |

| Variable | Description |
|---|---|
| `OPENAI_API_KEY` | OpenAI — default target `gpt-4o-mini`, judge `gpt-4o-mini` |
| `ANTHROPIC_API_KEY` | Anthropic — default target `claude-3-5-sonnet-latest`, judge `claude-3-5-haiku-latest` |

Provider is inferred from env if not set in `crilio.yaml`. Keys are **never** flags — env only.

<details>
<summary><strong>crilio.yaml</strong></summary>

```yaml
settings:
  target_model: gpt-4o
  judge_model: gpt-4o-mini
  max_monthly_budget_usd: 10.0  # delete or leave blank for unlimited

tests:
  - name: Refund Policy Check
    prompt: How long do I have to return a product?
    rules:
      - Must mention the 30-day return window.
      - Must NOT mention competitor names.
    tags: ["smoke", "critical"]

  - name: JSON Format Check
    prompt: |
      Return ONLY this JSON and nothing else: {"status": "shipped", "order_id": "12345"}
    rules:
      - Must return valid JSON with keys 'status' and 'order_id'.
      - Must NOT include apologies or extra prose.

  - name: Local Bot Check
    prompt: How long do I have to return a product?
    target:
      command: "python bot.py '{{prompt}}'"
    rules:
      - "Must mention the 30-day return window."
    tags: ["local"]
```

Per-test overrides: `provider`, `model`, `judge_model`, `system`, `tags`, `target`, `skip` can be set per test. `skip: true` pauses, `crilio ls` previews. Use `crilio run --tag smoke` to run only tagged tests. `target.command` runs local CLI (`{{prompt}}` → `shlex.quote`, no placeholder → stdin, 30s timeout).

</details>

<details>
<summary><strong>🦙 Ollama template — test any local model</strong></summary>

```yaml
# 1. ollama serve & ollama pull llama3  (or mistral, qwen2, etc.)
# 2. crilio.yaml:
settings:
  target_model: gpt-4o
  judge_model: gpt-4o-mini

tests:
  - name: Ollama Refund
    prompt: "How long do I have to return a product?"
    target:
      command: "ollama run llama3 '{{prompt}}'"  # any model: mistral, qwen2, gemma
      # no placeholder also works → stdin: command: "ollama run llama3"
    rules:
      - "Must mention the 30-day return window."
    tags: ["ollama", "local"]

# 3. export OPENAI_API_KEY="sk-..."  # Judge still API
# 4. crilio run --tag ollama --verbose
```

</details>

---

### 🔄 CI/CD Integration

`crilio init` can scaffold this for you. Or drop in `.github/workflows/crilio.yml`:

```yaml
name: Crilio AI Tests
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.10' }
      - run: pip install crilio
      - run: crilio run
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  # for PR failure comments (auto-provided)
```

Add the secret matching your provider. A `FAIL` gate exits `1` and blocks the PR. On failure in Actions, Crilio posts a `🛑 Crilio AI Test Failed` comment with test, rule, AI response and reason — requires `permissions: pull-requests: write` (or default `GITHUB_TOKEN`), fails silently locally or on API error and never blocks the gate.

---

### 📄 License

AGPL-3.0 — see [LICENSE](./LICENSE).
