Metadata-Version: 2.4
Name: fixtura
Version: 1.1.1
Summary: Deterministic execution recording and replay for AI agents
Author: Yashrajsinh Rathod
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: requests<3.0,>=2.0
Requires-Dist: zstandard<1.0,>=0.20
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: types-requests>=2.0; extra == "dev"
Provides-Extra: eval
Requires-Dist: openeval-core>=0.2.1; extra == "eval"
Provides-Extra: agent
Requires-Dist: openai>=1.0; extra == "agent"
Dynamic: license-file

# 🛡️ Fixtura

**Deterministic Execution Recording, Replay, & Drift Testing for AI Agents — Turn Real Agent Runs into Replayable Test Fixtures.**

[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Version: 1.1.0](https://img.shields.io/badge/version-1.1.0-green.svg)](pyproject.toml)
[![Tests: Passing](https://img.shields.io/badge/tests-131%20passed%2C%201%20skipped-success.svg)](PROJECT_STATUS.md)
[![Framework Agnostic](https://img.shields.io/badge/framework-agnostic-purple.svg)](ARCHITECTURE.md)

---

> 🚀 **Welcome to Fixtura!** Fixtura gives your AI agents **record & replay superpowers**. Capture real agent behavior once, redact sensitive secrets automatically, and replay those traces deterministically in CI/CD — with zero live API costs and zero flaky external calls!

---

## ✨ Features At A Glance — Why You'll Love Fixtura

* ⚡ **Lightning Fast Offline Replays**: Reproduce complex multi-step agent runs in milliseconds without calling live LLM models or touching real databases.
* 🔒 **Ironclad Security & Gatekeeping**: Capability tokens enforce granular read/write permissions at the tool execution boundary before side effects ever execute.
* 🧼 **Automatic Secret Redaction**: Built-in sanitizer scrubs planted API keys, passwords, and tokens before `.trace` files are stored.
* 🚨 **Automated Fixture Drift Shield**: Structural fingerprinting detects if a prompt or tool schema changed, preventing stale fixtures from giving false greens in CI.
* 📊 **Boundary Coverage Radar**: Instantly visualize tool coverage and boundary safety (permission denials, rate limits, errors).
* 🔌 **100% Framework Agnostic**: Plugs seamlessly underneath LangGraph, CrewAI, AutoGen, or custom agent loops in under 5 lines of code.

---

## 💡 What is Fixtura?

Fixtura is a **framework-agnostic execution-assurance layer** for AI agents. Sitting directly at the tool-execution boundary, Fixtura monitors an agent's tool calls, enforces real-time permission policies and rate limits, redacts secrets, records deterministic `.trace` files, and allows you to **replay, diff, and grade** those recordings offline without touching live systems.

```
       ┌────────────────┐
       │   AI  Agent    │ (LangGraph / CrewAI / AutoGen / Custom)
       └───────┬────────┘
               │ executor.call("tool_name", args)
               ▼
┌───────────────────────────────┐
│     Fixtura Executor Seam     │
├──────────────┬────────────────┤
│ Permission   │ Rate Limiter   │
│ Engine       │ & Circuit Bkr  │
└──────┬───────┴────────┬───────┘
       │                │
       ▼                ▼
┌──────────────┐ ┌──────────────┐
│  Real Tools  │ │ Execution    │ ───►  saved  .trace  artifact
│ (FS/SQL/HTTP)│ │ Recorder     │       (compressed & redacted)
└──────────────┘ └──────────────┘
```

---

## 🎯 Why Fixtura vs Production APM?

Most agent observability tools (**Langfuse**, **Phoenix**, **Braintrust**) focus on production monitoring and LLM latency metrics at scale. Fixtura takes a different approach: **recorded execution traces as literal, replayable test fixtures**.

| Metric | Production APM (Langfuse/Phoenix) | 🛡️ Fixtura Execution Assurance |
| :--- | :--- | :--- |
| **Primary Goal** | Telemetry, cost tracking, prompt latency | CI/CD regression testing & deterministic verification |
| **Execution Control** | Passive monitoring (read-only telemetry) | Active permission gatekeeper & circuit breaker |
| **Replay Mechanism** | Visual timeline inspection | Offline Passive & Verified Replay (zero live calls) |
| **Regression Testing** | LLM-as-a-judge scoring | Structural fingerprint drift detection (`check-drift`) |
| **CI Integration** | Post-hoc dashboards | Binary exit codes (`0` PASS, `1` DRIFTED, `2` UNVERIFIED) |

---

## 🏗️ System Architecture & Dataflow

```mermaid
flowchart TD
    classDef agent fill:#e1f5fe,stroke:#0288d1,stroke-width:2px,color:#01579b;
    classDef seam fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#e65100;
    classDef gate fill:#ffebee,stroke:#c62828,stroke-width:2px,color:#b71c1c;
    classDef storage fill:#ede7f6,stroke:#512da8,stroke-width:2px,color:#311b92;
    classDef engine fill:#e8f5e9,stroke:#388e3c,stroke-width:2px,color:#1b5e20;

    subgraph "Agent Runtime Layer"
        LLM["🤖 Agent Model / Orchestrator"]:::agent
    end

    subgraph "Fixtura Execution Seam"
        Exec["⚡ FixturaExecutor (Injected Seam)"]:::seam
    end

    subgraph "Protection & Security Gatekeeper"
        Perm["🔒 Permission Engine (Capability Token)"]:::gate
        Limiter["⏱️ Rate Limiter & Circuit Breaker"]:::gate
        Sanitizer["🧼 Secret Sanitizer (Redactor)"]:::gate
    end

    subgraph "Trace Artifacts"
        Rec["REC Execution Recorder"]:::storage
        TraceFile[("💾 .trace Artifact (Zstd Compressed)")]:::storage
    end

    subgraph "Verification & Replay Suite"
        Replay["⏪ Passive Replay"]:::engine
        Inspect["🔍 Step Inspector / UI"]:::engine
        Drift["🚨 check-drift (Fingerprint Verifier)"]:::engine
        Verify["✅ Verified Replay (Diff Engine)"]:::engine
        Coverage["📊 Fixture Library Coverage"]:::engine
        OpenEval["💯 OpenEval Adapter"]:::engine
    end

    LLM -->|"1. executor.call(tool, args)"| Exec
    Exec -->|"2. Authorize"| Perm
    Exec -->|"3. Quota check"| Limiter
    Perm -->|"4. Sanitize input/output"| Sanitizer
    Sanitizer -->|"5. Write event"| Rec
    Rec -->|"6. Produce"| TraceFile

    TraceFile --> Replay
    TraceFile --> Inspect
    TraceFile --> Drift
    TraceFile --> Verify
    TraceFile --> Coverage
    TraceFile --> OpenEval
```

---

## 🚀 Quickstart (5 Minutes)

### 1. Installation

```bash
# Install from PyPI:
pip install fixtura

# Or install in editable mode from a local checkout:
git clone https://github.com/yash161004/fixtura-core.git
cd fixtura-core
pip install -e .
```

### 2. Run the Interactive Quickstart Script

```bash
python quickstart.py
```

The script automatically demonstrates the complete 6-stage pipeline:
1. 🎬 **Record**: Captures a multi-step agent run to `quickstart_example.trace`.
2. ⏪ **Passive Replay**: Reproduces the run offline with zero live calls.
3. 🔍 **Step Inspection**: Interactively steps through each recorded event.
4. 🚨 **Drift Check**: Verifies trace structural fingerprints against current tool schemas and policies.
5. ✅ **Verified Replay**: Re-executes agent logic while substituting tool call responses.
6. 📊 **Coverage Analysis**: Computes tool and boundary coverage across the fixture library.

---

## 🔌 Wiring Fixtura into Your Agent

Fixtura integrates into your existing agent loop with **zero structural refactoring**. Simply inject `fixtura.executor` where your agent executes tools:

```python
import fixtura

# 1. Initialize executor in record or replay mode
executor = fixtura.executor(
    mode="record",  # Or mode="replay" for offline runs
    trace="agent_run.trace",
    tools=tool_registry,
    capability_token=capability_token,
)

# 2. Call tools inline from inside your agent loop
result = executor.call("filesystem_tool", {"operation": "read", "path": "notes.txt"})

if result.denied:
    print(f"🔒 Permission denied: {result.reason}")
else:
    print(f"✅ Tool output: {result.value}")
```

### Execution Modes:
* **Record Mode (`mode="record"`)**: Executes tools for real through the permission engine and rate limiter, sanitizing outputs and writing every event to `.trace`.
* **Replay Mode (`mode="replay"`)**: Returns recorded outputs inline with zero live tool execution. No capability token or real tools needed.

---

## ⚡ Core Features & Capabilities

### 1. 🚨 Fixture Drift Detection (`check-drift`)
A test fixture is only useful if it matches current code. If you update a tool's Pydantic schema or edit a system prompt, a stale trace replaying green creates a **false sense of security**.

Fixtura traces carry a cryptographic fingerprint of tool schemas, prompt text, and policy rules. `check-drift` detects changes before replaying:

```bash
fixtura check-drift fixtures/checkout.trace --agent my_agent:spec
```

```
DRIFTED: fixtures/checkout.trace
  agent:   my_agent:spec
  changed: tool schema changed: sqlite_tool
```

* **Exit Codes**: `0` PASS, `1` DRIFTED, `2` UNVERIFIED (malformed/unimportable spec).
* *Note*: Run from your project root or set `PYTHONPATH=.` so Python can resolve your custom `AgentSpec`.

### 2. ✅ Verified Replay (`verify`)
Verified Replay runs your agent's decision logic live while substituting tool call responses from the trace. It diffs generated tool requests against recorded tool requests while **normalizing volatile fields** (such as timestamps, UUIDs, and transient IDs).

```bash
fixtura verify fixtures/checkout.trace
```

### 3. 📊 Fixture Library Coverage (`coverage`)
Analyzes a corpus of trace files to report which registered tools have been tested, and whether boundary conditions (permission denials, errors, rate limits) have been exercised.

```bash
fixtura coverage fixtures/
```

```
Coverage over 4 trace(s), 12 recorded tool call(s)

  TOOL             CALLS  OUTCOMES SEEN
  ---------------  -----  ----------------------------------------
  filesystem_tool      4  allowed, denied
  sqlite_tool          5  allowed, error
  http_tool            3  denied, throttled

  Tool coverage:     100.0%  (registered tools appearing in any fixture)
  Boundary coverage:  83.3%  (registered tools seen failing, denied, or throttled)
```

### 4. 🔀 The 4 Replay Modes

```mermaid
matrix
    title Replay Mode Spectrum
    "Mode" : "Executes Tools?" : "Agent Runs?" : "Deterministic?"
    "Passive Replay" : "No" : "No" : "Yes (100%)"
    "Step Inspector" : "No" : "No" : "Yes (100%)"
    "Verified Replay" : "No" : "Yes" : "Partial (Tool-level)"
    "Live Branching" : "Yes" : "Yes" : "No (Live Run)"
```

| Mode | Command | Agent Runs? | Touches Live Systems? | Compares Output? |
| :--- | :--- | :---: | :---: | :---: |
| **Passive Replay** | `fixtura replay <trace>` | No | No | No |
| **Step Inspection** | `fixtura inspect <trace>` | No | No | No |
| **Verified Replay** | `fixtura verify <trace>` | Yes | No | **Yes** |
| **Live Branching** | `fixtura branch <parent> <out> <step> <prompt>` | Yes | **Yes (Live)** | No |

---

## 🛠️ CLI Reference Manual

| Command | Arguments | Description | Exit Code |
| :--- | :--- | :--- | :---: |
| `fixtura record` | `<trace>` | Record an agent run to a `.trace` artifact using demo agent | `0` |
| `fixtura replay` | `<trace>` | Passively replay a trace offline with zero live calls | `0` |
| `fixtura inspect` | `<trace>` | Step through a trace interactively line-by-line | `0` |
| `fixtura view` | `<trace>` | Print human-readable summary of recorded events | `0` |
| `fixtura html-view` | `<trace>` | Generate standalone HTML trace visualizer (`trace_viewer.html`) | `0` |
| `fixtura check-drift` | `<trace> [--agent module:attr]` | Verify if trace fingerprint matches current agent spec | `0` PASS / `1` DRIFTED / `2` UNVERIFIED |
| `fixtura verify` | `<trace> [prompt]` | Run agent decision logic offline & diff tool calls | `0` PASS / `1` MISMATCH / `2` DRIFTED |
| `fixtura coverage` | `<paths...>` | Report tool and boundary coverage across `.trace` files | `0` |
| `fixtura branch` | `<parent> <out> <step> <prompt>` | Branch trace execution from a step (executes live) | `0` |
| `fixtura eval` | `<trace>` | Score trace trajectory via OpenEval adapter | `0` |

---

## 🤖 Running Real Agent Loops

Fixtura includes a complete, provider-agnostic real agent loop in [`examples/real_agent.py`](file:///d:/fixtura-project-scaffold/fixtura/examples/real_agent.py). It works against any OpenAI-compatible LLM endpoint (Ollama, Groq, OpenRouter, Together, Gemini, OpenAI):

### Free Local Execution (Ollama):

```bash
ollama pull qwen3:4b
ollama serve
python -m examples.real_agent run.trace
```

### Hosted Provider Execution:

```bash
export FIXTURA_LLM_BASE_URL="https://api.groq.com/openai/v1"
export FIXTURA_LLM_MODEL="llama-3.3-70b-versatile"
export FIXTURA_LLM_API_KEY="your-api-key"

python -m examples.real_agent run.trace
```

---

## 💯 OpenEval Adapter (Optional)

Fixtura integrates with **OpenEval** for trajectory scoring. OpenEval is published on PyPI as [`openeval-core`](https://pypi.org/project/openeval-core/), so the `eval` extra installs it for you:

```bash
# Install Fixtura with the OpenEval adapter:
pip install 'fixtura[eval]'

# Score a trace trajectory:
fixtura eval quickstart_example.trace
```

Or add OpenEval to an existing install:

```bash
pip install 'openeval-core>=0.2.1'
```

> **Note on the distribution name.** The adapter requires **`openeval-core`**, not `openeval`. The bare `openeval` name on PyPI is an unrelated placeholder project that installs cleanly and does nothing. The native Fixtura adapter (`openeval.adapters.fixtura`) ships from **openeval-core 0.2.1** onward; earlier releases do not contain it, which is why the extra pins `>=0.2.1`.

---

## 📚 Repository Documentation Index

| Document | Description |
| :--- | :--- |
| 📐 [ARCHITECTURE.md](file:///d:/fixtura-project-scaffold/fixtura/ARCHITECTURE.md) | Detailed system design, trust boundaries, and scope decisions |
| 🛡️ [THREAT_MODEL.md](file:///d:/fixtura-project-scaffold/fixtura/THREAT_MODEL.md) | Security model, prompt injection defenses, secret sanitization |
| 🗺️ [ROADMAP.md](file:///d:/fixtura-project-scaffold/fixtura/ROADMAP.md) | Milestone tracking, frozen scope table, and acceptance criteria |
| 🚦 [PROJECT_STATUS.md](file:///d:/fixtura-project-scaffold/fixtura/PROJECT_STATUS.md) | Comprehensive test suite matrix and component build status |
| 📋 [CHANGELOG.md](file:///d:/fixtura-project-scaffold/fixtura/CHANGELOG.md) | Release notes and version history |
| 📄 [TRACE_FORMAT_SPEC.md](file:///d:/fixtura-project-scaffold/fixtura/docs/TRACE_FORMAT_SPEC.md) | Specification of the `.trace` binary format and JSON schema |
| 🔬 [LIVE_RUN_FINDINGS.md](file:///d:/fixtura-project-scaffold/fixtura/docs/LIVE_RUN_FINDINGS.md) | Empirical findings and fixes from live LLM model testing |
| 📜 [DENIAL_MESSAGE_PROPOSAL.md](file:///d:/fixtura-project-scaffold/fixtura/docs/DENIAL_MESSAGE_PROPOSAL.md) | Security proposal on permission-denial message granularity |

---

## 🤝 Community & Contributions

We welcome feedback, issues, and contributions from developer teams building reliable AI agents! 
* ⭐ **Star the repo** if Fixtura helps you catch agent regressions in CI!
* 💬 **Open an Issue** for suggestions, bug reports, or feature ideas.
* 📜 See [LICENSE](LICENSE) for MIT licensing details.

---

## 📄 License

This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
