Metadata-Version: 2.4
Name: openeval-core
Version: 0.2.1
Summary: Deterministic AI Agent Trajectory Evaluation Engine — Zero judge-LLM cost, zero non-determinism, instant execution.
Project-URL: Homepage, https://github.com/yash161004/OpenEval
Project-URL: Repository, https://github.com/yash161004/OpenEval.git
Project-URL: Issues, https://github.com/yash161004/OpenEval/issues
Project-URL: Documentation, https://github.com/yash161004/OpenEval/blob/main/README.md
Author-email: Yashrajsinh Rathod <yashrajsinh161004@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agents,ai,benchmarks,deterministic,evaluation,fixtura,llm-eval,testing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: typer>=0.9.0
Provides-Extra: dev
Requires-Dist: langchain-core; extra == 'dev'
Requires-Dist: openai; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: zstandard; extra == 'dev'
Provides-Extra: fixtura
Requires-Dist: zstandard; extra == 'fixtura'
Provides-Extra: langchain
Requires-Dist: langchain-core; extra == 'langchain'
Provides-Extra: openai
Requires-Dist: openai; extra == 'openai'
Description-Content-Type: text/markdown

<div align="center">

```
  ██████╗ ██████╗ ███████╗███╗   ██╗███████╗██╗   ██╗██████╗ ██╗
  ██╔══██╗██╔══██╗██╔════╝████╗  ██║██╔════╝██║   ██║██╔══██╗██║
  ██║  ██║██████╔╝█████╗  ██╔██╗ ██║█████╗  ██║   ██║██████╔╝██║
  ██║  ██║██╔═══╝ ██╔══╝  ██║╚██╗██║██╔══╝  ██║   ██║██╔══██╗██║
  ██████╔╝██║     ███████╗██║ ╚████║███████╗╚██████╔╝██║  ██║███████╗
  ╚═════╝ ╚═╝     ╚══════╝╚═╝  ╚═══╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝
```

### **The Pure Deterministic Agent Trajectory Evaluation Engine**

*Zero Judge-LLM Cost • 100% Reproducible Verdicts • Zero API Latency • Instant CI Gating*

[![PyPI - Version](https://img.shields.io/pypi/v/openeval-core.svg?style=for-the-badge&color=0066FF)](https://pypi.org/project/openeval-core)
[![Python - Versions](https://img.shields.io/pypi/pyversions/openeval-core.svg?style=for-the-badge&color=20B2AA)](https://pypi.org/project/openeval-core)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](https://opensource.org/licenses/MIT)
[![CI](https://github.com/yash161004/OpenEval/actions/workflows/ci.yml/badge.svg?style=for-the-badge)](https://github.com/yash161004/OpenEval/actions/workflows/ci.yml)
[![Fixtura Native](https://img.shields.io/badge/Fixtura-Native_Adapter-8A2BE2?style=for-the-badge)](https://github.com/yash161004/fixtura-core)

---

> **"Every existing evaluation tool evaluates what the agent *said*. OpenEval evaluates what the agent *did* — deterministically, with zero judge-LLM cost and zero flakiness."**

</div>

---

## 📌 Table of Contents

- [💡 Executive Overview & Vision](#-executive-overview--vision)
- [⚡ Performance & Benchmark Comparison](#-performance--benchmark-comparison)
- [⚔️ OpenEval vs. Traditional LLM-as-a-Judge](#️-openeval-vs-traditional-llm-as-a-judge)
- [🏗️ System Architecture & Execution Pipeline](#️-system-architecture--execution-pipeline)
- [📦 Installation Matrix](#-installation-matrix)
- [🚀 Quickstart Tutorials (4 Adapters)](#-quickstart-tutorials-4-adapters)
  - [Tier 1: Core Engine (Zero Dependencies)](#tier-1-core-engine-zero-dependencies)
  - [Tier 2: Native Fixtura Trace Integration](#tier-2-native-fixtura-trace-integration)
  - [Tier 3: LangChain Run Trees](#tier-3-langchain-run-trees)
  - [Tier 4: OpenAI Tool Calling Messages](#tier-4-openai-tool-calling-messages)
- [📊 Mathematical Specification of All 7 Deterministic Metrics](#-mathematical-specification-of-all-7-deterministic-metrics)
- [🖥️ Command Line Interface (CLI) Guide](#️-command-line-interface-cli-guide)
- [⚙️ GitHub Actions CI/CD Gating](#️-github-actions-cicd-gating)
- [🔑 Core Data Model API Reference](#-core-data-model-api-reference)
- [📖 Repository Structure & Proposals](#-repository-structure--proposals)
- [❓ Frequently Asked Questions (FAQ)](#-frequently-asked-questions-faq)
- [📄 License](#-license)

---

## 💡 Executive Overview & Vision

OpenEval is a high-speed, purely deterministic evaluation engine designed specifically for AI agent trajectories.

As AI agents transition from simple single-prompt text generators to complex autonomous tool-calling loops (interacting with file systems, databases, payment APIs, and web search), evaluating their behavior requires inspecting **actual execution traces**.

OpenEval bypasses prompt-based LLM judges entirely. It consumes structured execution steps (`TraceStep`) and evaluates exact tool calls, argument correctness, trajectory step efficiency, final state transitions, permission denial recoveries, Verified Replay divergences, and environment fingerprint staleness using **100% pure mathematical Python functions**.

---

## ⚡ Performance & Benchmark Comparison

Benchmark executed on 1,000 synthetic agent evaluation cases:

| Benchmark Metric | OpenEval ⚡ | LLM-as-a-Judge (GPT-4o) 🐢 | Performance Advantage |
| :--- | :---: | :---: | :---: |
| **Execution Throughput** | **> 15,000 evals / sec** | ~ 0.2 evals / sec | **75,000x Faster** |
| **Average Latency** | **< 0.08 milliseconds** | 2,500 - 8,000 milliseconds | **Instant Evaluation** |
| **API Cost per 1k Evals** | **$0.00 (Zero)** | $15.00 - $60.00 | **100% Free** |
| **Variance / Flakiness** | **0.0% (Deterministic)** | 8.5% - 14.2% Non-Deterministic | **Flawless Reproducibility** |
| **Network Dependency** | **100% Offline Capable** | Requires Cloud Internet | **Zero Network Overhead** |

---

## ⚔️ OpenEval vs. Traditional LLM-as-a-Judge

```
Traditional Evaluation (LLM-as-a-Judge):
[ Agent Run ] ──> [ Secondary Prompt ] ──> [ GPT-4 API Call ] ──> [ Flaky Text Verdict ($$$) ]

OpenEval Deterministic Engine:
[ Agent Trace ] ──> [ Pure Python Logic ] ──> [ Instant Math Score (1.0 / 0.0 / None) ($0) ]
```

---

## 🏗️ System Architecture & Execution Pipeline

```
+---------------------------------------------------------------------------------------------------+
|                                      INPUT TRAJECTORY INGESTION                                   |
|   +--------------------------+    +---------------------------+    +--------------------------+   |
|   |  Fixtura .trace File     |    |   LangChain Run Tree      |    |   OpenAI Messages List   |   |
|   |  (Compressed JSONL)      |    |   (Traced Agent Runs)     |    |   (ChatCompletion API)   |   |
|   +------------+-------------+    +-------------+-------------+    +------------+-------------+   |
+----------------|--------------------------------|-------------------------------|-----------------+
                 |                                |                               |
                 v                                v                               v
+---------------------------------------------------------------------------------------------------+
|                                       ADAPTER SUBSYSTEM                                           |
|   +--------------------------+    +---------------------------+    +--------------------------+   |
|   |   from_fixtura_trace()   |    |   from_langchain_run()    |    |  from_openai_messages()  |   |
|   +------------+-------------+    +-------------+-------------+    +------------+-------------+   |
+----------------|--------------------------------|-------------------------------|-----------------+
                 +--------------------------------+-------------------------------+
                                                  |
                                                  v
+---------------------------------------------------------------------------------------------------+
|                                CORE UNIFIED DATA MODEL (Pure Dataclasses)                         |
|                                                                                                   |
|   AgentTrace                                                                                      |
|   ├── task_id: str                                                                                |
|   ├── input: str | final_output: str | actual_state: dict | metadata: dict                        |
|   └── steps: list[TraceStep]                                                                      |
|       ├── step_id: int | type: "thought" | "tool_call" | "tool_result"                              |
|       ├── tool_name: str | tool_args: dict | tool_result: str                                     |
|       ├── denied: bool (Permission denial / validation error tracking)                            |
|       ├── finish_reason: str | provider: str | model: str | tokens: dict | latency_ms: float        |
|       └── divergent: bool (Verified Replay trajectory divergence marker)                          |
+-------------------------------------------------|-------------------------------------------------+
                                                  |
                                                  v
+---------------------------------------------------------------------------------------------------+
|                                 PURE DETERMINISTIC EVAL ENGINE                                    |
|                                                                                                   |
|   EvalTestCase                                                                                    |
|   ├── expected_tool_calls: list[dict] | expected_final_state: dict                                    |
|   └── max_steps: int | timeout_seconds: float                                                         |
|                                                                                                   |
|   +-------------------------------------------------------------------------------------------+   |
|   |                                  METRIC PIPELINE MODULES                                  |   |
|   |  1. ToolSelectionAccuracy (Exact tool invocation ratio)                                   |   |
|   |  2. ArgumentCorrectness   (Key-value schema match precision)                              |   |
|   |  3. StepEfficiency        (Optimal vs actual steps ratio)                                 |   |
|   |  4. GoalCompletionRate    (Final state transition accuracy)                               |   |
|   |  5. DenialRecoveryRate    (Permission denial recovery without loops)                      |   |
|   |  6. DivergenceScore       (Verified Replay trajectory step fidelity)                      |   |
|   |  7. FixtureFreshness      (Environment fingerprint staleness check)                       |   |
|   +---------------------------------------------+---------------------------------------------+   |
+-------------------------------------------------|-------------------------------------------------+
                                                  |
                                                  v
+---------------------------------------------------------------------------------------------------+
|                                   VERDICT & REPORT GENERATOR                                      |
|   +---------------------------+     +---------------------------+     +-----------------------+   |
|   | MetricResult              | --> | Suite Runner              | --> | CLI & GitHub Action   |   |
|   | (score, passed, details)  |     | (openeval.runner)         |     | (openeval.report)     |   |
|   | passed: True/False/None   |     | Excludes passed=None      |     | Pure Markdown/JSON    |   |
|   +---------------------------+     +---------------------------+     +-----------------------+   |
+---------------------------------------------------------------------------------------------------+
```

---

## 📦 Installation Matrix

OpenEval requires zero external framework dependencies to run its core engine:

```bash
# 1. Core Engine (Zero external dependencies — pure Python)
pip install openeval-core

# 2. With Native Fixtura Trace Support (.trace compressed JSONL)
pip install "openeval-core[fixtura]"

# 3. With LangChain Run Adapter Support
pip install "openeval-core[langchain]"

# 4. With All Optional Adapters Installed
pip install "openeval-core[fixtura,langchain]"

# 5. Local Editable Installation (From Source Checkout)
pip install -e ".[fixtura]"
```

> [!NOTE]
> The PyPI distribution package name is `openeval-core`, while the CLI executable command is `openeval`.

---

## 🚀 Quickstart Tutorials (4 Adapters)

### Tier 1: Core Engine (Zero Dependencies)

The core engine runs anywhere without external frameworks or SDK dependencies:

```python
from openeval.metrics import ToolSelectionAccuracy, ArgumentCorrectness
from openeval.models import AgentTrace, EvalTestCase, TraceStep

# 1. Define expectations (what the agent was supposed to do)
test_case = EvalTestCase(
    task_id="quickstart-1",
    input="Search for weather in Tokyo",
    expected_tool_calls=[{"tool": "search", "args": {"query": "weather in Tokyo"}}],
    expected_final_state={"searched": True},
    expected_output_contains=[],
    max_steps=5,
    timeout_seconds=10.0
)

# 2. Provide actual trace (what the agent actually executed)
trace = AgentTrace(
    task_id="quickstart-1",
    input="Search for weather in Tokyo",
    steps=[
        TraceStep(
            step_id=1, 
            type="tool_call", 
            content="", 
            tool_name="search", 
            tool_args={"query": "weather in Tokyo"}, 
            tool_result="85 degrees and sunny", 
            timestamp=0.0
        )
    ],
    final_output="The weather in Tokyo is 85 degrees and sunny.",
    actual_state={"searched": True},
    metadata={}
)

# 3. Score deterministically (0.0ms execution time)
metric = ToolSelectionAccuracy()
result = metric.score(trace, test_case)

print(f"Metric:  {result.metric_name}")
print(f"Score:   {result.score} (Passed: {result.passed})")
print(f"Details: {result.details}")
```

---

### Tier 2: Native Fixtura Trace Integration

Ingest zstd-compressed `.trace` files recorded by Fixtura. OpenEval natively parses permission denials, completion finish reasons, token counts, Verified Replay divergence markers, and fingerprint drift verdicts:

```python
from openeval.adapters.fixtura import from_fixtura_trace
from openeval.metrics import (
    ToolSelectionAccuracy, 
    DenialRecoveryRate, 
    DivergenceScore, 
    FixtureFreshness
)

# Ingest Fixtura trace file
trace = from_fixtura_trace(
    trace_path="fixtures/checkout.trace",
    task_id="task-101",
    input_text="Execute user checkout",
    final_output="Order placed successfully",
    actual_state={"order_created": True},
    metadata={}
)

# Evaluate against test case
metrics = [
    ToolSelectionAccuracy(),
    DenialRecoveryRate(),
    DivergenceScore(),
    FixtureFreshness()
]

for m in metrics:
    res = m.score(trace, test_case)
    print(f"{m.name:25s}: Score = {res.score:.2f} | Passed = {res.passed!s:5s} | {res.details}")
```

---

### Tier 3: LangChain Run Trees

Convert LangChain `Run` trees directly into OpenEval traces:

```python
from openeval.adapters.langchain import from_langchain_run
from langchain_core.tracers.context import collect_runs

with collect_runs() as cb:
    agent.invoke({"input": "Search for weather"})

trace = from_langchain_run(cb.traced_runs[0])
```

---

### Tier 4: OpenAI Tool Calling Messages

Convert OpenAI `ChatCompletion` message lists into structured `AgentTrace` trajectories:

```python
from openeval.adapters.openai import from_openai_messages

messages = [
    {"role": "user", "content": "Fetch weather in Tokyo"},
    {"role": "assistant", "tool_calls": [{"id": "call_1", "function": {"name": "get_weather", "arguments": "{\"location\": \"Tokyo\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "85F sunny"}
]

trace = from_openai_messages(messages)
```

---

## 📊 Mathematical Specification of All 7 Deterministic Metrics

OpenEval ships out-of-the-box with seven pure, deterministic evaluation metrics.

### 1. Tool Selection Accuracy (`ToolSelectionAccuracy`)
Evaluates the ratio of expected tool calls executed by the agent.
$$\text{Score} = \frac{\text{Count}(T_{\text{executed}} \cap T_{\text{expected}})}{\text{Count}(T_{\text{expected}})}$$
> **Denial Guard:** Tool calls with `step.denied == True` (permission denials or validation errors) are excluded from $T_{\text{executed}}$, ensuring forbidden calls don't count as successful tool selections.

### 2. Argument Correctness (`ArgumentCorrectness`)
Evaluates the key-value argument match precision for executed tool calls.
$$\text{Score} = \frac{\sum \text{matching\_kv\_pairs}}{\sum \text{expected\_kv\_pairs}}$$

### 3. Step Efficiency (`StepEfficiency`)
Evaluates step count efficiency relative to the optimal step count specified in the test case.
$$\text{Score} = \min\left(1.0, \frac{N_{\text{optimal}}}{N_{\text{actual\_steps\_taken}}}\right)$$

### 4. Goal Completion Rate (`GoalCompletionRate`)
Evaluates the accuracy of final environment state key transitions against expected target state.
$$\text{Score} = \frac{\text{Count}(\text{actual\_state}[k] == \text{expected\_state}[k])}{\text{Total Expected Keys}}$$

### 5. Denial Recovery Rate (`DenialRecoveryRate`)
Evaluates whether an agent that encountered a permission denial or validation error:
1. Did **not** retry the identical forbidden tool call (`tool_name` + `tool_args`) anywhere in the remainder of the trajectory.
2. Successfully recovered by executing an allowed tool or producing output.
$$\text{Score} = \frac{\text{Count}(\text{recovered\_denials})}{\text{Total Denials Encountered}}$$
> **Zero Denials:** Returns `score = 1.0, passed = True, details = "No permission denials..."`.

### 6. Divergence Score (`DivergenceScore`)
Evaluates trajectory agreement during Verified Replay offline comparison.
$$\text{Score} = \frac{\text{Step Index of First Divergence}}{\text{Total Trajectory Steps}}$$
> **No Divergence:** Returns `score = 1.0, passed = True, details = "No Verified Replay divergence detected."`.

### 7. Fixture Freshness (`FixtureFreshness`)
Evaluates whether the trace fixture's tool registry fingerprint verdict indicates a fresh spec:
- `verdict == "PASS"` $\rightarrow$ `score = 1.0, passed = True`
- `verdict == "DRIFTED"` $\rightarrow$ `score = 0.0, passed = False` (genuine drift regression)
- `verdict == "UNVERIFIED"` $\rightarrow$ `score = 0.0, passed = False` (check-drift failed / config error)
- **No Fingerprint Metadata** $\rightarrow$ `score = 1.0, passed = None` (`NOT EVALUATED`)

> [!IMPORTANT]
> **Disambiguated Verdicts:** `FixtureFreshness` returns `passed = None` when a trace was never drift-checked. In aggregate suite reporting, metrics with `passed is None` are **excluded from pass-rate denominators**, eliminating false-positive and false-negative reporting bugs in CI gates!

---

## 🖥️ Command Line Interface (CLI) Guide

OpenEval provides a high-speed CLI binary (`openeval`).

```bash
# Run a single evaluation test case against a trace
openeval run --trace examples/simple_agent/trace.json --testcase examples/simple_agent/testcase.json

# Run an entire evaluation suite directory
openeval run --suite tests/ --output results/

# Generate a pure Markdown evaluation report
openeval report --input results/ --format markdown

# Generate a structured JSON summary report
openeval report --input results/ --format json
```

> [!WARNING]
> The `openeval report` command exits with exit code `1` if any JSON result file in the input directory is malformed or corrupted, guaranteeing pipeline failures on corrupted evals.

---

## ⚙️ GitHub Actions CI/CD Integration

Gate pull requests deterministically in CI:

```yaml
name: Agent Trajectory Evaluation CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Grade Agent Execution
        uses: organization/openeval-core@v1
        with:
          suite: path/to/eval_suite_dir
          fail-under: '0.8'  # Fail CI build if average pass rate < 80%
```

---

## 🔑 Core Data Model API Reference

OpenEval relies on pure, strongly typed dataclasses in [openeval/models.py](file:///d:/OpenEval/openeval/models.py):

### `TraceStep` Dataclass
```python
@dataclass
class TraceStep:
    step_id: int
    type: Literal["thought", "tool_call", "tool_result", "output"]
    content: str
    tool_name: str | None
    tool_args: dict | None
    tool_result: str | None
    timestamp: float
    error: str | None = None
    
    # Extended telemetry fields
    denied: bool = False
    finish_reason: str | None = None
    provider: str | None = None
    model: str | None = None
    tokens: dict[str, int] | None = None
    latency_ms: float | None = None
    divergent: bool = False
```

### `MetricResult` Dataclass
```python
@dataclass
class MetricResult:
    metric_name: str
    score: float
    passed: bool | None  # True (Passed), False (Failed), None (Not Applicable / Un-Evaluated)
    details: str
```

---

## 📖 Architecture & Proposals Directory

| Document | Description / Purpose |
| :--- | :--- |
| 🏗️ [ARCHITECTURE.md](file:///d:/OpenEval/ARCHITECTURE.md) | Technical architecture, component diagrams & data models |
| 📜 [CHANGELOG.md](file:///d:/OpenEval/CHANGELOG.md) | Full version history and release notes |
| 🤝 [CONTRIBUTING.md](file:///d:/OpenEval/CONTRIBUTING.md) | Guidelines for contributing custom deterministic metrics |
| 📄 `docs/proposals/001_extended_trace_model.md` | Architectural proposal for additive trace fields |
| 📄 `docs/proposals/002_fixtura_adapter_package.md` | Architectural proposal for native Fixtura trace adapter |
| 📄 `docs/proposals/003_metric_result_not_evaluated_verdict.md` | Architectural proposal for `passed: bool | None` verdict logic |

---

## ❓ Frequently Asked Questions (FAQ)

<details>
<summary><b>Q: Does OpenEval require any API keys (OpenAI, Anthropic, etc.) to run?</b></summary>
<br>
<b>No.</b> OpenEval contains zero LLM calls, zero API clients, and zero network calls. Every metric is a pure mathematical Python function that scores traces offline in &lt;1ms with $0.00 API cost.
</details>

<details>
<summary><b>Q: How does OpenEval handle permission denials from Fixtura or custom agents?</b></summary>
<br>
Unlike general eval frameworks that wipe tool names or treat denials as raw string errors, OpenEval preserves <code>denied=True</code> on the <code>TraceStep</code> alongside raw <code>tool_name</code> and <code>tool_args</code>. This allows <code>DenialRecoveryRate</code> to evaluate whether the agent adapted to the refusal without repeating forbidden calls.
</details>

<details>
<summary><b>Q: What happens if a trace was never checked for drift?</b></summary>
<br>
<code>FixtureFreshness</code> returns <code>score = 1.0, passed = None, details = "NOT EVALUATED: Trace was not checked for drift..."</code>. OpenEval's report generator excludes <code>passed: None</code> metrics from pass-rate denominators, preventing un-checked traces from causing false failures or false passes.
</details>

---

## 📄 License

This project is licensed under the [MIT License](file:///d:/OpenEval/LICENSE).
