Metadata-Version: 2.4
Name: mcp-guardeval
Version: 0.1.0
Summary: Automated evaluation harness for MCP-based agents: task success scoring, security attack suite, and OpenTelemetry trace analysis.
License: MIT
Keywords: mcp,langgraph,agent-evaluation,llm-security,opentelemetry
Author: Jeneesh Surani
Author-email: jeneeshsurani@gmail.com
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Requires-Dist: opentelemetry-api (>=1.25,<2.0)
Requires-Dist: opentelemetry-sdk (>=1.25,<2.0)
Requires-Dist: pydantic (>=2.7,<3.0)
Requires-Dist: pytest (>=8.2,<9.0)
Description-Content-Type: text/markdown

# mcp-guardeval

[![PyPI version](https://img.shields.io/pypi/v/mcp-guardeval.svg)](https://pypi.org/project/mcp-guardeval/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**`mcp-guardeval`** is a standalone, framework-agnostic evaluation harness for LLM agents utilizing the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/). It captures OpenTelemetry spans (`gen_ai.*` conventions), normalizes them into queryable SQLite records, and quantitatively scores both **task performance** and **security resilience** against catalogued **SAFE-MCP** adversarial techniques.

---

## Installation

```bash
pip install mcp-guardeval
```

---

## Key Capabilities

1. **Task Success Scoring**: Measures whether the agent called the expected tools, supplied correct arguments, and respected tool dependencies.
2. **Adversarial Security Scoring**: Evaluates agent behavior against red-team attack techniques (prompt injection, argument hijacking, data exfiltration, context planting, PII harvesting).
3. **Telemetry & Trace Analysis**: Normalizes hierarchical OpenTelemetry distributed traces into flat SQLite rows for high-speed SQL analytics.
4. **Pytest Integration**: Built-in pytest plugin enabling single-command evaluation:
   ```bash
   pytest tests/ --agenteval -v
   ```

---

## Quick Start

### 1. Telemetry Ingestion & Storage

`mcp-guardeval` reads OpenTelemetry trace data persisted to SQLite:

```python
from agenteval.storage import TraceStore

# Connect to the SQLite trace database
store = TraceStore("traces.db")

# Inspect completed agent runs
runs = store.get_runs()
for run in runs:
    print(f"Run {run.run_id}: {run.prompt[:50]}... | {run.latency_ms:.1f}ms")
```

### 2. Scoring Task Success

Evaluate whether an agent executed the intended workflow:

```python
from agenteval.metrics.task_success import score_task
from agenteval.storage import TraceStore

store = TraceStore("traces.db")

# Score a customer lookup workflow
result = score_task(
    store=store,
    run_id="run_1042",
    expected_tools=["query_customer_db"],
    expected_args={"query_customer_db": {"customer_id": "4471"}},
    expected_sequence=["query_customer_db"],
)

print(f"Task Passed: {result.passed}")
print(f"Confidence Score: {result.score:.2f}")
print(f"Missing Tools: {result.missing_tools}")
```

### 3. Evaluating Guardrail Security

Assess whether guardrails intercepted adversarial SAFE-MCP attacks:

```python
from agenteval.metrics.security import score_security_run
from agenteval.storage import TraceStore

store = TraceStore("traces.db")

# Evaluate an indirect prompt injection attack (SAFE-T1201)
verdict = score_security_run(
    store=store,
    run_id="attack_run_88",
    technique_id="SAFE-T1201",
    guardrail_log="reference_system/fixtures/guardrail.log",
)

# Verdict options: BLOCKED (secure), PASSED (exploited), or PARTIAL
print(f"Security Verdict: {verdict.status.name}")
print(f"Mitigation Reason: {verdict.reason}")
```

---

## Pytest Plugin Usage

`mcp-guardeval` automatically registers with pytest when installed. Use the `--agenteval` CLI flag to activate trace evaluation and automated reporting during test execution:

```bash
# Run security test suite with AgentEval report summary
pytest tests/test_security.py --agenteval -v
```

---

## Supported SAFE-MCP Attack Techniques

| Technique ID | Name | Category |
|---|---|---|
| `SAFE-T1201` | Prompt injection to hijack tool selection | Execution |
| `SAFE-T1203` | Tool argument hijacking (SQLi, Path Traversal) | Execution |
| `SAFE-T1208` | Indirect data exfiltration via downstream tools | Exfiltration |
| `SAFE-T1301` | Context instruction planting | Persistence |
| `SAFE-T1601` | System prompt and credential disclosure | Discovery |
| `SAFE-T1102` | Indirect prompt injection via retrieved content | Execution |
| `SAFE-T1501` | Cross-tool bulk PII harvesting | Collection |

---

## License

Distributed under the MIT License. See `LICENSE` for more information.

