Metadata-Version: 2.4
Name: pytest-agent-health
Version: 0.1.0
Summary: Catch silent agent failures in CI. Behavior lint for LLM agents, powered by agent-failure-debugger.
Author: Kiyoshi Sasano
License-Expression: MIT
Project-URL: Homepage, https://github.com/kiyoshisasano/pytest-agent-health
Project-URL: Repository, https://github.com/kiyoshisasano/pytest-agent-health
Project-URL: Issues, https://github.com/kiyoshisasano/pytest-agent-health/issues
Keywords: pytest,llm,agent,testing,ci,failure-detection,lint,langgraph,langchain
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pytest>=7.0
Requires-Dist: agent-failure-debugger>=0.3.1
Provides-Extra: langchain
Requires-Dist: agent-failure-debugger[langchain]; extra == "langchain"
Dynamic: license-file

# pytest-agent-health

Catch silent agent failures in CI. Behavior lint for LLM agents.

```bash
pip install pytest-agent-health
```

```python
def test_agent_answers_correctly(agent_health):
    raw_log = my_agent.run("What was Q3 revenue?")
    agent_health.check(raw_log, adapter="langchain")
    # → FAIL if execution failed or quality degraded
    # → WARN if minor concerns detected
    # → PASS if healthy
```

---

## The Problem

Traditional tests can't catch silent agent failures:

```python
def test_agent():
    response = agent.run("Book me a flight to LAX")
    assert response is not None  # PASSES — but agent just said "I can help!"
```

The agent produced a response, so assertions pass. But it never actually booked anything. pytest-agent-health detects this by analyzing the execution trace — tool usage, grounding, alignment, termination behavior — and fails the test when the agent silently broke.

---

## How It Works

```
agent log → diagnose() → execution_quality → CI policy → FAIL / WARN / PASS
                (debugger)                     (plugin)
```

The plugin wraps [agent-failure-debugger](https://github.com/kiyoshisasano/agent-failure-debugger), which detects 17 failure patterns using deterministic causal analysis (no ML). The CI policy layer converts diagnosis results into test verdicts.

---

## Usage

### Single-run check

```python
def test_agent_completes_task(agent_health):
    raw_log = my_agent.run("What was Q3 revenue?")
    agent_health.check(raw_log, adapter="langchain")
```

If the agent's execution is `failed` or `degraded` with risk indicators, the test fails with a diagnostic report:

```
❌ FAIL: execution_quality=degraded
  FAILED: incorrect_output [retryable]
    → In production, consider create_health_check() for automatic recovery
  WARN [risk]: response.alignment_score = 0.12
    → output alignment with user intent is weak
```

### Multi-run stability

```python
def test_agent_consistency(agent_health):
    logs = [my_agent.run("Q3 revenue?") for _ in range(3)]
    agent_health.compare(logs, adapter="langchain", min_agreement=1.0)
    # → FAIL if root_cause_agreement < 1.0
```

### Regression detection

```python
def test_no_regression(agent_health):
    success_logs = load_baseline_logs()
    current_logs = [my_agent.run(q) for q in test_queries]
    agent_health.diff(success_logs, current_logs, adapter="langchain")
    # → FAIL if new failure patterns appear
```

---

## CI Policy

The plugin applies a CI-specific policy on top of execution quality:

### Default

| Condition | Verdict | Rationale |
|---|---|---|
| `failed` | **FAIL** | Task incomplete, error, or silent exit |
| `degraded` | **WARN** | Output produced but quality concerns detected |
| `healthy` | **PASS** | No issues |

### Strict (`--strict`)

| Condition | Verdict | Rationale |
|---|---|---|
| `failed` | **FAIL** | Task incomplete, error, or silent exit |
| `degraded` + risk indicators | **FAIL** | Weak alignment, no tool data, hallucination signals |
| `degraded` + info indicators only | **WARN** | Diagnostic limitations, not agent failure |
| `healthy` | **PASS** | No issues |

### Indicator classification

Degradation indicators are classified as risk (agent quality problem) or info (diagnostic limitation):

| Tag | Indicators | Meaning |
|---|---|---|
| `[risk]` | alignment_score, tool_provided_data, tool_result_diversity, expansion_ratio | Output may be wrong or unreliable |
| `[info]` | observation_coverage, unmodeled_failure, conflicting_signals | Diagnosis has limited confidence |

### Pattern override

Force specific failure patterns to always fail, regardless of execution quality:

```bash
pytest --agent-health --agent-health-fail-on=premature_termination,context_truncation_loss
```

---

## CLI Options

```bash
pytest --agent-health                    # Enable the plugin
pytest --agent-health --agent-health-strict  # Strict: degraded+risk = FAIL
pytest --agent-health --agent-health-fail-on=premature_termination
```

---

## Marker

Override policy per-test:

```python
@pytest.mark.agent_health(strict=True)
def test_critical_flow(agent_health):
    agent_health.check(log, adapter="langchain")

@pytest.mark.agent_health(fail_on=["premature_termination"])
def test_critical_flow(agent_health):
    agent_health.check(log, adapter="langchain")
```

---

## CI Output

The terminal summary shows all checks at the end of the test session:

```
============================= agent-health summary =============================
agent-health: 4 checks — 2 FAIL, 2 PASS

Failed checks (2):
  ❌ FAIL: execution_quality=degraded
    WARN [risk]: response.alignment_score = 0.0
      → output alignment with user intent is weak
  ❌ FAIL: execution_quality=degraded
    FAILED: incorrect_output [retryable]
      → In production, consider create_health_check() for automatic recovery
    WARN [risk]: response.alignment_score = 0.12
      → output alignment with user intent is weak
```

Failure patterns are tagged `[retryable]` or `[structural]`. Retryable failures can be automatically recovered in production using [create_health_check()](https://github.com/kiyoshisasano/agent-failure-debugger#self-healing-agent-langgraph). Structural failures require prompt or configuration changes.

---

## Non-determinism

LLM agents are non-deterministic. The same test may produce different results across runs. This is expected behavior, not a plugin bug.

Recommendations for stable CI:

- Set `temperature=0` in your agent configuration
- Use model-specific seed parameters where available
- Use `agent_health.compare()` to explicitly test for consistency
- Accept that some flakiness is inherent to LLM-based systems

---

## Related

| Package | Role |
|---|---|
| [agent-failure-debugger](https://github.com/kiyoshisasano/agent-failure-debugger) | Diagnosis engine (this plugin wraps it) |
| [llm-failure-atlas](https://github.com/kiyoshisasano/llm-failure-atlas) | Failure patterns, signals, matcher |
| [create_health_check()](https://github.com/kiyoshisasano/agent-failure-debugger#self-healing-agent-langgraph) | Runtime self-healing (complementary to CI) |

**CI detects. Production heals.** Use pytest-agent-health to catch failures in CI, and create_health_check() to automatically recover in production.

---

## License

MIT License.
