Metadata-Version: 2.4
Name: agent-diagnostician
Version: 0.1.0
Summary: Framework-agnostic Python library for diagnosing LLM agent execution failures
Author-email: Param Patel <parampatel.005@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/Param-Patel-o5/agent-failure-diagnostician
Project-URL: Repository, https://github.com/Param-Patel-o5/agent-failure-diagnostician
Project-URL: Issues, https://github.com/Param-Patel-o5/agent-failure-diagnostician/issues
Keywords: llm,agents,diagnostics,debugging,langchain,langgraph
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.5.0
Requires-Dist: sentence-transformers>=2.7.0
Requires-Dist: scikit-learn>=1.4.2
Requires-Dist: google-generativeai>=0.5.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Provides-Extra: llm-openai
Requires-Dist: openai>=1.0.0; extra == "llm-openai"
Provides-Extra: llm-anthropic
Requires-Dist: anthropic>=0.25.0; extra == "llm-anthropic"
Provides-Extra: llm-all
Requires-Dist: openai>=1.0.0; extra == "llm-all"
Requires-Dist: anthropic>=0.25.0; extra == "llm-all"
Requires-Dist: google-generativeai>=0.5.0; extra == "llm-all"
Dynamic: license-file

# Agent Diagnostician

When an LLM agent fails, the run log usually tells you *that* it failed — not *why*.

**Agent Diagnostician** reads a JSON trace (task, steps, tool calls, outputs) and returns a diagnosis: failure type, subtype, confidence, evidence, and a suggested fix direction. It does not depend on LangChain, LangGraph, or any other agent framework.

Python 3.11+.

```python
from agent_diagnostician import Classifier
from agent_diagnostician.tracer import load_fixture
from agent_diagnostician.reporter import Reporter

classifier = Classifier()  # no API key; uses rules + embeddings
trace = load_fixture("test cases/context_loss/context_loss__grounding__dropped_value.json")
Reporter.print(classifier.diagnose(trace))
```

Pass an LLM judge when you want the third detection tier (ambiguous tool use, goal misinterpretation, hallucination, and similar cases). Without a key, those paths stay conservative rather than calling an API.

## Install

```bash
pip install agent-diagnostician
```

From a clone of this repo:

```bash
pip install -e .
pip install -e ".[llm-openai]"      # optional
pip install -e ".[llm-anthropic]"   # optional
```

The first `diagnose()` download of the embedding model (`all-MiniLM-L6-v2` via Hugging Face) is about **90MB** and needs network access. After that, embeddings run locally.

Core install currently includes Pydantic, sentence-transformers, scikit-learn, and the Gemini SDK (default live provider). OpenAI and Anthropic are extras.

## Quick start

Build a trace in code:

```python
from agent_diagnostician import Classifier
from agent_diagnostician.models.trace import AgentTrace, Step
from agent_diagnostician.reporter import Reporter

trace = AgentTrace(
    run_id="run_001",
    task="Refund order ORD-123 for $49.99",
    status="failed",
    total_steps=1,
    final_output=None,
    steps=[
        Step(
            step_index=0,
            tool_name="issue_refund",
            tool_input={"order_id": "ORD-999", "amount": 49.99},
            tool_output={"error": "Order not found"},
        ),
    ],
)

classifier = Classifier()
result = classifier.diagnose(trace)
Reporter.print(result)
print(result.failure_type.value, result.subtype, result.confidence_score)
```

Or load one of the synthetic fixtures in [`test cases/`](test%20cases/):

```python
from agent_diagnostician.tracer import load_fixture

trace = load_fixture("test cases/context_loss/context_loss__grounding__dropped_value.json")
```

More examples: [`examples/basic_usage.py`](examples/basic_usage.py).

### Optional: live LLM judge

Ambiguous cases can call Gemini, OpenAI, or Anthropic. `Classifier()` never does this on its own.

```bash
export LLM_PROVIDER=gemini          # gemini | openai | anthropic | mock
export LLM_API_KEY=your-key         # or GEMINI_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY
export LLM_MODEL=gemini-3.5-flash-lite
```

```python
from agent_diagnostician import Classifier
from agent_diagnostician.analysis.llm import create_llm_judge_from_env

classifier = Classifier(llm_judge=create_llm_judge_from_env())
```

Smoke test: `python scripts/configure_llm.py --test`. Rate limits and mock vs live behavior: [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md).

Run a subset of detectors:

```python
from agent_diagnostician.models.enums import FailureType

classifier = Classifier(
    enabled_detectors=[FailureType.TOKEN_EXHAUSTION, FailureType.INFINITE_LOOP],
)
```

## What it detects

| Failure | Typical question it answers |
|---------|-----------------------------|
| Tool use | Wrong tool, bad parameters, or wrong values? |
| Goal satisfaction | Did the agent ignore a constraint or misread the task? |
| Hallucination | Did a step invent facts not in the tools or context? |
| Context loss | Did later steps drop earlier information? |
| Token exhaustion | Did the run die on context-length / token errors? |
| Premature termination | Did it stop before the task was done? |
| Infinite loop | Same tool, same error, or spinning reasoning? |

The classifier runs the enabled detectors and returns the **single strongest diagnosis** (with a fixed priority order on ties).

Most detectors try cheap signals first, then embeddings, then an LLM only if needed:

1. **Rules** — schemas, constraints, error text
2. **Embeddings** — similarity (tool ranking, thought vs output, grounding)
3. **LLM judge** — structured prompts when the first two are inconclusive

Token exhaustion and infinite loop stay on rules/embeddings (no LLM). Hallucination uses the judge on each step. Thresholds live in [`agent_diagnostician/config.py`](agent_diagnostician/config.py). Pipeline detail: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).

## Evaluation

**53 / 60 (88.3%)** on synthetic JSON traces in [`test cases/`](test%20cases/) (Gemini `gemini-3.5-flash-lite` where a live judge was required). 36 fixtures resolved without an API call; 24 used live LLM.

| Category | Fixtures | Passed |
|----------|----------|--------|
| Context loss | 6 | 6 (100%) |
| Infinite loop | 9 | 9 (100%) |
| Hallucination | 9 | 9 (100%, live judge) |
| Premature termination | 6 | 6 (100%) |
| Classifier (multi-detector) | 5 | 4 (80%) |
| Goal satisfaction | 11 | 9 (81.8%) |
| Tool use | 14 | 10 (71.4%) |

These traces are synthetic and included in the repo so you can re-run the numbers. Known misses (7 fixtures): goal-failure aggregator edges, one classifier tiebreaker, and four tool-use fallback / negative-control cases. Per-fixture detail: [`docs/evaluation/SUMMARY.md`](docs/evaluation/SUMMARY.md).

```bash
python scripts/run_phased_fixture_evaluation.py \
  --initial-used 0 \
  --first-batch 3 \
  --batch-size 15 \
  --output docs/evaluation
```

Methodology: [`docs/evaluation/README.md`](docs/evaluation/README.md).

## Trace format

Minimum JSON:

```json
{
  "run_id": "run_001",
  "task": "User task text",
  "status": "failed",
  "total_steps": 2,
  "final_output": null,
  "steps": [
    {
      "step_index": 0,
      "tool_name": "search",
      "tool_input": {"query": "..."},
      "tool_output": {"results": []},
      "thought": "optional reasoning",
      "error_message": "optional"
    }
  ]
}
```

Optional: `available_tools`, `constraints`, `constraint_list`, token counts. Schema: [`agent_diagnostician/models/README.md`](agent_diagnostician/models/README.md). Adapters: [`docs/INTEGRATIONS.md`](docs/INTEGRATIONS.md).

## Docs

| Document | Contents |
|----------|----------|
| [API](docs/API.md) | Classes, methods, enums |
| [Architecture](docs/ARCHITECTURE.md) | Pipeline and detection tiers |
| [Integrations](docs/INTEGRATIONS.md) | LangChain and other frameworks |
| [Performance](docs/PERFORMANCE.md) | Embeddings, batching, cost |
| [Troubleshooting](docs/TROUBLESHOOTING.md) | API keys, quotas, mock vs live |
| [Evaluation](docs/evaluation/README.md) | How the fixture benchmark is run |
| [Contributing](CONTRIBUTING.md) | Development workflow |

## License

MIT — see [LICENSE](LICENSE).
