Metadata-Version: 2.4
Name: trace-use
Version: 0.1.4
Summary: Learn failure patterns from LLM agent traces and intercept recurrences before they execute.
License-Expression: MIT
Keywords: llm,agents,reliability,failure-prediction,ai
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: anthropic>=0.40
Requires-Dist: openai>=1.40
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Requires-Dist: sentence-transformers>=2.7
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rich>=13.0
Requires-Dist: matplotlib>=3.7
Provides-Extra: bench
Requires-Dist: datasets>=2.18; extra == "bench"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"

# trace_use

[![PyPI](https://img.shields.io/pypi/v/trace-use)](https://pypi.org/project/trace-use/)
[![Python](https://img.shields.io/pypi/pyversions/trace-use)](https://pypi.org/project/trace-use/)

**A developer's failure memory — so you never make the same logical mistake twice.**

`trace_use` learns from LLM agent failures, extracts the logical principle behind each one, and warns the agent before the next occurrence — even if it happened a week ago on completely different code.

---

## The story

Agents fail in patterns, not randomly. The same logical mistake — catching all exceptions instead of selective ones, sorting on a single key when a tiebreak is required, returning `None` instead of raising on a missing required field — recurs across different tasks, different code, different sessions. Each recurrence burns tokens on a failure → diagnosis → retry loop that was avoidable.

`trace_use` intercepts at two points:

| Layer | When | What it does |
|---|---|---|
| **`TrajectoryDetector`** | Before the LLM writes any code | Compares the task description against past failures; injects targeted warnings into the prompt |
| **`BrainAgent`** | Before each `python_exec` call | Checks proposed code against stored motifs; fires a STOP message if the exact logical gap is present |

---

## How the learning works

When a task fails, a single background LLM call extracts *why* — not what the code looked like, but what logical requirement was violated. This produces a `FailureMotif`:

```
FailureMotif
  id:                  "unconditional_retry_on_failure"
  name:                "Retry Logic Does Not Differentiate Retryable Errors"
  description:         "Retry logic does not distinguish between retryable
                        and non-retryable exceptions."
  required_condition:  "task requires selective retry based on error type or status code"
  violation_condition: "except Exception catches all types without type check"
  recommendation:      "check exception type or status code before deciding to retry"
```

The motif is abstract — no variable names, no task-specific constants. In a 56-task benchmark, one failure on `retry_request` produced this motif. It then fired correctly on `retry_on_type` and `safe_request` — tasks with completely different code and vocabulary — converting both from failures to passes.

Motifs persist to `~/.trace_use/motifs.json` across sessions. A developer who hits a retry-logic bug on Monday will get a warning injected into their prompt on Friday, before writing a single line of code.

---

## Two-layer detection

### Layer 1 — TrajectoryDetector (pre-prompt, before any code is written)

Before the LLM generates anything, the detector retrieves candidate motifs by embedding similarity and runs a cheap LLM judge call for each:

> "Does this task explicitly mention the logical requirement that caused the past failure? Quote the exact text."

The judge must return a verbatim quote from the task text — no paraphrasing. If found, a `KNOWN PITFALLS` block is prepended:

```
⚠️  KNOWN PITFALLS — based on recorded failures:

  [1] Retry Logic Does Not Differentiate Retryable Errors
      Why this applies: "only retry on ConnectionError or Timeout"
      Watch out for:    Code may catch all exceptions instead of just the listed types.
      Recommendation:   check exception type or status code before deciding to retry

Address these before writing your implementation.
```

### Layer 2 — BrainAgent (pre-execution, before each tool call)

Before each `python_exec` call, the brain checks the proposed code against stored motifs. The judge must return verbatim quotes from both the task description AND the proposed code. When both are found, a STOP message fires before the bad code runs:

```
⚠️ BRAIN:
STOP: The monitor detected a likely logical failure before execution.

Evidence (Learned pattern: Retry Logic Does Not Differentiate Retryable Errors):
  - Requirement: Only retry exceptions listed in retry_on.
  - Violation:   except retry_on as e:
  - Explanation: code retries unconditionally instead of checking type

Required correction:
  check exception type or status code before retrying

Revise the code before calling the tool again.
```

Both layers share the same motif store. A motif learned mid-execution is immediately available for pre-prompt injection on the next task.

---

## False positive prevention

Both layers use the same two-stage design: LLM judge for recall, deterministic gate for precision.

The gate requires:
- Confidence ≥ threshold (0.80 for execution-time, 0.70 for pre-prompt)
- Both quotes non-empty and not vague (`"task implies"`, `"likely"`, `"might"`, etc.)
- `requirement_quote` is a verbatim substring of the actual task text (or ≥70% word overlap)
- `violation_quote` is a verbatim substring of the actual proposed code (or ≥70% word overlap)

**Cross-domain contamination is structurally impossible.** A retry motif requires quoting the phrase "retry" from the task text — that phrase will not appear in a sort task. No similarity threshold to tune; the predicate either holds or it does not.

Result: **0% false positives on 16 near-miss tasks** across 8 failure families in the cold-start benchmark.

---

## Quick start

```bash
pip install trace-use
```

Works with **Anthropic** or **OpenAI** — set whichever key you have:

```bash
export ANTHROPIC_API_KEY=sk-ant-...   # uses claude-haiku-4-5
# or
export OPENAI_API_KEY=sk-proj-...     # uses gpt-4o-mini (agent) + gpt-4o (judge)
```

`build_embedder()` prefers OpenAI embeddings when `OPENAI_API_KEY` is set (avoids loading a local model). Falls back to `sentence-transformers` (free, no key needed) otherwise.

### Minimal setup

```python
from trace_use import (
    BrainAgent, PersistentMotifStore, TrajectoryDetector,
    BrainConfig, build_embedder, tool_agent,
)

embedder = build_embedder()
store    = PersistentMotifStore(embedder)           # loads ~/.trace_use/motifs.json
detector = TrajectoryDetector(store, embedder)      # pre-task injection
brain    = BrainAgent(embedder, motif_store=store)  # mid-execution interception

agent         = tool_agent(["python_exec"], max_turns=8)
agent.monitor = brain

for i, task in enumerate(tasks):
    brain.set_task(i, task=task["prompt"])
    brain.reset()

    # Layer 1: inject known pitfalls before the LLM starts
    enriched_prompt, matches = detector.inject(task["prompt"])
    if matches:
        print(f"⚠️  {len(matches)} pitfall(s) injected")

    # Layer 2: brain fires mid-execution via agent.monitor
    trace, tokens = agent(enriched_prompt)
    passed = run_checks(trace)

    # Always store the first-attempt trace with the first-attempt label
    brain.store(trace, int(passed), metadata=failure_reason_if_failed)
```

### Interactive terminal demo

```bash
python demo_session.py           # inspect mode: type tasks, see what warnings fire
python demo_session.py --run     # run mode: actually executes tasks with the agent
python demo_session.py --show    # list all stored motifs
python demo_session.py --clear   # wipe motif store
python demo_session.py --store ./my_project.json   # project-specific store
```

---

## API reference

### `TrajectoryDetector`

| Method | Description |
|---|---|
| `detector.check(task)` | Returns `list[MotifMatch]` — relevant motifs with grounded evidence |
| `detector.inject(task)` | Returns `(enriched_prompt, matches)` — prepends KNOWN PITFALLS block if matches exist |

### `BrainAgent`

| Method / property | Description |
|---|---|
| `brain.set_task(idx, task="")` | Register task index and description before each task |
| `brain.reset()` | Clear reasoning buffer and counters before each task |
| `brain.push(text)` | Accumulate reasoning chunk — called automatically via `agent.monitor` |
| `brain.before_tool_call(name, input_dict)` | Pre-execution hook — returns STOP message or `None` |
| `brain.on_tool_call(name, input_dict, result)` | Post-execution stall detection |
| `brain.store(trace, label, metadata="")` | Store result; extracts motif on `label=0` (failure) |
| `brain.n_stored` | Number of learned motifs |
| `brain.last_fire` | Dict with motif id, confidence, and verbatim quotes from the most recent fire |

### `PersistentMotifStore`

```python
store = PersistentMotifStore(embedder)                     # default: ~/.trace_use/motifs.json
store = PersistentMotifStore(embedder, path="./proj.json") # project-specific
store.clear()                                              # wipe all motifs
store.count                                                # number of stored motifs
store.motifs                                               # list[FailureMotif]
```

### Configuration

All tunable constants live in `BrainConfig` / `DetectorConfig`. The `provider` field selects the LLM backend:

```python
from trace_use import BrainConfig, DetectorConfig, BrainAgent, TrajectoryDetector

# OpenAI backend — gpt-4o for judge (verbatim-quote compliance), gpt-4o-mini for agent
brain_cfg = BrainConfig(
    provider      = "openai",           # "anthropic" (default) or "openai"
    judge_model   = "gpt-4o",           # stronger model for judge accuracy
    extract_model = "gpt-4o",
    judge_threshold  = 0.80,            # min confidence to fire
    max_interventions = 2,              # max fires per task
    exec_tool_name   = "python_exec",   # tool name to intercept
)
brain = BrainAgent(embedder, config=brain_cfg)

# Anthropic backend
brain_cfg = BrainConfig(
    provider    = "anthropic",
    judge_model = "claude-haiku-4-5-20251001",
)

det_cfg = DetectorConfig(
    provider        = "openai",
    model           = "gpt-4o",
    judge_threshold = 0.70,
    retrieval_top_k = 5,
    storage_path    = "./project_motifs.json",
)
detector = TrajectoryDetector(store, embedder, config=det_cfg)
```

### Storage invariant

Always store the **first-attempt trace** with the **first-attempt label** — even when a retry fires and recovers a failed task. Storing retry traces conflates recovery patterns with failure patterns.

---

## Results

### Cold-start learning — 56 tasks, 8 failure families (`eval_dev_learning`)

The core benchmark. 56 tasks across 8 programming families (nested key access, shared state, API key mapping, validation, off-by-one, unit scaling, secondary sort, retry classification). Each family has 1 discovery task (brain cold-starts with no motifs), 4 recurrence tasks (brain fires if a motif was learned), and 2 near-miss tasks (same domain, correct code — brain must stay silent).

Run: `gpt-4o-mini` agent, `gpt-4o` judge, OpenAI embeddings.

| Metric | Value |
|---|---|
| Overall pass rate | **82.1%** (46/56) |
| Tasks saved by brain fires | **4** (retry_on_type, safe_request, merge_response_lists, rank_leaderboard) |
| False positive rate on 16 near-miss tasks | **0%** |
| Motif generalization | retry_request → retry_on_type + safe_request (different code, same logical error) |

**What a fire looks like in practice:**

```
[BRAIN JUDGE RAW] motif='unconditional_retry_on_failure' applies=True conf=1.00
  req='Only retry exceptions listed in retry_on.'
  viol='except retry_on as e:'
[BRAIN FIRE] motif: unconditional_retry_on_failure, conf: 1.00
[16/56] retry_on_type   retry_classification   | PASS [FIRED] | 13.4s
```

The agent wrote `except retry_on as e:` — iterating the list as a catch-all instead of checking type membership. The brain caught this, injected a STOP, the agent corrected the code, and the task passed.

**Where it didn't fire:**

The api_key and nested_key families had recurring failures that the brain missed. Root cause: embedding similarity between abstract motif descriptions and concrete task prompts was below the retrieval threshold (~0.05–0.09 cosine similarity), so those motifs never reached the judge. This is an active limitation — the pre-filter is too coarse for short, concrete task descriptions.

### Portfolio Risk Analyzer — 15 sequential tasks (`eval_project`)

Task 3 (rolling statistics) failed: the agent computed `returns.rolling(window).mean().std()` instead of `returns.rolling(window).std()`. Wrong volatility at Task 3 would have propagated silently into covariance (Task 4), Sharpe ratio (Task 10), and the final risk report (Task 15). Brain caught it before execution.

### Where brain fires don't help

`eval_extensive` — 5 fires, 0 tasks fixed. When a task fails because the entire algorithm is wrong (bitmask TSP that needs DP from scratch), motif-based feedback cannot recover it. The brain's value is highest when the error is localized — a boundary condition, a missing type check, a swallowed exception — not when the approach itself needs replacing.

---

## Repo layout

| Path | Role |
|---|---|
| `trace_use/brain.py` | `BrainAgent`, `MotifStore`, `FailureMotif` — mid-execution motif detection |
| `trace_use/trajectory.py` | `TrajectoryDetector`, `MotifMatch` — pre-task pre-prompt injection |
| `trace_use/motif_store.py` | `PersistentMotifStore` — JSON-backed cross-session persistence |
| `trace_use/config.py` | `BrainConfig`, `DetectorConfig` — all tunable constants |
| `trace_use/agents.py` | `tool_agent`, `haiku`, `opus`, `build_embedder`, `_llm_call` |
| `demo_session.py` | Interactive TUI: inspect trajectory detection, teach the system |
| `eval/eval_dev_learning.py` | 56-task cold-start learning benchmark (supports Anthropic + OpenAI) |
| `eval/eval_project.py` | 15-task portfolio risk analyzer session |
| `eval/eval_real_world.py` | 30 hard tasks |
| `eval/results/` | JSON run logs |
| `tests/test_brain.py` | BrainAgent unit tests (offline, stubbed) |
| `tests/test_trajectory.py` | TrajectoryDetector + developer week simulation (offline, stubbed) |

---

## Limitations

- **Cannot prevent the first occurrence.** The brain learns from failure — it cannot warn about something it has never seen. Seed the store manually with `:seed` in `demo_session.py` for known failure modes.
- **Embedding retrieval is coarse.** Cosine similarity between abstract motif descriptions and short task prompts can be low (~0.05–0.10 with OpenAI embeddings). Motifs with similarity below `retrieval_min_sim` never reach the judge. Effective when task text and motif description share vocabulary; weaker for terse prompts.
- **Judge model matters for quote compliance.** Smaller models (gpt-4o-mini, Haiku) paraphrase instead of copy-pasting verbatim quotes — the grounding check rejects these. Use a stronger model (gpt-4o, claude-sonnet) for the judge when accuracy matters.
- **Most impactful in the 15–40% failure band.** Above ~90% pass rate, fires are rare. Below ~60%, the model likely needs a fundamentally different approach rather than mid-turn correction.
- **Motif quality depends on extraction.** If the extraction LLM produces a motif with task-specific field names in `required_condition`, it will fail to generalize to surface-different recurrences.

---

## Negative results

- **kNN trajectory scoring produced false positives.** Embedding-based `p_fail` scores (earlier versions) caused cross-domain contamination. Removed entirely. The verbatim-quote grounding requirement eliminates this class of error.
- **`eval_extensive` fires didn't help.** 5 fires, 0 tasks fixed. Motif correction works for localized logical errors, not wrong algorithmic approaches.
- **GSM8K is too easy.** Models solve grade-school math at >95% — nothing to learn from.
- **Intervention is failure-rate-dependent.** When pass rate is above 90%, the store fills slowly and motifs stay sparse. The system adds value most when there is a recurring failure class to learn from.
