Metadata-Version: 2.3
Name: anchor-eval
Version: 0.3.1
Summary: Retrieval evaluation harness for technical corpora — span-anchored ground truth, archetype-conditioned diagnosis.
Author: Prasant Mishra
Author-email: Prasant Mishra <mishraprasant73@gmail.com>
License: BSL-1.1
Requires-Dist: pydantic>=2.7
Requires-Dist: typer>=0.12
Requires-Dist: rich>=13.7
Requires-Dist: pynacl>=1.5
Requires-Dist: opensearch-py>=2.4
Requires-Dist: jsonpath-ng>=1.8.0
Requires-Dist: tree-sitter>=0.22 ; extra == 'ast'
Requires-Dist: tree-sitter-python>=0.23 ; extra == 'ast'
Requires-Dist: tree-sitter-typescript>=0.23 ; extra == 'ast'
Requires-Dist: tree-sitter-go>=0.23 ; extra == 'ast'
Requires-Dist: tree-sitter-rust>=0.23 ; extra == 'ast'
Requires-Dist: pypdf>=3.0 ; extra == 'docs'
Requires-Dist: python-docx>=1.0 ; extra == 'docs'
Requires-Dist: pymupdf>=1.24 ; extra == 'docs'
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/pmishra73/anchor-eval
Project-URL: Documentation, https://github.com/pmishra73/anchor-eval/tree/main/docs
Project-URL: Repository, https://github.com/pmishra73/anchor-eval
Project-URL: Bug Tracker, https://github.com/pmishra73/anchor-eval/issues
Project-URL: License, https://github.com/pmishra73/anchor-eval/blob/main/LICENSE
Project-URL: Licensing, https://github.com/pmishra73/anchor-eval/blob/main/docs/reference/licensing.md
Project-URL: Changelog, https://github.com/pmishra73/anchor-eval/blob/main/CHANGELOG.md
Provides-Extra: ast
Provides-Extra: docs
Description-Content-Type: text/markdown

# anchor-eval

**Span-anchored retrieval evaluation for any text corpus.**

anchor-eval measures how well your retrieval pipeline (chunking, embedding, reranking) finds
the *exact source spans* that answer questions about your corpus — source code, documentation,
legal contracts, support tickets, or any collection of text files.
Unlike chunk-overlap metrics, span-anchored evaluation catches regressions that look fine
in aggregate but silently break specific retrieval patterns.

## Quickstart

```bash
pip install anchor-eval
# or: uv add anchor-eval
# For PDF and DOCX support:       pip install 'anchor-eval[docs]'
# For AST drift detection:        pip install 'anchor-eval[ast]'

# Scaffold a config, then generate and run CI
anchor init --domain code
anchor generate --corpus ./my-repo --domain code --output question_set.json
anchor ci --question-set question_set.json --corpus ./my-repo --baseline anchor-baseline.json

# Document corpus (Markdown, HTML, plaintext, RST)
anchor generate --corpus ./docs --domain docs --output doc_qs.json
anchor score doc_qs.json --corpus ./docs --chunking-strategy document_structure

# Try locally with Ollama (no API key required)
anchor generate --corpus ./docs --domain docs --llm-provider ollama --model llama3.2:3b

# Try with no setup at all
anchor demo
anchor demo --domain docs
```

`--domain` selects the correct pack automatically. `--pack` overrides it when you want a custom pack.

See [docs/guides/quickstart.md](docs/guides/quickstart.md) for the full walkthrough.

## LLM providers

`anchor generate` supports three providers via `--llm-provider`:

| Provider | Flag | Key required | Notes |
|---|---|---|---|
| `fake` | `--llm-provider fake` | No | Deterministic, instant. Good for CI smoke tests. |
| `openai` | `--llm-provider openai` | `OPENAI_API_KEY` | Default model: `gpt-4o-mini`. Override with `--model`. |
| `ollama` | `--llm-provider ollama` | No | Runs against a local Ollama server. Default model: `qwen2.5-coder:7b`. Override with `--model` and `--llm-base-url`. |

```bash
# Ollama on a remote host
anchor generate --corpus ./docs --domain docs \
  --llm-provider ollama --model llama3.2:3b \
  --llm-base-url http://gpu-host:11434/v1
```

## The problem, explained from scratch

**RAG in one sentence:** Split documents into chunks → embed them → when a user asks a question, find the most similar chunks → give those chunks to an LLM.

The "documents" can be anything — PDFs, web pages, Slack messages, or code files. A `.py` file is a document. A `.go` file is a document. A code repository is just a folder of text files. The pipeline is identical.

**Who builds RAG over text?**

Almost everyone with a knowledge base. A few common shapes:

- **Code corpora** — "Ask our codebase anything": "Where does authentication happen?", "Which function validates JWT tokens?", "What config controls the timeout?" Developer assistants (Cursor, Copilot workspace) and onboarding bots fall here too.
- **Documentation & wikis** — internal runbooks, product docs, API references. Users ask in natural language; retrieval has to find the right section.
- **Legal & compliance** — contracts, policies, regulations. Questions have exact answers buried in specific clauses; a wrong retrieval is a liability.
- **Support & ticketing** — historical tickets, knowledge-base articles. Retrieval drives deflection; a missed span means a human handles it instead.

The retriever's job in all of these: given a natural-language question, find the correct chunk of text that answers it.

**The evaluation problem**

Say you want to test if your retriever is working. The obvious approach:

1. Take a chunk — say, lines 45–90 of `auth.py`
2. Generate a question from it: "How does this codebase validate API keys?"
3. Run retrieval and check: did you get that chunk back?

This works. Until you change your chunking strategy.

If you go from 512-token chunks to 1024-token chunks, `lines 45–90` no longer exists as a chunk — it got merged into `lines 1–120`. Your benchmark just broke. Every question is now tied to a chunk that doesn't exist. You either throw away the benchmark or you never experiment with chunking at all.

Most teams never change chunking because they can't measure the impact. They're flying blind.

**What anchor-eval does differently**

Instead of anchoring a question to a chunk ID, it anchors it to a character span in the *raw source file*:

```json
{
  "question": "How does this codebase validate API keys?",
  "anchor": { "file": "auth.py", "char_start": 1820, "char_end": 2140 }
}
```

`auth.py` characters 1820–2140 exist forever, regardless of how you chunk. When you run retrieval, a chunk is a "hit" if it covers that span. Change your chunking from 512 to 1024 tokens: the span is still there, the check still works, the benchmark still runs.

You can now run two configs side by side and get a real score for each. You know which one is better and *why*.

**The diagnosis**

Questions are tagged by failure type — `cross_file_causality`, `lexical_collision`, `clause_cross_reference`, etc. When one archetype scores 33% and another scores 90%, that pattern names the broken component. anchor-eval says "your chunker is splitting conditional clauses from their consequences" rather than "retrieval is 61% overall." It then proposes the targeted A/Bs that would confirm the cause.

## What are spans?

A *span* is a `(doc_id, char_start, char_end)` triple that points to an exact range of text in a
source document. Question answers in anchor-eval are defined as one or more anchor spans —
the minimal text ranges that contain the answer. Hit-rate@k is computed by checking whether
the retriever's top-k chunks cover those spans at a configurable IOU threshold.

Document spans also carry a `section_id` (e.g. `"doc#section/page:3/p:2"`) for human-readable
location display in reports.

## Commands

| Command | Description |
|---|---|
| `anchor init` | Scaffold an `anchor.json` config with domain-appropriate defaults. |
| `anchor generate` | Generate a graded QuestionSet from a corpus. Requires a license. |
| `anchor score` | Score a QuestionSet against a corpus; print hit-rate@k by archetype. |
| `anchor run` | Run a grid of retrieval configs, checkpointed to SQLite. Safe to interrupt. |
| `anchor resume` | Resume an interrupted `anchor run` from its checkpoint store. |
| `anchor ci` | CI gate: compare current scores against a committed baseline. Exits 0/1/2. |
| `anchor demo` | Run a 5-question demo against a built-in micro corpus. No license required. |
| `anchor license install` | Install a license token to `~/.config/anchor/license.token`. |
| `anchor license verify` | Verify the installed license token and print its status. |

### Grid runs (`anchor run` / `anchor resume`)

`anchor run` executes a matrix of retrieval configs over a QuestionSet and writes a `RunReport`. Each config is evaluated independently so you can compare chunking strategies, retrieval modes, and k-values side by side.

```bash
# Single default config (BM25, fixed-token chunking)
anchor run --question-set qs.json --corpus ./my-repo --output report.json

# Custom grid from a JSON config file
anchor run --question-set qs.json --corpus . --grid-config grid.json --output report.json

# Estimate cost without running
anchor run --question-set qs.json --corpus . --estimate-only

# Resume after Ctrl-C
anchor resume --question-set qs.json --corpus . --store run.db --output report.json
```

The checkpoint store (`run.db`) is a WAL-mode SQLite file. Each RunUnit (question × config) is claimed and completed atomically, so interrupted runs restart from exactly where they left off.

### Large-document handling

anchor-eval handles large documents — books, long legal contracts, multi-page PDFs, enormous Markdown files — without dropping coverage or blowing up LLM context budgets.

**The problem:** a 100-page PDF may have one giant "section" spanning the entire document. Previous versions either kept it as a single span (too large for any LLM prompt) or silently skipped it (losing all questions from that content). Neither is acceptable.

**What anchor-eval does:** when a naturally-selected region exceeds `MAX_SPAN_CHARS`, the extractor splits it into overlapping sliding-window sub-spans instead of emitting a single oversized span or dropping it. Each sub-span:

- Is at most `MAX_SPAN_CHARS` characters (default 2 000)
- Overlaps the previous by `SPAN_WINDOW_OVERLAP_CHARS` characters (default 200, ~10 %) so content at boundaries is never missed
- Snaps its end to the nearest word boundary so spans never cut mid-token
- Must be at least `MIN_SPAN_CHARS` characters to be emitted — the minimum-size contract is enforced per sub-window, not just for the region as a whole

This applies to all prose and document extractors: Markdown sections, HTML blocks, PDF paragraphs, plaintext/RST/DOCX paragraphs.

To prevent cost explosion on very large documents (after sliding-window expansion, a 200k-char document could produce hundreds of candidate spans), `SpanSelector` caps output at `MAX_SPANS_PER_DOC = 20` spans per archetype per document. The cap uses a deterministic, per-`(doc, archetype)` random sample so each archetype gets an independent selection and coverage is spread across the document rather than biased to its head.

**Tunable defaults** (override in Python or via future CLI flags):

| Constant | Default | What it controls |
|---|---|---|
| `MAX_SPAN_CHARS` | `2000` | Maximum chars per anchor span (window size for large-doc splits) |
| `SPAN_WINDOW_OVERLAP_CHARS` | `200` | Overlap between consecutive sliding-window sub-spans |
| `MAX_SPANS_PER_DOC` | `20` | Max candidate spans per document per archetype after expansion |
| `MIN_SPAN_CHARS` | `30` | Minimum chars for any span or sub-window to be emitted |

```python
import anchor.defaults as d
d.MAX_SPAN_CHARS = 3000           # longer windows for dense legal prose
d.SPAN_WINDOW_OVERLAP_CHARS = 300 # more overlap for tightly-coupled clauses
d.MAX_SPANS_PER_DOC = 30          # more coverage on large docs, higher cost
```

### AST spans and drift detection (`--ast`)

Pass `--ast` to `anchor generate` to enrich spans with tree-sitter AST node IDs. This enables structural drift detection: if a function is renamed or moved, `anchor drift` detects it and flags the affected questions before a CI run wastes time on stale anchors.

```bash
# Requires: pip install 'anchor-eval[ast]'  and a 'generate:ast' license entitlement
anchor generate --corpus ./my-repo --domain code --ast --output qs.json
```

AST enrichment is off by default because it adds tree-sitter parsing overhead and requires the `ast` optional dependency.

## Key concepts

- **QuestionSet**: A committed JSON file of verified questions with anchor spans and difficulty scores.
- **Archetype**: A failure pattern category tagged on each question. Regressions appear per-archetype, not just in aggregate.
- **Pack**: A JSON bundle of archetype definitions and generator prompts for a specific domain.
- **`anchor ci`**: The CI gate. Compares current scores against a committed baseline JSON, exits 1 on per-archetype regression, exits 2 on corpus drift.
- **`anchor run`**: A checkpoint-based grid runner. Evaluates multiple retrieval configs in one pass and writes a structured RunReport.
- **RunStore**: SQLite-backed checkpoint store. Each unit (question × config) is claimed atomically so concurrent or resumed runs are safe.

## Built-in packs

| Pack | Domain | Archetypes |
|---|---|---|
| `code-oss-v0` | Python, TS, Go, Rust | `identifier_free_intent`, `cross_file_causality`, `negative_existence`, `shadow_identifier`, `lexical_collision`, `parameter_level` |
| `jvm-oss-v0` | Java, Kotlin | `identifier_free_intent`, `cross_file_causality`, `parameter_level`, `overload_disambiguation`, `annotation_sensitivity`, `generic_type_boundary` |
| `systems-oss-v0` | Go, Rust | `identifier_free_intent`, `cross_file_causality`, `ownership_borrow_context`, `unsafe_block_scope`, `trait_impl_dispatch`, `cgo_ffi_boundary` |
| `docs-general-v0` | Markdown, HTML, RST, plaintext | `section_cross_reference`, `implicit_negative`, `term_definition_lookup`, `conditional_answer`, `table_cell_lookup`, `procedural_step` |
| `docs-legal-v0` | Legal contracts, regulations | `clause_cross_reference`, `effective_date_shadowed`, `defined_term_collision`, `implicit_obligation`, `negative_obligation`, `jurisdiction_qualifier` |
| `docs-support-v0` | Support tickets, runbooks | `symptom_cause_link`, `workaround_vs_fix`, `version_specific_answer`, `escalation_path`, `product_name_alias` |

Custom packs can be written for any domain — define archetypes, write prompts with `{text_excerpt}` / `{archetype_description}` / `{output_format}`, and point `--pack` at the directory.

## `anchor-corpus.json` manifest

Drop an `anchor-corpus.json` at the root of any corpus directory to configure domain defaults:

```json
{
  "domain": "legal",
  "version": "2024-q4",
  "description": "ACME Corp master service agreement corpus, 2024 edition"
}
```

The loader merges `domain`, `version`, and `description` into every `SourceDocument.metadata` and activates domain-specific chunking (e.g. numbered-section splitting for `"domain": "legal"`).

> **Note (v0.2.2):** only the three keys above are forwarded into document metadata. Custom fields are ignored to prevent unintended data leakage.

## Supported file types

| Extension | Content type | Notes |
|---|---|---|
| `.py` | `code_python` | |
| `.ts`, `.tsx` | `code_typescript` | |
| `.js`, `.jsx`, `.mjs` | `code_javascript` | |
| `.java` | `code_java` | |
| `.kt`, `.kts` | `code_kotlin` | |
| `.cs` | `code_csharp` | |
| `.go` | `code_golang` | |
| `.rs` | `code_rust` | |
| `.md` | `docs_markdown` | |
| `.txt`, `.log` | `docs_plaintext` | |
| `.html`, `.htm` | `docs_html` | Tags stripped; plain text stored |
| `.rst` | `docs_rst` | |
| `.pdf` | `docs_pdf` | Requires `anchor-eval[docs]` |
| `.docx` | `docs_docx` | Requires `anchor-eval[docs]` |
| `.yaml`, `.yml`, `.json` | `docs_openapi` | |

## Changelog

### v0.3.1

Large-document handling and generation correctness:

- **Sliding-window span generation** — prose and document extractors (Markdown, HTML, PDF, plaintext/RST/DOCX) now split sections that exceed `MAX_SPAN_CHARS` into overlapping sub-spans instead of emitting a single oversized span or dropping the content. No coverage is lost on large documents regardless of how they are structured.
- **`SPAN_WINDOW_OVERLAP_CHARS`** (default 200) — configurable overlap between consecutive sub-spans so content at window boundaries is never missed.
- **`MAX_SPANS_PER_DOC`** (default 20) — per-`(document, archetype)` cap on candidate spans after sliding-window expansion. Uses a deterministic, archetype-seeded random sample so each archetype gets an independent selection. Without this, a single 100k-char document could produce hundreds of candidates and multiply LLM call cost by 10×.
- **Anti-leakage gate fix** — the Jaccard leakage check now compares against the same span text the LLM was shown (capped at `MAX_PROMPT_SPAN_CHARS`). Previously, for spans over 1 500 chars, tail tokens the LLM never saw inflated the Jaccard denominator and allowed genuinely leaking questions to pass the gate.
- **Anchor integrity fix** — `MAX_PROMPT_SPAN_CHARS` is now set equal to `MAX_SPAN_CHARS` so the anchor recorded in the question set (`char_start`/`char_end`) always covers exactly what the LLM saw when generating the question. A gap between these two values would produce questions that are unanswerable from their own anchors.
- **Sub-window minimum-size fix** — `_sliding_window_spans` now enforces the caller's `min_chars` per sub-window. Previously, the last sub-window could be shorter than `_MIN_HTML_BLOCK_CHARS` (300) because the guard only applied to the overall region, not to individual windows produced by the splitter.

### v0.3.0

New features:
- `anchor corpus readiness` — corpus health scanner (parse failures, near-duplicates, coverage gaps, table anomalies)
- `anchor run --trajectories` — score agentic tool-call sequences alongside span recall
- `anchor merge-sets` — merge question sets from multiple packs or corpus subdirectories
- `anchor drift query` — vocabulary drift detection (BM25 zero-result check)
- `anchor drift doc` — document section drift detection (section_id re-anchoring check)
- `docs-agentic-v0` pack with 5 trajectory archetypes (tool_call_abstention, tool_call_hallucination, tool_selection_error, parameter_grounding, loop_nontermination)
- Dual-ground-truth question schema: questions can carry both `anchors` (span) and `expected_trajectory` (agentic) — at least one required
- `schema_version "1.1"` auto-set by `anchor generate` when any question has `expected_trajectory`
- `RunReport.latency_metrics`: P50/P95/P99 latency, quality-per-second, quality-per-GPU-hour
- `Archetype.trajectory_archetype` flag; `GenerationPipeline` now synthesizes hybrid questions for trajectory archetypes

### v0.2.2

Security:
- `anchor generate --ast` now uses full ed25519 verification for the `generate:ast` entitlement — previously the second gate fell into placeholder-key mode and accepted any non-empty token
- License server: email plain-text body strips newlines from the customer org name, preventing RFC 2822 header injection
- License server: `ANCHOR_SIGNING_KEY_HEX` is validated to be exactly 64 hex characters before being passed to nacl

Performance:
- Dense retrieval (`_cosine_top_k`) uses `numpy.dot` when available — ~300–1000× faster than the previous Python loop for real embedding dimensions (768–3072)
- `UniquenessVerifier` default `top_k` raised from 5 → 20, catching ambiguous questions in larger corpora that were previously missed

Correctness:
- `EmbeddingCache.get_or_compute` is now thread-safe (double-checked lock pattern)
- `engine_version` in generated `QuestionSet` files now reflects the actual installed package version, fixing false engine-mismatch exits from `anchor ci`
- OpenSearch `_execute_search` skips malformed hits (missing `_source` fields) with a warning instead of crashing the scoring run
- PDF and DOCX extraction failures (corrupt files, bad ZIP, etc.) are now logged and skipped rather than aborting the entire corpus load
- `LocalTelemetryEmitter` handles read-only filesystems gracefully; telemetry file rotates at 10 MB

Configuration:
- `OpenSearchIndexBackend` accepts `http_auth`, `use_ssl`, and `verify_certs` constructor parameters, with automatic fallback to `OPENSEARCH_USERNAME`/`OPENSEARCH_PASSWORD` environment variables

**Breaking change — `anchor-corpus.json` metadata forwarding:**
Custom string fields in `anchor-corpus.json` beyond `domain`, `version`, and `description` are no longer forwarded into `SourceDocument.metadata`. If you relied on a custom field (e.g. `"project"`) appearing in document metadata, add it under one of the three allowlisted keys or access it from the manifest directly.

### v0.2.1

- Ollama LLM provider (`--llm-provider ollama`)
- Tree-sitter AST adapters for Python, TypeScript, Go, Rust (`--ast`)
- PDF and DOCX corpus support (`anchor-eval[docs]`)
- Production ed25519 license key deployed; Svix replay protection via SQLite
- HTML report XSS protection

### v0.2.0

- `anchor run` / `anchor resume` grid runner with SQLite checkpoint store
- `anchor drift` semantic drift detection
- `anchor-corpus.json` manifest
- JVM pack (`jvm-oss-v0`) and Systems pack (`systems-oss-v0`)
- Docs packs: `docs-general-v0`, `docs-legal-v0`, `docs-support-v0`

## Roadmap

### v0.3.1 (shipped)

- **Sliding-window span generation** — large documents no longer lose coverage; oversized sections are split into overlapping sub-spans with configurable window size and overlap
- **Anti-leakage and anchor integrity fixes** — leakage check now uses the same token set the LLM was shown; anchor spans always match the LLM's view of the text

### v0.3.0 (shipped)

- **Corpus readiness diagnostics** — `anchor corpus readiness` scans for parse failures, near-duplicate documents, coverage gaps, and DOCX/PDF table anomalies before generation
- **Efficiency metrics** — `RunReport` now includes `latency_metrics` (P50/P95/P99 latency, quality-per-second, quality-per-GPU-hour)
- **Agentic / trajectory evaluation** — `docs-agentic-v0` pack with 5 trajectory archetypes; `anchor run --trajectories` scores tool-call sequences alongside span recall; dual-ground-truth question schema (span anchor + `ExpectedTrajectory`)
- **`anchor merge-sets`** — merge question sets from different corpus subdirectories or packs
- **`anchor drift query`** and **`anchor drift doc`** — vocabulary drift and document section drift detection
- **schema_version 1.1** — automatically set by `anchor generate` when trajectory archetypes produce `ExpectedTrajectory` fields

### v0.4 — Shard-aware generation (planned)

v0.4 introduces corpus partitioning so large repos (50k+ files) can generate question sets
in parallel shards and merge them with a `CrossShardUniquenessFilter`.

**Why it's blocked:** the filter makes a precision tradeoff — it will discard some genuinely
unique questions because BM25 similarity across shards is noisier than within-shard checks.
The acceptable false-positive rate is unknown until validated on a real large-scale corpus.

**Gate:** a public 50k-file corpus benchmark must confirm `CrossShardUniquenessFilter`
precision before this ships.

## License

anchor-eval is licensed under the [Business Source License 1.1](https://github.com/pmishra73/anchor-eval/blob/main/LICENSE).
Scoring, CI, and question sets always work without a license.
A license is required only for generating new question sets.

**Pricing:** $49 / user / month · $149 / user / year — [see licensing details](https://github.com/pmishra73/anchor-eval/blob/main/docs/reference/licensing.md).
