Metadata-Version: 2.4
Name: retrieval-observatory
Version: 0.6.0
Summary: Local-first retrieval reliability: integrate, evaluate, compare under release policy, and inspect candidate lineage
Author-email: Ameya Kiwalkar <akiwalkar@berkeley.edu>
License-Expression: MIT
Project-URL: PyPI, https://pypi.org/project/retrieval-observatory/
Project-URL: Homepage, https://github.com/AmeyaKI/retrieval-observatory
Project-URL: Documentation, https://github.com/AmeyaKI/retrieval-observatory#readme
Project-URL: Repository, https://github.com/AmeyaKI/retrieval-observatory
Project-URL: Issues, https://github.com/AmeyaKI/retrieval-observatory/issues
Project-URL: Changelog, https://github.com/AmeyaKI/retrieval-observatory/releases
Keywords: rag,retrieval,evaluation,diagnostics,observability
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0
Requires-Dist: httpx>=0.27
Requires-Dist: aiosqlite>=0.20
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Requires-Dist: pyyaml>=6.0
Requires-Dist: numpy>=1.26
Provides-Extra: beir
Requires-Dist: datasets>=2.0; extra == "beir"
Requires-Dist: beir>=2.0; extra == "beir"
Provides-Extra: cohere
Requires-Dist: cohere>=5.0; extra == "cohere"
Provides-Extra: hf
Requires-Dist: sentence-transformers>=3.0; extra == "hf"
Requires-Dist: torch; extra == "hf"
Provides-Extra: dense
Requires-Dist: sentence-transformers>=3.0; extra == "dense"
Requires-Dist: faiss-cpu>=1.7; extra == "dense"
Requires-Dist: torch; extra == "dense"
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.2; extra == "langchain"
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.10; extra == "llamaindex"
Provides-Extra: haystack
Requires-Dist: haystack-ai>=2.0; extra == "haystack"
Provides-Extra: dspy
Requires-Dist: dspy-ai>=2.4; extra == "dspy"
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.0.3; extra == "openai-agents"
Provides-Extra: pgvector
Requires-Dist: asyncpg>=0.29; extra == "pgvector"
Requires-Dist: pgvector>=0.3; extra == "pgvector"
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.29; extra == "postgres"
Provides-Extra: dashboard
Requires-Dist: fastapi>=0.111; extra == "dashboard"
Requires-Dist: uvicorn>=0.29; extra == "dashboard"
Requires-Dist: python-multipart>=0.0.9; extra == "dashboard"
Provides-Extra: serve
Requires-Dist: fastapi>=0.111; extra == "serve"
Requires-Dist: uvicorn>=0.29; extra == "serve"
Requires-Dist: python-multipart>=0.0.9; extra == "serve"
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.9; extra == "qdrant"
Provides-Extra: llm-judge
Requires-Dist: google-generativeai>=0.8; extra == "llm-judge"
Requires-Dist: anthropic>=0.28; extra == "llm-judge"
Requires-Dist: openai>=1.0; extra == "llm-judge"
Provides-Extra: demo
Requires-Dist: beir>=2.0; extra == "demo"
Requires-Dist: datasets>=2.0; extra == "demo"
Requires-Dist: rank-bm25>=0.2.2; extra == "demo"
Provides-Extra: classifier
Requires-Dist: scikit-learn>=1.4; extra == "classifier"
Requires-Dist: joblib>=1.3; extra == "classifier"
Provides-Extra: mcp
Requires-Dist: mcp>=1.2; extra == "mcp"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: coverage; extra == "dev"
Requires-Dist: respx>=0.21; extra == "dev"
Requires-Dist: scikit-learn>=1.4; extra == "dev"
Requires-Dist: joblib>=1.3; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Requires-Dist: mcp>=1.2; extra == "dev"
Requires-Dist: fastapi>=0.111; extra == "dev"
Requires-Dist: playwright>=1.45; extra == "dev"
Dynamic: license-file

# retobs

[PyPI](https://pypi.org/project/retrieval-observatory/) · [Case study](results/flagship_demo/CASE_STUDY.md)

**Hosted demo (read-only dashboard):** [retobs-demo.happywater-562fb4f3.westus2.azurecontainerapps.io](https://retobs-demo.happywater-562fb4f3.westus2.azurecontainerapps.io) — the BEIR sweep on Azure Container Apps; see [deployment](docs/deployment.md).

retobs tells you which stage of your retrieval pipeline earned or destroyed your metric, with attribution you can audit.

A metrics dashboard says *recall 0.5 on query 5abccf67* and stops. retobs records every candidate through every operator, so it can say this instead:

```
bm25_lane        gold 1/2   ranks [1]
dense_lane       gold 2/2   ranks [2, 27]
hybrid_fusion    gold 1/2   !! dropped: karen_dotrice
bridge_hop2      gold 2/2   ranks [1, 45]      <- the second hop found it again
route_merge      gold 1/2   !! dropped: karen_dotrice
final_selection  gold 1/2
```

The vector lane found the missing document at rank 27. Fusion, keeping 40 candidates, dropped it. The two-hop expansion found it again at rank 45. The second merge dropped it again. The fix was one number, merge width 40 to 100, and the same tool then proved the fix held: recall@10 +0.0088, 95% CI [+0.0019, +0.0181], n=400, concentrated in the two-hop questions where a wider merge is the only thing that could help. The full story, including a change that passed every metric while making the system worse, is in the [case study](results/flagship_demo/CASE_STUDY.md).

## Install

```bash
pip install "retrieval-observatory[dashboard,mcp]"
```

## See it work first

One command, no arguments, no API keys. It builds a regression story end to end and hands you a dashboard to explore it.

```bash
retobs demo
retobs serve --db .retobs/demo/results.db
```

Everything below is the same workflow pointed at your own code.

## Integrate an existing project

Plan first, review the plan, then apply. Verify reports ready only after it has seen real traces from the instrumented pipeline.

```bash
retobs integrate . --phase plan --output retobs/integration-plan.json
retobs integrate . --phase apply --plan retobs/integration-plan.json
retobs integrate . --phase verify --plan retobs/integration-plan.json
```

Apply refuses unresolved mappings and stale file hashes, lists every changed file, and keeps reversal information. For agents, the same three phases are one MCP tool; see the [agent runbook](docs/integrations/AGENT_QUICKSTART.md).

## Evaluate a callable

```bash
retobs evaluate mypackage.search:retrieve --queries data/queries.jsonl --qrels data/qrels.jsonl --corpus data/corpus.jsonl
```

The returned Run ID feeds `retobs report`, `retobs compare`, and `retobs inspect-query`.

## Gate a release

```bash
retobs compare BASELINE CANDIDATE --db .retobs/results.db --policy retobs/release-policy.yaml --format html --output artifacts/retobs-release.html --fail-on hold-or-block-or-fail
```

The verdict is one of four words. `PASS`: bounded non-inferiority under the declared policy. `HOLD`: valid but inconclusive. `BLOCK`: required evidence is missing or the two runs are not comparable (different corpus, index, or model revision). `FAIL`: a proven regression on a policy-critical metric. Paired bootstrap confidence intervals, seeded, with multiple-comparison correction. See [retrieval release decisions](docs/guides/retrieval-release-decisions.md).

## Attribution you can audit

Two mechanisms produce the per-stage story above, and both are inspectable in the dashboard and through the SDK.

- **Candidate lineage.** Every candidate's rank and score at the input and output of every operator, recorded by the instrumentation rather than inferred afterwards. When an integration cannot supply a field, retobs reports it as unavailable instead of guessing. See the [Candidate Lineage Explorer](docs/guides/candidate-lineage-explorer.md).
- **Counterfactual replay.** For a given operator, retobs re-executes the recorded trace without it and reports the metric delta, labelled by how trustworthy that replay is: exact, observed ablation, or not replayable. See [counterfactual replay](docs/guides/counterfactual-replay.md).

## Investigate locally

```bash
retobs serve --db .retobs/results.db
```

The dashboard binds to `127.0.0.1` by default and is unauthenticated. Put it behind trusted controls before exposing it beyond loopback.

## What retobs records

Evaluation Runs with their manifests, per-query evidence, and operator traces; production traces scoped to a service and pipeline, including candidate transitions when instrumentation provides them; and instrumentation health (sampling, drops, serialization failures, export failures). A recorded field is a contract about what was observed, not a guarantee that every integration can supply it.

## Integration support

First-class: plain Python, HTTP, FastAPI, LangChain, LlamaIndex. Supported examples with narrower guarantees: DSPy, Haystack, OpenAI Agents. See [integration support](docs/INTEGRATIONS.md).

## Privacy and production safety

Queries, candidates, metadata, labels, and traces may be sensitive. Redaction runs before enqueue and persistence; queue capacity, overflow policy, and sampling are explicit configuration. Read [privacy](docs/PRIVACY.md) and [security](SECURITY.md) before production use.

## Documentation

- [Start](docs/START.md)
- [Workflow](docs/WORKFLOW.md)
- [Concepts](docs/CONCEPTS.md)
- [CLI, SDK, and MCP reference](docs/REFERENCE.md)
- [Guides](docs/guides/README.md)
- [Architecture](docs/ARCHITECTURE.md)
- [Known limitations](FUTURE_WORK.md)
- [Releases](https://github.com/AmeyaKI/retrieval-observatory/releases)

License: [MIT](LICENSE).
