Metadata-Version: 2.4
Name: munch-bench
Version: 0.2.0
Summary: Retrieval + Inference benchmark for LLM-powered codebase Q&A
Author: J. Gravelle
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/jgravelle/munch-bench
Project-URL: Issues, https://github.com/jgravelle/munch-bench/issues
Keywords: benchmark,llm,retrieval,codebase,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jcodemunch-mcp>=1.40.0
Requires-Dist: openai>=1.0.0
Requires-Dist: anthropic>=0.30.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: jinja2>=3.1
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
Dynamic: license-file

# munch-bench

Retrieval + Inference benchmark for LLM-powered codebase Q&A.

**munch-bench** measures how well different LLMs answer questions about real codebases when given retrieval-augmented context from [jCodeMunch](https://github.com/jgravelle/jcodemunch-mcp).

## What it measures

| Metric | Description |
|--------|-------------|
| **Retrieval P@5 / P@10** | Fraction of top-k retrieved symbols that match ground truth |
| **Retrieval Recall** | Fraction of ground-truth symbols found in retrieved context |
| **LLM Judge Score** | 0–1 accuracy rating by an LLM judge comparing answers to ground truth |
| **Exact Match Rate** | Whether the answer contains key content from ground truth |
| **Wall Time** | End-to-end latency (retrieval + inference) |
| **Cost** | Estimated API cost per run |

## Corpus

110 questions across 11 repos spanning Python, JavaScript, TypeScript, Go, Rust, and Java:

| Repo | Questions | Languages |
|------|-----------|-----------|
| pallets/flask | 11 | Python |
| tiangolo/fastapi | 10 | Python |
| django/django | 10 | Python |
| psf/requests | 10 | Python |
| langchain-ai/langchain | 10 | Python |
| pytorch/pytorch | 10 | Python/C++ |
| expressjs/express | 10 | JavaScript |
| facebook/react | 10 | JavaScript |
| vercel/next.js | 10 | TypeScript |
| vuejs/core | 10 | TypeScript |
| gin-gonic/gin | 10 | Go |
| tokio-rs/axum | 10 | Rust |
| spring-projects/spring-boot | 10 | Java |
| jgravelle/jcodemunch-mcp | 10 | Python |

Questions are categorized by difficulty (easy/medium/hard) and type (api/architecture/debugging/refactoring).

## Quick start

```bash
pip install munch-bench
```

### Prerequisites

1. **Index the repos** you want to benchmark against using jCodeMunch:
   ```bash
   jcodemunch-mcp index pallets/flask
   jcodemunch-mcp index tiangolo/fastapi
   # ... etc
   ```

2. **Set API keys** for your chosen provider:
   ```bash
   export GROQ_API_KEY=gsk_...        # for Groq
   export OPENAI_API_KEY=sk-...       # for OpenAI
   export ANTHROPIC_API_KEY=sk-ant-...  # for Anthropic
   ```

### Run a benchmark

```bash
# Run with Groq (default — fastest + cheapest)
munch-bench run --provider groq

# Run with a specific model
munch-bench run --provider groq --model llama-3.3-70b-versatile

# Run with OpenAI
munch-bench run --provider openai --model gpt-4o-mini

# Run with Anthropic
munch-bench run --provider anthropic --model claude-sonnet-4-6

# Filter to specific repos or difficulty
munch-bench run --provider groq --repo pallets/flask --difficulty hard

# Custom token budget for retrieval
munch-bench run --provider groq --token-budget 16000 -v
```

Results are saved to `results/<provider>_<model>_<date>.json`.

### Compare runs and generate leaderboard

```bash
# Compare multiple runs
munch-bench compare results/*.json -o leaderboard.html

# View corpus statistics
munch-bench corpus-stats
```

### Leaderboard

The leaderboard is a static HTML page with interactive Chart.js visualizations, deployed to GitHub Pages on every push to `main`.

## Architecture

```
munch-bench/
  corpus/              # YAML question files (one per repo)
  src/munch_bench/
    cli.py             # CLI entrypoint: run, compare, corpus-stats
    corpus.py          # YAML corpus loader + filtering
    retrieval.py       # jCodeMunch retrieval wrapper
    inference.py       # Provider dispatch (Groq, OpenAI, Anthropic)
    evaluate.py        # Metrics: P@k, recall, exact match, LLM judge
    runner.py          # Orchestrator with rich progress
    leaderboard.py     # Static HTML + Chart.js generator
  tests/
  results/             # JSON benchmark outputs (gitignored)
```

## Adding questions

Create a YAML file in `corpus/`:

```yaml
repo: owner/name
questions:
  - id: unique-id-001
    question: "How does X work?"
    ground_truth_answer: "X works by..."
    ground_truth_symbols: ["function_name", "ClassName"]
    difficulty: medium  # easy, medium, hard
    category: architecture  # api, architecture, debugging, refactoring
    tags: [optional, tags]
```

## License

Apache 2.0

---

Powered by [jCodeMunch](https://github.com/jgravelle/jcodemunch-mcp) + [Groq](https://groq.com)
