Measured, with the tradeoffs visible
Less context.
Check the outcome.
Compare answer scores, estimated total API costs, and supporting evidence. The differences and uncertainty below come from the frozen evaluation records.
Controlled study: isolated questions, repeated answers, and a neural baseline ↗ · SciFact corpus retrieval ↗
Research round 2: 400 new questions ↗ — better evidence retention, mixed answer results, and the measured cost tradeoff.
| Dataset (100 questions) | Pipeline | Answer EM | Answer F1 | API cost / 100 | Support recall |
|---|---|---|---|---|---|
| HotpotQA | All context | 65.00 | 77.97 | $0.034281 | 100.00% |
| HotpotQA | BM25 top 5 | 51.00 | 59.88 | $0.020512 | 78.50% |
| HotpotQA | Jev contextual | 69.00 | 80.08 | $0.025130 | 97.00% |
| MuSiQue-Ans | All context | 52.00 | 64.30 | $0.066016 | 100.00% |
| MuSiQue-Ans | BM25 top 5 | 26.00 | 34.46 | $0.027427 | 58.42% |
| MuSiQue-Ans | Jev contextual | 58.00 | 64.76 | $0.050748 | 87.67% |
Fixed validation-set subsets; dataset-provided candidates; Jev 1.13.0; GPT-5.6 Luna with low reasoning; one response per branch. Policy selected on 40 separate development questions. Not full benchmark or leaderboard results.
HotpotQA: F1 difference +2.11 points, 95% paired bootstrap interval [-2.08, +6.68]; estimated API cost 26.7% lower.
MuSiQue-Ans: F1 difference +0.46 points, 95% paired bootstrap interval [-5.52, +6.63]; estimated API cost 23.1% lower.
Predeclared accuracy-and-cost gate: not met.
Neither query expansion nor a modern neural reranker was evaluated.
Method, sources, and what the costs include
Official answer normalization and token F1, checked against the authors' metric functions on all 760 development/evaluation predictions. Paired bootstrap: 2,000 resamples. Errors: 0; missing cost observations: 0; truncated answers: 0.
Estimated API cost includes Jev and generation with reported cache discounts. Excludes common retrieval, hosting and networking. Rates: Luna $0.20 input/$0.02 cached/$1.20 output per million tokens; Jev $0.042 input/free output.
HotpotQA · MuSiQue (CC BY 4.0) · Luna pricing · Jev pricing
MuSiQue came from the pinned dgslibisey/MuSiQue mirror. Model training overlap is unknown. No modern neural reranker baseline was evaluated. Full protocol, source hashes, per-case records and reproduction commands are in benchmarks/public in the project.