EvidenceBench v0.1 — groundedness, not prediction

Can your model answer questions about real SEC disclosures with verifiable grounding — and abstain when it cannot? Items regenerate weekly from the newest post-cutoff evidence, so no static answer key can have been memorized from a training corpus: the answers did not exist at training time. That property is mechanical, not aspirational — keys ship openly.

Current release: 872 items · item-set hash 1bb76176b8598ad5… · generation spec registered as protocol 6adb3a06f578 · window: trailing 7 days (post-cutoff, regenerated weekly)

How to run (v1 rubric — checkout method)

The released 7.0.0 package does not ship the scorer module (tools.yuclaw_evidencebench lives in the repository, not in the wheel), so score from a pinned checkout. The released script reads the fixed docs/evidencebench/items.jsonl of that checkout and ignores extra arguments — there is no --items flag in 7.0.0. Run from the checkout root:

git clone --branch v7.0.0 --depth 1 https://github.com/YuClawLab/yuclaw-brain.git
cd yuclaw-brain
python3 -m tools.yuclaw_evidencebench score /absolute/path/predictions.json "your-model-name"

predictions.json maps item_id → answer string; the literal abstention string is "cannot verify from the evidence provided". The pinned checkout carries the same item file the leaderboard was scored on.

Rubric v1 limitation (disclosed, preserved): T1 credit is lexical — token overlap ≥ 0.5 with the keyed excerpt or the keyed accession appearing in the answer. Because T1 questions quote the accession, an answer that merely echoes the question scores 1.0 on those items. v1 numbers therefore reproduce a flawed rubric; they do not measure groundedness. v1 items and results stay byte-identical for reproducibility.

Rubric v2 status: candidate, not registered, not available. A structured-fact rule (accession + event type + keyed numeric facts, question tokens excluded, contradictions score 0) is implemented and contract-tested in the next patch release as yuclaw evidencebench score … --items … --rubric v2; it is bounded lexical/numeric matching, not semantic verification. No v2 item set exists: generating one requires a prospective protocol registration (a research-chain append) that has not been adopted. There is no pooled leaderboard across versions.

Leaderboard

The only current row is our own extraction stack scored against its own corpus — a format demonstration, loudly labeled self-evaluation; nothing is claimed by it.

Dataset

Items + keys: items.jsonl (JSONL — one item per line: {item_id, template, question, key}) · meta.json. Under the export rule: derived events, verified excerpts, and keys only — no raw vendor OHLCV is published. Weekly snapshots are tagged in the repository (dataset citability: see CITATION.cff at the repo root and the replication page).