acceptance report
Generated {{ report.generated_at }} by juried {{ report.version }}.
Judge: {{ report.judge.provider }} / {{ report.judge.model }} ({% if report.judge.temperature is none %}temperature not set{% else %}temperature {{ report.judge.temperature }}{% endif %}{% if report.judge.votes > 1 %}, majority of {{ report.judge.votes }} votes per response{% else %}, one verdict per response{% endif %}).
Defaults: {{ report.defaults.runs }} runs per scenario, gate needs {{ report.defaults.required_passes }} of {{ report.defaults.runs }} to pass ({{ report.defaults.misses }} miss{{ 'es' if report.defaults.misses != 1 else '' }} tolerated){% if report.defaults.gate_on == 'corrected' %}, gated on the judge-corrected rate{% endif %}.
{% if report.defaults.early_stop %}Early stop on: a scenario ends once its gate is decided{% if report.summary.early_stopped %}, which saved {{ report.summary.attempts_planned - report.summary.attempts_made }} of {{ report.summary.attempts_planned }} planned attempts across {{ report.summary.scenarios_stopped }} scenario{{ 's' if report.summary.scenarios_stopped != 1 else '' }}{% endif %}.{% else %}Early stop off: every planned attempt was made.{% endif %}
{# Eight tiles always, plus one for replayed responses and one for split verdicts when
they apply. Up to eight sit on one row; more split into two even rows, so no tile is
left orphaned on a row of its own. #}
{% set tiles = 8 + (1 if report.summary.responses_from_cache else 0) + (1 if report.judge.votes > 1 else 0) %}
{% set columns = tiles if tiles <= 8 else (tiles + 1) // 2 %}
{{ report.summary.criteria }}criteria
{{ report.summary.scenarios }}scenarios
{{ report.summary.gates_passed }}gates upheld
{{ report.summary.gates_failed }}gates failed
{{ report.summary.incomplete }}incomplete
{{ report.summary.transport_errors }}transport errors
{{ report.summary.judge_errors }}judge errors
{% if report.summary.responses_from_cache %}
{{ report.summary.responses_from_cache }}responses replayed from cache
{% endif %}
{% if report.judge.votes > 1 %}
{{ report.summary.split_verdicts }}split verdicts
{% endif %}
{% set judge_usage = report.summary.usage.judge %}
{% set target_usage = report.summary.usage.target %}
{% if report.summary.usage.total_estimate_usd is not none %}${{ '%.4f' | format(report.summary.usage.total_estimate_usd) }}{% elif judge_usage.estimated_cost_usd is not none %}${{ '%.4f' | format(judge_usage.estimated_cost_usd) }} + ?{% else %}?{% endif %}estimated spend
judge {% if judge_usage.estimated_cost_usd is none %}?{% else %}${{ '%.4f' | format(judge_usage.estimated_cost_usd) }}{% endif %}: {{ judge_usage.input_tokens }} in / {{ judge_usage.output_tokens }} out, {{ judge_usage.calls }} calls
target {% if target_usage.estimated_cost_usd is none %}?{% else %}${{ '%.4f' | format(target_usage.estimated_cost_usd) }}{% endif %}: {{ target_usage.requests }} requests{% if target_usage.counted %}, {{ target_usage.input_tokens }} in / {{ target_usage.output_tokens }} out{% endif %}
{% if judge_usage.estimated_cost_usd is none and judge_usage.calls %}
No list price is known for {{ report.judge.model }}. Set input_price and output_price under [judge] (US dollars per million tokens) to estimate the judge spend.
{% endif %}
{% if target_usage.estimated_cost_usd is none and target_usage.requests %}
The target's cost is unknown. Set input_price and output_price under [target] with usage_input_path and usage_output_path, or cost_per_request, to estimate it.
{% endif %}
{% if report.summary.responses_from_cache %}
{{ report.summary.responses_from_cache }} of the responses in this report were replayed from the cache rather than sampled from the feature, so their pass rates say nothing about how the feature behaves now. Run without --cache-responses to sample again.
{% endif %}
{% if report.summary.criteria_without_scenarios %}
Criteria with no scenarios: {{ report.summary.criteria_without_scenarios | join(', ') }}.
{% endif %}
{% set rule = namespace(shown=false) %}
{% for criterion in report.criteria %}
{{ criterion.title }} ({{ criterion.id }})
{% if criterion.description %}
{{ criterion.description }}
{% endif %}
{% if not criterion.scenarios %}
No scenarios were run for this criterion.
{% else %}
{% if not rule.shown %}
A scenario is upheld when at least the number of runs under "Gate needs" pass, that is when no more than the tolerated misses fail{% if report.defaults.gate_on == 'corrected' %}, judged on the lower bound of the corrected interval against the rate that count implies{% endif %}. The interval is the Wilson 95% interval on the pass rate; it describes how much the rate could move on another sample{% if report.defaults.gate_on != 'corrected' %} and does not decide the gate{% endif %}. The rate, interval and gate count only attempts that reached a verdict; a scenario with transport or judge errors is incomplete, whatever its rate.{% if report.summary.early_stopped %} A scenario marked "stopped after" ended once its gate was decided: its verdict is sound, but its rate and interval are a bound rather than an estimate, since a lost scenario stops at the moment its failures cross the line.{% endif %}
{% if report.calibration %}
Corrected rates use {{ report.calibration.path }}: {{ report.calibration.provider }} / {{ report.calibration.model }} against {{ report.calibration.cases }} labelled cases on {{ report.calibration.generated_at[:10] }}, accuracy {{ report.calibration.accuracy | percent }}. The correction divides out the judge's false pass and false fail rates; its interval is a bootstrap over the calibration cases and the attempts together.
{% else %}
No calibration report matched this judge, so there are no corrected rates: run juried calibrate with labelled cases to get them.
{% endif %}
{% set rule.shown = true %}
{% endif %}
| Scenario |
Kind |
Runs upheld / judged |
Gate needs |
Rate |
Corrected |
Interval |
Latency |
Gate |
{% for scenario in criterion.scenarios %}
| {{ scenario.name }} |
{{ scenario.kind.replace('_', ' ') }} |
{{ scenario.passes }} / {{ scenario.judged }}{% if scenario.checks_failed %}
{{ scenario.checks_failed }} by checks{% endif %}{% if scenario.transport_errors %}
{{ scenario.transport_errors }} transport{% endif %}{% if scenario.judge_errors %}
{{ scenario.judge_errors }} judge{% endif %}{% if scenario.split_verdicts %}
{{ scenario.split_verdicts }} split{% endif %}{% if scenario.responses_from_cache %}
{{ scenario.responses_from_cache }} replayed{% endif %}{% if scenario.early_stopped %}
stopped after {{ scenario.attempts_made }} of {{ scenario.attempts_planned }}{% endif %} |
{{ scenario.required_passes }} / {{ scenario.attempts_planned }}
{{ scenario.misses }} miss{{ 'es' if scenario.misses != 1 else '' }} |
{{ scenario.pass_rate | percent }} |
{% if scenario.corrected_rate is not none %}{{ scenario.corrected_rate | percent }}
{{ scenario.corrected_interval.lower | percent }} to {{ scenario.corrected_interval.upper | percent }}{% elif scenario.corrected_refused %}judge too weak{% else %}none{% endif %} |
{{ scenario.interval.lower | percent }} to {{ scenario.interval.upper | percent }} |
{% if scenario.latency.measured %}{{ scenario.latency.mean_ms | round | int }} ms
max {{ scenario.latency.max_ms | round | int }} ms{% if scenario.latency.first_token.measured %}
first token {{ scenario.latency.first_token.mean_ms | round | int }} ms{% endif %}{% else %}not measured{% endif %} |
{{ scenario.status }} |
{% endfor %}
{% for scenario in criterion.scenarios %}
{{ scenario.name }} ({{ scenario.id }}) {{ scenario.status }}
{% for turn in scenario.history %}
{{ turn.role }}{{ turn.content }}
{% endfor %}
{% for turn in scenario.turns %}
user{{ turn }}
assistantlive reply, shown per run below
{% endfor %}
user{{ scenario.message }}
expected{{ scenario.expected }}
{% if scenario.checks %}
checks{% for check in scenario.checks %}{% for kind, value in check.items() %}{{ kind }} {{ value }}{% endfor %}{% if not loop.last %}; {% endif %}{% endfor %}
{% endif %}
{{ scenario.failures | length }} failing run{{ 's' if scenario.failures | length != 1 else '' }}
{% for failure in scenario.failures %}
attempt {{ failure.attempt }}{{ failure.outcome.replace('_', ' ') if failure.outcome.endswith('error') else 'failed' }}
{% for turn in failure.transcript %}
{{ turn.role }} (live){{ turn.content }}
{% endfor %}
{% if failure.response is not none %}
response{{ failure.response }}
{% endif %}
{% if failure.checks %}
checks{% for outcome in failure.checks %}{{ outcome.kind }} {{ outcome.value }}: {{ 'passed' if outcome.passed else outcome.reason }}{% if not loop.last %}; {% endif %}{% endfor %}
{% endif %}
{% if failure.by_checks %}
judgenot called, a check decided the attempt
{% else %}
judge{{ failure.reason }}{% if failure.model %} ({{ failure.model }}{% if failure.votes > 1 %}, {{ (failure.agreement * failure.votes) | round | int }} of {{ failure.votes }} votes{% endif %}, {{ failure.judged_at }}){% endif %}
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}