acceptance report
Generated {{ report.generated_at }} by juried {{ report.version }}.
Judge: {{ report.judge.provider }} / {{ report.judge.model }} ({% if report.judge.temperature is none %}temperature not set{% else %}temperature {{ report.judge.temperature }}{% endif %}{% if report.judge.votes > 1 %}, majority of {{ report.judge.votes }} votes per response{% else %}, one verdict per response{% endif %}).
Defaults: {{ report.defaults.runs }} runs per scenario, gate at {{ report.defaults.threshold }} on the lower bound of the Wilson 95% interval{% if report.defaults.required_passes is not none %}, which needs {{ report.defaults.required_passes }} of {{ report.defaults.runs }} runs to pass{% endif %}.
{# Eight tiles always, plus one for replayed responses and one for split verdicts when
they apply. Up to eight sit on one row; more split into two even rows, so no tile is
left orphaned on a row of its own. #}
{% set tiles = 8 + (1 if report.summary.responses_from_cache else 0) + (1 if report.judge.votes > 1 else 0) %}
{% set columns = tiles if tiles <= 8 else (tiles + 1) // 2 %}
{{ report.summary.criteria }}criteria
{{ report.summary.scenarios }}scenarios
{{ report.summary.gates_passed }}gates upheld
{{ report.summary.gates_failed }}gates failed
{{ report.summary.incomplete }}incomplete
{{ report.summary.transport_errors }}transport errors
{{ report.summary.judge_errors }}judge errors
{% if report.summary.responses_from_cache %}
{{ report.summary.responses_from_cache }}responses replayed from cache
{% endif %}
{% if report.judge.votes > 1 %}
{{ report.summary.split_verdicts }}split verdicts
{% endif %}
{% if report.summary.usage.estimated_cost_usd is none %}?{% else %}${{ '%.4f' | format(report.summary.usage.estimated_cost_usd) }}{% endif %}judge spend
{{ report.summary.usage.input_tokens }} in / {{ report.summary.usage.output_tokens }} out, {{ report.summary.usage.calls }} calls
{% if report.summary.usage.estimated_cost_usd is none and report.summary.usage.calls %}
No list price is known for {{ report.judge.model }}. Set input_price and output_price under [judge] (US dollars per million tokens) to estimate the spend.
{% endif %}
{% if report.summary.responses_from_cache %}
{{ report.summary.responses_from_cache }} of the responses in this report were replayed from the cache rather than sampled from the feature, so their pass rates say nothing about how the feature behaves now. Run without --cache-responses to sample again.
{% endif %}
{% if report.summary.criteria_without_scenarios %}
Criteria with no scenarios: {{ report.summary.criteria_without_scenarios | join(', ') }}.
{% endif %}
{% set rule = namespace(shown=false) %}
{% for criterion in report.criteria %}
{{ criterion.title }} ({{ criterion.id }})
{% if criterion.description %}
{{ criterion.description }}
{% endif %}
{% if not criterion.scenarios %}
No scenarios were run for this criterion.
{% else %}
{% if not rule.shown %}
A scenario is upheld when the lower bound of its 95% interval meets the threshold. The threshold is not a pass rate: the "needs" figure under it is the number of runs that must pass. The rate and interval count only attempts that reached a verdict; a scenario with transport or judge errors is incomplete, whatever its rate.
{% set rule.shown = true %}
{% endif %}
| Scenario |
Kind |
Runs upheld / judged |
Rate |
Lower bound |
Threshold |
Upper bound |
Latency |
Gate |
{% for scenario in criterion.scenarios %}
| {{ scenario.name }} |
{{ scenario.kind.replace('_', ' ') }} |
{{ scenario.passes }} / {{ scenario.judged }}{% if scenario.transport_errors %}
{{ scenario.transport_errors }} transport{% endif %}{% if scenario.judge_errors %}
{{ scenario.judge_errors }} judge{% endif %}{% if scenario.split_verdicts %}
{{ scenario.split_verdicts }} split{% endif %}{% if scenario.responses_from_cache %}
{{ scenario.responses_from_cache }} replayed{% endif %} |
{{ scenario.pass_rate | percent }} |
{{ scenario.interval.lower | percent }} |
{{ scenario.threshold | percent }}
{% if scenario.required_passes is none %}unattainable{% else %}needs {{ scenario.required_passes }} / {{ scenario.runs }}{% endif %} |
{{ scenario.interval.upper | percent }} |
{% if scenario.latency.measured %}{{ scenario.latency.mean_ms | round | int }} ms
max {{ scenario.latency.max_ms | round | int }} ms{% else %}not measured{% endif %} |
{{ scenario.status }} |
{% endfor %}
{% for scenario in criterion.scenarios %}
{{ scenario.name }} ({{ scenario.id }}) {{ scenario.status }}
{% for turn in scenario.history %}
{{ turn.role }}{{ turn.content }}
{% endfor %}
{% for turn in scenario.turns %}
user{{ turn }}
assistantlive reply, shown per run below
{% endfor %}
user{{ scenario.message }}
expected{{ scenario.expected }}
{{ scenario.failures | length }} failing run{{ 's' if scenario.failures | length != 1 else '' }}
{% for failure in scenario.failures %}
attempt {{ failure.attempt }}{{ failure.outcome.replace('_', ' ') if failure.outcome.endswith('error') else 'failed' }}
{% for turn in failure.transcript %}
{{ turn.role }} (live){{ turn.content }}
{% endfor %}
{% if failure.response is not none %}
response{{ failure.response }}
{% endif %}
judge{{ failure.reason }}{% if failure.model %} ({{ failure.model }}{% if failure.votes > 1 %}, {{ (failure.agreement * failure.votes) | round | int }} of {{ failure.votes }} votes{% endif %}, {{ failure.judged_at }}){% endif %}
{% endfor %}
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}