Valtron Summarization Report

Ranking

Scored with no reference summaries. Models sharing a tier are being called equivalent, not merely close.

{% for label in axis_labels.values() %}{% endfor %} {% for row in scores %} {% for name in axis_labels %} {% endfor %} {% endfor %}

Tier	Model	Score	{{ label }}	Docs
{{ row.tier }}	{{ row.model }} {% if row.gated %}gated{% endif %}	{{ "%.4f"\|format(row.score) }}	{{ row.axes[name] }}	{{ row.documents_scored }}

Faithfulness is a gate, not a term. Below {{ "%.0f"|format(parameters.gate * 100) }}% a model scores zero however well it covers the document, so a fluent fabrication can never outrank a dull, correct summary. Coverage and precision are combined as a harmonic mean (β = {{ parameters.beta }}), which is what stops the score being a length proxy: padding a summary raises coverage but costs precision. A checklist, when supplied, enters as its own term at weight {{ parameters.requirement_weight }}. The dash shown for an axis means it was undefined for that model, which is not the same as a zero.

Why each model placed where it did

The score is a blend, so it cannot say what went wrong. These can: a low coverage bar and a low faithfulness bar mean very different problems.

Cost and latency

Split by what incurred it. The shared row is the per-document fact extraction and salience pass every candidate reuses; it is divided evenly between them in the per-model figures, since it belongs to no single one.

What	Calls	Prompt tokens	Completion tokens	Cost
Generating summaries	{{ usage.generation.calls }}	{{ "{:,}".format(usage.generation.prompt_tokens) }}	{{ "{:,}".format(usage.generation.completion_tokens) }}	${{ "%.6f"\|format(usage.generation.cost_usd) }}
Judging each summary	{{ usage.judge_per_candidate.calls }}	{{ "{:,}".format(usage.judge_per_candidate.prompt_tokens) }}	{{ "{:,}".format(usage.judge_per_candidate.completion_tokens) }}	${{ "%.6f"\|format(usage.judge_per_candidate.cost_usd) }}
Shared per-document work	{{ usage.judge_shared.calls }}	{{ "{:,}".format(usage.judge_shared.prompt_tokens) }}	{{ "{:,}".format(usage.judge_shared.completion_tokens) }}	${{ "%.6f"\|format(usage.judge_shared.cost_usd) }}

Total cost by model

Average seconds per document

Per-document detail

What the judge actually decided. The ranking is arguable only if you can see which facts it turned on.

{% for doc in documents %}

{{ doc.id }} ({{ doc.salient_facts|length }} must-convey fact{% if doc.salient_facts|length != 1 %}s{% endif %})

{% if doc.salient_facts %} Facts a good summary must convey

{{ fact }}

{% endif %} {% for label in axis_labels.values() %}{% endfor %} {% for entry in doc.models %} {% for name in axis_labels %} {% endfor %} {% endfor %}

Model	{{ label }}	Salient hits	Cost
{{ entry.model }}	{{ entry.axes[name] }}	{{ entry.hits }}/{{ entry.salient_total }}	${{ "%.6f"\|format(entry.cost) }}

{% for entry in doc.models %} {{ entry.model }} {% if entry.error %}

Failed: {{ entry.error }}

{% else %}

{% endif %} {% if doc.requirements and entry.requirement_verdicts %}

{% for req in doc.requirements %} {% set key = 'r' ~ loop.index0 %} {% if entry.requirement_verdicts.get(key) %}✓{% else %}×{% endif %} {{ req[:60] }}{% if req|length > 60 %}…{% endif %} {% endfor %}

{% endif %} {% endfor %}

{% endfor %}

Prompts

Exactly what each candidate was asked, checklist included.

{% for model, prompt in model_prompts.items() %}

{{ prompt }}