Seating Jev (orq classify) as a judge in evaluatorq

1 · Endpoint routing inside run_judge P1

The one shared seam every judge caller goes through (common/judge.py:510). It already picks Responses vs Chat from the catalogue. Jev becomes a third branch keyed on the catalogue's metadata.supports_classify, and only when the client routes through the Orq router. When the catalogue is empty or its fetch failed, a hardcoded set holding typesafe/jev-latest answers instead and warns once, so an outage never sends Jev to chat.

2 · Verdict modes map one-to-one onto Jev question types P1

Nothing about the jury contract changes. Each existing llm_jury mode is one Jev primitive; the derivation of value is code, never a model opinion.

3 · Prompt vs state: what each judge kind sees P1

An LLM judge gets a rendered prompt. Jev gets instructions (the criteria) plus a state object. The state is built from the template's {{…}} placeholders, resolved through the same template engine and keyed by dotted path, with criteria excluded. An explicit state_fields list replaces the placeholder scan.

tone = llm_jury(
    name="tone",
    criteria="Is the reply polite and on-topic?",
    judges=["openai/gpt-5.4-mini", "anthropic/claude-sonnet-4-6", "typesafe/jev-latest"],
    labels={"rude": "hostile or dismissive", "neutral": None, "friendly": "warm and helpful"},
    passing_labels=["neutral", "friendly"],
)
# LLM judges: DEFAULT_TEMPLATE rendered, labels + descriptions in schema and system prompt.
# Jev: instructions=criteria, criteria=labels, state = the template's placeholders minus criteria.

tone = llm_jury(
    name="tone",
    prompt="# Reply\n{{output.response}}\n\n# Conversation\n{{input.all_messages}}\n\nRate the tone.",
    criteria="Is the reply polite and on-topic?",   # both allowed only when a Jev judge is seated
    judges=["openai/gpt-5.4-mini", "typesafe/jev-latest"],
    labels={"rude": "hostile or dismissive", "neutral": None, "friendly": "warm and helpful"},
    state_fields=["output.response"],             # optional: override the placeholder scan
)
# LLM judges render `prompt` exactly as before. Jev never sees the prompt.

helpfulness = llm_jury(
    name="helpfulness",
    criteria="How helpful is the answer?",
    judges=["openai/gpt-5.4-mini", "typesafe/jev-latest"],
    verdict_kind="numeric",
    levels=["does not address the question", "partially addresses it", "fully answers it"],
    threshold=0.7,                     # score_range stays (0, 1) — non-default raises at construction
)
# Jev: score in [0, 2] → value = score / 2. LLM judges: levels go into the value field description.

POST https://my.orq.ai/v3/router/classify      (0.7 s, 2026-09-19)
{"model": "typesafe/jev-latest",
 "state": {"input.all_messages": "...", "output.response": "Paris."},
 "questions": {"verdict": {"type": "choice", "instructions": "Grade the answer",
                           "criteria": {"correct": "fully correct", "partial": "partly", "wrong": "incorrect"}}}}

{"model": "jev-latest",
 "answers": {"verdict": {"type": "choice", "choice": "correct",
                         "probabilities": {"correct": 1, "partial": 0, "wrong": 0}, "confidence": 1}},
 "usage": {"input_tokens": 379, "output_tokens": 68,
           "input_cost": 0.00001592, "output_cost": 0, "total_cost": 0.00001592}}

What changes for an LLM judge on the same panel

The prompted verdict shape stays {explanation, value, abstain}, with one JuryVote per judge. The local EvaluationResult.raw_output["jury"] record now adds optional raw_output on each repetition: the complete validated answer for a classify judge, None for a prompted judge. Three inputs change, all additive.

A panel without a Jev judge and without levels or dict labels produces byte-identical requests to today.

4 · What llm_jury gains, field by field P2

Construction-time checks use the same hardcoded classify set (the catalogue is async, so a constructor cannot consult it): prompt without criteria, numeric without levels, non-default score_range. Run time uses the catalogue first, the set as fallback.

5 · Decisions to validate, most consequential first

These were settled in the grill. Each names the alternative that was rejected, so a disagreement is one line to state.

6 · Out of scope, and gotchas already known

• Red teaming and agent simulation are not adapted in this ticket. They call run_judge with freeform templates and no verdict spec, so they need their own ClassifyQuestion construction later. The seam is ready.
• Presets do not seat Jev. jury_presets.py is untouched; a user lists typesafe/jev-latest in judges by hand.
• Only on the Orq router. Classify is gated on client_routes_through_orq; an injected OpenAI client never sends it.
• Catalogue shape. ModelInfo gains supports_classify from metadata.supports_classify; the entry has no supports_responses_api, so today's routing legitimately falls to chat and 400s.
• Response model id comes back bare ("jev-latest", no typesafe/ prefix), same pattern price_usage already tolerates. Cost is router-priced, so price_usage is a no-op.
• Limits. 64k context; state plus the longest question must fit in 32k tokens. A too-large state is a provider 400, surfaced as JudgeError.API_STATUS.
• Empty choice criteria is a 400 ("criteria must contain at least one option"); construction already requires two or more labels.

7 · Files touched