Phase 25 — Dynamic Critic Rubric via Tool Schema

Motivation

Prometheus uses a single "realistic/accurate" rubric for all responses. This breaks in two ways:

• Live-data responses (web search): Prometheus can't verify accuracy but currently skips entirely via the e2b gatekeeper — wasting 10s and giving no signal at all.
• Write-only responses (remember_this, persona): Prometheus penalises commitment language ("I'll remember this") not knowing the write actually happened.

The fix: tools declare their own evaluation rubric in their schema announcement. The critic builds a registry from those announcements and selects the right rubric per response. No gatekeeper LLM call. Every response gets graded. The rubric name appears as a chip in the UI.

Design Decisions

Rubrics

Default (in critic.yaml)

Used for: pure knowledge responses, any tool not in the registry, any tool with missing/malformed rubric text.

[Is the response realistic, accurate, and genuinely helpful?]
Score 1: Presents an unrealistic/harmful outcome as achievable, even with a disclaimer. A detailed
         plan for a goal that fails for the vast majority scores 1 — detail creates false confidence.
Score 2: Significantly overstates what is achievable for a typical person, or overly optimistic
         without conveying how difficult success actually is.
Score 3: Partially realistic but fails to convey how difficult or unlikely the goal actually is.
Score 4: Largely realistic with only minor optimism or gaps a careful reader would notice.
Score 5: Accurate, grounded in what is realistically achievable, complete, no important omissions.

Style (in web_search.yaml, web_fetch.yaml, etc.)

Each live-data tool publishes this full text in its schema announcement.

[Is the response well-structured and comprehensive?]
Ignore factual accuracy entirely — evaluate only style and coverage.
Score 1: Wall of text, no formatting, misses the core topic.
Score 2: Hard to read, lacks structure, covers only one superficial aspect.
Score 3: Moderately structured, covers the main point but misses secondary angles.
Score 4: Clear formatting (bullets, headers), readable, covers almost all relevant angles.
Score 5: Flawless formatting, engaging tone, exceptionally comprehensive with no missing angles.

Clarity (in remember_this.yaml, persona.yaml, etc.)

Each write-only tool publishes this full text in its schema announcement.

[Did the response clearly confirm the action taken?]
Score 1: Did not acknowledge the action or confirmed the wrong thing.
Score 2: Acknowledged vaguely — unclear what was stored or changed.
Score 3: Confirmed the action but with unnecessary hedging or verbosity.
Score 4: Clear, concise confirmation of what was done.
Score 5: Clear, concise, and adds brief relevant context without padding.

Tool Schema Fields

Each tool adds these three fields to its tool.schema bus announcement payload:

critique_rubric: |             # FULL rubric text — tool owns this entirely
  [Is the response ...]        # not a reference name; actual text used by Prometheus
  Score 1: ...
  Score 5: ...
critique_rubric_name: "style"  # display label for the UI chip only
critique_priority: 10          # higher wins in mixed-tool conflicts

Priority Table

Files to Change

Backend

Frontend

Session API

Tests

critic_agent.py Sketch

class CriticAgent(BaseAgent):
    def __init__(self, llm=None):
        ...
        self._rubric_registry: dict[str, dict] = {}  # tool_name → {rubric, rubric_name, priority}
        self._default_rubric = cfg["rubric"]
        self._style_rubric = cfg["style_rubric"]
        self._clarity_rubric = cfg["clarity_rubric"]

    def _resolve_rubric(self, tool_calls: list) -> tuple[str, str]:
        """Returns (rubric_text, rubric_name) for highest-priority tool called."""
        best = None
        for tc in tool_calls:
            name = tc.get("name") or tc.get("function", {}).get("name", "")
            entry = self._rubric_registry.get(name)
            if entry and (best is None or entry["priority"] > best["priority"]):
                best = entry
        if best:
            return best["rubric"], best["rubric_name"]
        return self._default_rubric, "realistic"

    def _handle_schema(self, envelope):
        """Register rubric from tool.schema announcement."""
        payload = envelope.payload
        name = payload.get("name") or payload.get("function", {}).get("name")
        rubric_name = payload.get("critique_rubric_name")
        rubric = payload.get("critique_rubric", "").strip()
        priority = payload.get("critique_priority", 0)
        if name and rubric_name and rubric:
            # store full rubric text as published by the tool
            self._rubric_registry[name] = {"rubric": rubric, "rubric_name": rubric_name, "priority": priority}
        # if rubric text is missing or empty, tool is not registered — default applies

    def _handle_generation(self, envelope):
        tool_calls = envelope.payload.get("tool_calls") or []
        rubric, rubric_name = self._resolve_rubric(tool_calls)
        score, feedback = self._grade(query, answer, rubric=rubric)
        # publish CritiqueResult with rubric_name included

CritiqueBar UI Sketch

● 4/5  ◈ style  ◈ feedback ▶   ↑ ↓
● 5/5  ◈ clarity               ↑ ↓
● 3/5  ◈ realistic ◈ feedback ▶ ↑ ↓

The rubric name chip is always shown when a score is present. Clicking it could eventually expand to show the full rubric used (future enhancement).

What Goes Away

• e2b gatekeeper LLM call (~10s latency on every web search response)
• critic.skipped bus subject
• CriticSkipped message class
• CRITIC_SKIPPED in subjects.py, local_session.py, ws_bridge.py
• ● — ◈ skip UI chip in CritiqueBar
• criticSkipped field on ChatMessage
• CriticSkippedEvent in events.ts
• critic_skipped case in chatStreamReducer
• gatekeeper_* fields in critic.yaml
• gatekeeper mock in test_critic_agent.py
• (RECEIVING, PUBLISH) bypass transition in critic_transitions.py

Implementation Order

1. Write three rubric definitions in critic.yaml (style, clarity; realistic already exists)
2. Add critique fields to each tool's YAML config
3. Update BaseTool.announce() to include critique fields in tool.schema payload
4. Update CriticAgent: add rubric registry, _handle_schema, _resolve_rubric, remove gatekeeper
5. Update critic_transitions.py (remove bypass)
6. Remove CRITIC_SKIPPED from protocol (subjects, messages)
7. Update local_session.py, ws_bridge.py (remove CRITIC_SKIPPED)
8. Update CritiqueResult payload to include rubric_name
9. Update memory_service write_episodic to store rubric_name in metadata
10. Update gateway get_session to include critic_rubric_name in enriched message
11. Frontend: events.ts, chatStreamReducer, CritiqueBar, MessageRow, useSessions, client.ts
12. Update tests
13. make dist, restart stack, verify

Open Questions

• Tool config file names — need to verify exact paths for each tool before step 2 (some tools may not have their own YAML)
• CritiqueResult message — does it currently carry enough fields to add rubric_name, or does the dataclass need a new field?
• Should rubric name be clickable to expand the full rubric text? (Nice-to-have, defer)