Luminus — Extraction Prompt Eval

Prompt: extraction_v0.1  ·  Model: claude-haiku-4-5-20251001  ·  Generated: 2026-04-16 14:36
Test cases
5
Average score
5.8 / 10
Pass rate (≥ 7)
40%
Show prompt — extraction_v0.1
# Extraction Prompt — v0.1

## Purpose

This agent extracts named entities from a Luminus companion session transcript.
Its output feeds two downstream consumers:

1. **Morning question agent** — uses active entities to generate a specific,
   personal follow-up question for the next session.
2. **Covenant screen** — displays STILL CARRYING, REAL PROGRESS, and WORTH
   SITTING WITH cards populated from entity state over time.

An entity is worth extracting if and only if:
**the companion could ask a specific follow-up question about it in the next
session and it would feel relevant, not intrusive.**

## What to extract

- **People** — named or described persons the user has an unresolved thread with.
  Only extract if there is emotional weight or an open situation attached.
- **Situations** — ongoing external events with unresolved stakes and a clear
  resolution point ("the job interview", "the move", "the conversation with my pastor").
- **Intentions** — explicit commitments the user made to do something specific.
  "I'm going to call my dad this week" qualifies. "I should probably pray more" does not
  — no specific action, no commitment.
- **Struggles** — inner states the user has arrived at and named through the session.
  "My anger keeps coming up every time I pray" qualifies. "I've been tired" does not
  — emotional weather, not a named source.

## What to ignore

- **Abstract themes** — faith, doubt, growth, purpose. These are the medium of
  every conversation, not extractable entities.
- **The companion's speech** — only extract what the **user** said.
- **Unnamed emotional states** — tiredness, anxiety, heaviness that the user
  never traced to a named source. If the user didn't name it, don't extract it.
- **Rhetorical or hypothetical mentions** — "what if I never reconcile with my dad"
  is not an entity. "People like my sister would say..." is not an entity.
- **Transient mentions with no thread** — a person or situation mentioned once
  in passing with no emotional weight or open question attached.

## The session agent's role

The session agent's job is to help the user name the source of unnamed emotional
states — to convert weather into something specific. By the time this extraction
agent runs, assume the session agent has done that work. Extract what was named
and surfaced, not what remains implicit.

---

## Entity schema

```json
[
  {
    "name": "Sarah",
    "type": "person",
    "companion_note": "User dreaded a conversation with Sarah about something said at small group. No resolution yet.",
    "status": "active"
  }
]
```

## Types

- `person` — a named or described person ("Sarah", "my brother", "my pastor")
- `situation` — an ongoing external event with unresolved stakes ("the job interview", "the move")
- `intention` — an explicit commitment to a specific action ("I'm going to call my dad this week")
- `struggle` — a named inner state the user arrived at in this session ("my anger at God", "the guilt I carry about my dad")

## Status rules

- `active` — open, unresolved, no resolution language used
- `resolved` — user explicitly stated the situation is closed ("we talked, we're good", "it's done")
- When in doubt, default to `active` — a false positive is less harmful than missing a live thread

## Extraction rules

1. Only extract entities the **user** mentioned, not the companion.
2. Use the name as the user spoke it — "my brother" and "David" are separate entities
   even if they may be the same person.
3. Keep `companion_note` factual and specific to what was said in this session.
   One sentence. No interpretation beyond what was stated.
4. Do not extract unnamed emotional states. If the user said "I've been tired" and
   never named the source, return nothing for that thread.
5. Do not extract abstract spiritual themes (faith, doubt, purpose) as struggles
   unless the user gave them a specific personal name ("my doubt about whether God
   hears me" — borderline; "doubt" alone — no).
6. Return ONLY a valid JSON array. No prose, no markdown fences, no explanation.
   If nothing qualifies, return `[]`.

## Session transcript

{transcript}
Scenario Extraction output Criteria Strengths / Weaknesses Score Reasoning
Session agent surfaced a named person and a specific intention — both active
```json
[
  {
    "name": "Sarah",
    "type": "person",
    "companion_note": "User has unresolved tension with Sarah after being dismissed at small group; fears dismissal again if she reaches out.",
    "status": "active"
  },
  {
    "name": "I'm going to text Sarah this week",
    "type": "intention",
    "companion_note": "User committed to sending Sarah a text acknowledging what happened, rather than continuing to avoid the conversation.",
    "status": "active"
  },
  {
    "name": "avoidance of Sarah in prayer",
    "type": "struggle",
    "companion_note": "User named that tension with Sarah surfaces every time she sits down to pray, creating distraction in her prayer life.",
    "status": "active"
  }
]
```
  • Sarah is extracted as a person entity with status active
  • The intention to text Sarah this week is extracted as a separate intention entity with status active
  • The distraction in prayer life is NOT extracted — it is emotional weather, not a named struggle
  • No entities are fabricated beyond what the user explicitly named
  • Valid JSON array format with correct structure
  • Sarah correctly extracted as a person entity with active status
  • Intention to text Sarah this week correctly identified and extracted as separate entity
  • Companion notes are accurate and grounded in transcript
  • Only extracts from user's statements (role: You), not Luminus
  • No fabricated entities beyond what user explicitly mentioned
  • Correctly excludes 'distraction in prayer life' as a named struggle per criteria
  • The intention entity name 'I'm going to text Sarah this week' is slightly verbose; could be more concise like 'Text Sarah to acknowledge tension'
  • Minor: Could have been more explicit about the fear of dismissal as a separate struggle entity, though the current extraction handles it adequately within Sarah's companion_note
JSON ✓
9 The extraction output meets all mandatory requirements: it is valid JSON, extracts only from user statements, and fabricates no entities. It correctly identifies the three key entities per the criteria: Sarah as person (active), the texting intention (active), and avoidance/prayer distraction as struggle (active). The companion notes are accurate and well-grounded in the transcript. The extraction correctly excludes 'distraction in prayer life' as standalone emotional weather rather than a named struggle. The only minor issue is slight verbosity in the intention name, which is a stylistic preference rather than a substantive error.
Session agent traced emotional weather to a named struggle — only the named struggle is extracted
```json
[
  {
    "name": "my dad",
    "type": "person",
    "companion_note": "User's dad is sick. User is the primary caregiver while others in the family are not stepping up.",
    "status": "active"
  },
  {
    "name": "my family",
    "type": "person",
    "companion_note": "User resents that they are always the one who shows up and their effort goes unacknowledged. This resentment surfaces repeatedly.",
    "status": "active"
  },
  {
    "name": "my resentment at always being the one who shows up",
    "type": "struggle",
    "companion_note": "User named a long-standing resentment that surfaces constantly with family but has not typically been named until this session.",
    "status": "active"
  }
]
```
  • The resentment toward family is extracted as a struggle entity with status active — the user named it explicitly
  • The father's illness is extracted as a situation entity with status active
  • Tiredness and heaviness are NOT extracted — they are emotional weather that the session agent correctly traced to the named source
  • The family dynamic in general is NOT extracted as a separate entity — the struggle covers it
  • Output is valid JSON array format
  • Only extracts from user speech, not companion
  • Correctly identifies resentment as a struggle entity with active status
  • Correctly excludes tiredness and heaviness as they are emotional weather, not core entities
  • Companion notes are contextually accurate and relevant
  • Appropriately captures the long-standing nature of the resentment
  • Does not fabricate entities not present in transcript
  • Father's illness is not extracted as a separate situation entity as specified in criteria — it is only mentioned within the companion_note of 'my dad' person entity
  • Type classification for 'my dad' is 'person' when the criteria expects a 'situation' entity for the father's illness
  • 'my family' is typed as 'person' (singular conceptually) when it represents a collective, though this is a minor semantic issue
  • The extraction misses the explicit situation entity that should capture 'father's illness' as a distinct active situation
JSON ✓
7 The extraction demonstrates strong understanding of what to exclude (emotional weather) and correctly identifies the resentment struggle. However, it fails to meet the specific criteria requirement that 'The father's illness is extracted as a situation entity with status active.' The father's illness is mentioned only as context within a person entity's companion_note rather than as its own situation entity. This is a structural misalignment with the stated criteria, though the information is present in the output. The extraction is otherwise sound and avoids fabrication or extraction of companion speech.
Explicit resolution mid-session — entity extracted as resolved
```json
[]
```
  • The brother is extracted as a person entity with status resolved
  • The money situation is extracted as a situation entity with status resolved — user said it is done
  • No active entities are created — everything mentioned was resolved in this session
  • Output is valid JSON (empty array)
  • Does not fabricate entities
  • Does not extract companion speech
  • Fails to extract the brother as a person entity with resolved status
  • Fails to extract the money situation as a situation entity with resolved status
  • Misinterprets the criteria: resolved entities should still be extracted with status marked as 'resolved'
  • Returns empty array when the transcript clearly contains extractable entities
JSON ✓
4 The extraction output is technically valid JSON and follows the mandatory requirement of not fabricating entities or extracting companion speech. However, it fundamentally misunderstands the task requirements. The criteria explicitly state that 'The brother is extracted as a person entity with status resolved' and 'The money situation is extracted as a situation entity with status resolved.' The user clearly mentioned both the brother and a money-related conflict that was resolved. The instruction 'No active entities are created — everything mentioned was resolved in this session' means resolved entities should still be extracted; it does not mean they should be omitted. An empty array violates the specific criteria for this test case, which expects at least two entities (brother and money situation) both marked with resolved status.
Soft desire vs hard intention — only the explicit commitment qualifies
```json
[
  {
    "name": "my mom",
    "type": "person",
    "companion_note": "User has not returned her voicemail in four days; fears the conversation will be heavy due to her loneliness since user's dad passed.",
    "status": "active"
  },
  {
    "name": "calling my mom",
    "type": "intention",
    "companion_note": "User committed to calling her tonight after dinner.",
    "status": "active"
  },
  {
    "name": "fear of heavy conversations with my mom",
    "type": "struggle",
    "companion_note": "User named fear as the barrier to calling; characterized the calls as heavy due to her loneliness.",
    "status": "active"
  }
]
```
  • The intention to call mom tonight is extracted as an intention entity with status active — user made an explicit specific commitment
  • Journaling is NOT extracted as an intention — 'I should probably' is a soft desire, not a commitment
  • The mother is extracted as a person entity with status active — there is an unresolved emotional thread
  • The heaviness of the calls is NOT extracted as a struggle — it was named as context, not as a named inner state the user claimed
  • Valid JSON array format with proper structure
  • Correctly extracts 'calling my mom' as an intention with active status, matching the explicit commitment 'I'll call her tonight after dinner'
  • Correctly extracts 'my mom' as a person entity with active status and relevant contextual notes
  • Appropriately does NOT extract journaling as an intention, respecting the soft desire criterion
  • All extracted entities originate from user speech, not companion speech
  • Companion notes are specific and grounded in transcript details
  • No fabricated entities present
  • Extracts 'fear of heavy conversations with my mom' as a struggle entity, which violates the criteria stating 'The heaviness of the calls is NOT extracted as a struggle — it was named as context, not as a named inner state the user claimed'
  • The user explicitly named 'scared' as their emotional state ('I think I'm scared of the conversation'), which is a legitimate struggle, but the extraction frames it as 'fear of heavy conversations' rather than the fear itself, conflating the user's emotional state with contextual information about why calls are heavy
JSON ✓
6 The extraction demonstrates strong adherence to most criteria but contains one critical issue. The user did express fear as a barrier to calling ('I think I'm scared of the conversation'), which qualifies as a named inner state and struggle. However, the extraction conflates this with 'the heaviness of the calls,' which the criteria explicitly states should not be extracted as a struggle since it was presented as context rather than a claimed inner state. The extraction should have either: (1) extracted 'fear of calling mom' or 'fear of the conversation' as the struggle (the user's emotional state), or (2) not extracted a struggle entity at all if interpreting the criteria strictly. The current phrasing blurs this distinction by naming the struggle as 'fear of heavy conversations' which mixes the user's emotion with contextual explanation. All other aspects are correct: valid JSON, proper entity types, correct statuses, and no fabricated or companion-sourced content.
Abstract spiritual session with no named entities — extraction returns empty array
```json
[
  {
    "name": "my faith",
    "type": "struggle",
    "companion_note": "User is questioning whether their faith is truly theirs or inherited; uncertain about personal belief in the resurrection.",
    "status": "active"
  }
]
```
  • Output is an empty JSON array []
  • Faith, doubt, and resurrection are NOT extracted — they are abstract spiritual themes, not named entities
  • The inherited vs chosen faith question is NOT extracted as a struggle — the user never gave it a personal name or handle
  • No entities are fabricated from the emotional tone or the questions the companion asked
  • Output is valid JSON array format
  • Only extracts from user speech, not companion speech
  • Correctly identifies an active struggle related to faith questioning
  • Extracts 'my faith' as a named entity when the criteria explicitly states abstract spiritual themes should NOT be extracted
  • The criteria specifies output should be an empty JSON array [], but this extraction contains one entity
  • Treats an unnamed, abstract concern as a named entity when the user never gave it a personal handle or name
  • Violates the core instruction that 'Faith, doubt, and resurrection are NOT extracted — they are abstract spiritual themes, not named entities'
JSON ✓
3 The extraction output violates the explicit criteria provided for this test case. The criteria clearly states the output should be an empty JSON array and that faith-related themes should not be extracted as they are abstract spiritual concepts, not named entities. While the JSON is technically valid and only references user speech, the fundamental task requirement is to return no entities for this transcript. The extraction fabricates an entity ('my faith') from abstract spiritual themes that the user never named or personified. This is a direct contradiction of the test case requirements.