Luminus — Extraction Prompt Eval

Prompt: extraction_v0.2  ·  Model: claude-haiku-4-5-20251001  ·  Generated: 2026-04-16 14:55
Test cases
5
Average score
6.8 / 10
Pass rate (≥ 7)
60%
Show prompt — extraction_v0.2
# Extraction Prompt — v0.2

## Purpose

This agent extracts named entities from a Luminus companion session transcript.
Its output feeds two downstream consumers:

1. **Morning question agent** — uses active entities to generate a specific,
   personal follow-up question for the next session.
2. **Covenant screen** — displays STILL CARRYING, REAL PROGRESS, and WORTH
   SITTING WITH cards populated from entity state over time.

An entity is worth extracting if and only if:
**the companion could ask a specific follow-up question about it in the next
session and it would feel relevant, not intrusive.**

## What to extract

- **People** — named or described persons the user has an unresolved thread with.
  Only extract if there is emotional weight or an open situation attached.
- **Situations** — ongoing external events with unresolved stakes and a clear
  resolution point ("the job interview", "the move", "the conversation with my pastor").
- **Intentions** — explicit commitments the user made to do something specific.
  "I'm going to call my dad this week" qualifies. "I should probably pray more" does not
  — no specific action, no commitment.
- **Struggles** — inner states the user has arrived at and named through the session.
  "My anger keeps coming up every time I pray" qualifies. "I've been tired" does not
  — emotional weather, not a named source.

## What to ignore

- **Abstract themes** — faith, doubt, growth, purpose. These are the medium of
  every conversation, not extractable entities.
- **The companion's speech** — only extract what the **user** said.
- **Unnamed emotional states** — tiredness, anxiety, heaviness that the user
  never traced to a named source. If the user didn't name it, don't extract it.
- **Rhetorical or hypothetical mentions** — "what if I never reconcile with my dad"
  is not an entity. "People like my sister would say..." is not an entity.
- **Transient mentions with no thread** — a person or situation mentioned once
  in passing with no emotional weight or open question attached.
- **Emotional states caused by an already-extracted entity** — if the user feels
  anxious *because of* Sarah, that anxiety is not a separate struggle entity.
  It belongs in Sarah's `companion_note`.

## The session agent's role

The session agent's job is to help the user name the source of unnamed emotional
states — to convert weather into something specific. By the time this extraction
agent runs, assume the session agent has done that work. Extract what was named
and surfaced, not what remains implicit.

---

## Entity schema

```json
[
  {
    "name": "Sarah",
    "type": "person",
    "companion_note": "User dreaded a conversation with Sarah about something said at small group. No resolution yet.",
    "status": "active"
  }
]
```

## Types

- `person` — a named or described person ("Sarah", "my brother", "my pastor")
- `situation` — an ongoing external event with unresolved stakes ("the job interview", "the move")
- `intention` — an explicit commitment to a specific action ("I'm going to call my dad this week")
- `struggle` — a named inner state the user arrived at in this session ("my anger at God", "the guilt I carry about my dad")

## Status rules

- `active` — open, unresolved, no resolution language used
- `resolved` — user explicitly stated the situation is closed ("we talked, we're good", "it's done")
- When in doubt, default to `active` — a false positive is less harmful than missing a live thread
- **Always extract resolved entities.** Do not skip an entity because it was resolved.
  Downstream consumers need to know it existed and was closed. Mark it `status: resolved`
  and include it in the output.

## Extraction rules

1. Only extract entities the **user** mentioned, not the companion.
2. Use the name as the user spoke it — "my brother" and "David" are separate entities
   even if they may be the same person.
3. Keep `companion_note` factual and specific to what was said in this session.
   One sentence. No interpretation beyond what was stated.
4. Do not extract unnamed emotional states. If the user said "I've been tired" and
   never named the source, return nothing for that thread.
5. **Bounded follow-up test for struggles** — before extracting a struggle, ask:
   could the morning agent ask "How is [this] going?" and receive a specific, bounded
   answer? "How is your guilt about your dad going?" — yes, extract it.
   "How is your faith going?" — no, too abstract, do not extract.
6. **Name struggles using the user's own words.** Do not compound the emotion with
   its cause or context. If the user said "I think I'm scared", the struggle name is
   "fear of calling my mom" — not "fear of heavy conversations because of her loneliness
   since my dad passed." Keep it close to what the user actually said.
7. **Emotional states caused by a named entity are not separate struggles.** If the
   user feels anxious because of a situation already extracted, note the emotion in
   that entity's `companion_note`. Do not create a second entity for the emotional state.
8. Do not extract abstract spiritual themes (faith, doubt, purpose) as struggles
   unless the user gave them a specific personal name and the bounded follow-up test
   passes ("my doubt about whether God hears me" — borderline; "faith" alone — no).
9. Return ONLY a valid JSON array. No prose, no markdown fences, no explanation.
   If nothing qualifies, return `[]`.

## Session transcript

{transcript}
Scenario Extraction output Criteria Strengths / Weaknesses Score Reasoning
Session agent surfaced a named person and a specific intention — both active
```json
[
  {
    "name": "Sarah",
    "type": "person",
    "companion_note": "User has been avoiding a conversation with Sarah after she dismissed them at small group. User plans to send a text this week to acknowledge what happened.",
    "status": "active"
  },
  {
    "name": "I'm going to send Sarah a text this week",
    "type": "intention",
    "companion_note": "User committed to texting Sarah this week to acknowledge the small group incident rather than continuing to avoid the conversation.",
    "status": "active"
  },
  {
    "name": "fear of Sarah dismissing me again",
    "type": "struggle",
    "companion_note": "User is afraid Sarah will dismiss them again like she did at small group, which is what has kept them from reaching out.",
    "status": "active"
  }
]
```
  • Sarah is extracted as a person entity with status active
  • The intention to text Sarah this week is extracted as a separate intention entity with status active
  • The distraction in prayer life is NOT extracted — it is emotional weather, not a named struggle
  • No entities are fabricated beyond what the user explicitly named
  • Valid JSON array format with proper structure
  • Correctly extracts Sarah as a person entity with active status
  • Correctly identifies and extracts the intention to text Sarah this week as a separate entity
  • Properly excludes the prayer life distraction as emotional weather rather than a named struggle
  • Accurately captures the fear of dismissal as a struggle entity
  • Only extracts from user speech (role: You), not from Luminus
  • Companion notes are accurate, specific, and grounded in transcript content
  • No fabricated entities beyond what the user explicitly mentioned
  • The intention entity name could be slightly more concise or formally structured (e.g., 'Text Sarah to acknowledge small group incident' vs. 'I'm going to send Sarah a text this week'), though the current phrasing is acceptable
JSON ✓
9 The extraction output meets all mandatory requirements: it is valid JSON, extracts only from user speech, and does not fabricate entities. All three entities are explicitly present in the transcript. Sarah is correctly identified as a person with active status. The intention to text Sarah is properly extracted as a separate entity with active status and accurate companion notes. The struggle (fear of dismissal) is correctly identified and typed. Critically, the extraction correctly excludes the prayer life distraction as emotional weather rather than a named struggle, demonstrating proper judgment about what constitutes a struggle entity. The companion notes are specific, accurate, and grounded in the transcript. The output fully satisfies all criteria specified in the test case.
Session agent traced emotional weather to a named struggle — only the named struggle is extracted
```json
[
  {
    "name": "my dad's sickness",
    "type": "situation",
    "companion_note": "User's father is sick and user feels they are the only one stepping up to care for him.",
    "status": "active"
  },
  {
    "name": "resentment toward my family",
    "type": "struggle",
    "companion_note": "User resents being the one who always shows up and feels unacknowledged; this pattern has been longstanding.",
    "status": "active"
  }
]
```
  • The resentment toward family is extracted as a struggle entity with status active — the user named it explicitly
  • The father is extracted as a person entity OR the father's illness as a situation entity with status active — either type is acceptable
  • Tiredness and heaviness are NOT extracted — they are emotional weather that the session agent correctly traced to the named source
  • The family dynamic in general is NOT extracted as a separate entity — the struggle covers it
  • Valid JSON array format with correct structure
  • Correctly identifies resentment as a struggle entity with active status
  • Appropriately extracts father's sickness as a situation entity
  • Companion notes are accurate and well-sourced from transcript
  • Does not extract emotional weather (tiredness, heaviness) as separate entities
  • Does not fabricate entities absent from transcript
  • Only extracts from user's statements, not Luminus's prompts
  • Correctly recognizes the longstanding nature of the resentment pattern
  • Minor: Could have been more explicit about the lack-of-acknowledgment component in the resentment entity name (though companion_note captures it)
JSON ✓
9 The extraction meets all mandatory requirements: it is valid JSON, contains only entities mentioned by the user, and fabricates nothing. Against the specific criteria: (1) resentment is correctly extracted as a struggle with active status; (2) father's sickness is extracted as a situation entity, which satisfies the 'either type acceptable' requirement; (3) tiredness and heaviness are correctly omitted; (4) family dynamic is not extracted as a separate entity. The companion notes are accurate, grounded in the transcript, and capture the essential context. The extraction demonstrates strong understanding of what constitutes emotional weather versus named struggles, and correctly identifies the active, ongoing nature of both entities.
Explicit resolution mid-session — entity extracted as resolved
```json
[
  {
    "name": "my brother",
    "type": "person",
    "companion_note": "User had a dreaded conversation about money on Sunday. They listened instead of defending themselves, worked out a plan, and resolved it.",
    "status": "resolved"
  }
]
```
  • The brother is extracted as a person entity with status resolved
  • The money situation is extracted as a situation entity with status resolved — user said it is done
  • No active entities are created — everything mentioned was resolved in this session
  • Valid JSON array format
  • Correctly extracted the brother as a person entity
  • Accurately marked status as 'resolved'
  • Only extracted from user's statements, not Luminus
  • Companion note is accurate and relevant
  • Missing the money situation entity - criteria explicitly requires extracting 'the money situation' as a situation entity with status resolved
  • Incomplete extraction - only 1 of 2 required entities present
  • Did not capture the emotional outcome/struggle entity (the dread, the defensive pattern, the feeling of being 'lighter')
JSON ✓
4 The extraction is technically valid JSON and follows the mandatory requirement of only extracting user-mentioned entities without fabrication. However, it fails to meet the specific criteria for this test case. The criteria explicitly states that 'The money situation is extracted as a situation entity with status resolved' and 'No active entities are created.' The output only extracts the brother as a person entity but completely omits the money situation entity that should have been extracted as a separate entity. The companion_note for the brother is accurate, but the extraction is incomplete according to the stated criteria.
Soft desire vs hard intention — only the explicit commitment qualifies
```json
[
  {
    "name": "my mom",
    "type": "person",
    "companion_note": "User hasn't returned her voicemail from four days ago and feels scared of the conversation because she's been lonely since his dad passed.",
    "status": "active"
  },
  {
    "name": "call my mom tonight after dinner",
    "type": "intention",
    "companion_note": "User committed to calling his mom tonight after dinner and explicitly stated he won't let another day go by.",
    "status": "active"
  },
  {
    "name": "fear of the conversation with my mom",
    "type": "struggle",
    "companion_note": "User named fear as what's stopping him from calling; the fear is specifically about heavy conversations with his lonely mom.",
    "status": "active"
  }
]
```
  • The intention to call mom tonight is extracted as an intention entity with status active — user made an explicit specific commitment
  • Journaling is NOT extracted as an intention — 'I should probably' is a soft desire, not a commitment
  • The mother is extracted as a person entity with status active — there is an unresolved emotional thread
  • The heaviness of the calls is NOT extracted as a struggle — it was named as context, not as a named inner state the user claimed
  • Valid JSON array format with proper structure
  • Correctly extracts the commitment to call mom tonight as an intention with active status
  • Correctly excludes journaling as it was only a soft desire ('should probably'), not a commitment
  • Correctly identifies the mother as a person entity with active status
  • Correctly identifies fear of the conversation as a struggle entity
  • All entities are sourced from user speech only, not from Luminus
  • Companion notes are accurate and well-contextualized
  • No fabricated entities present
  • The companion_note for 'my mom' uses 'his dad' instead of 'her dad' - minor pronoun inconsistency that could reflect uncertainty about user gender, though the transcript doesn't explicitly state it
JSON ✓
9 The extraction output meets all mandatory requirements: it is valid JSON, contains only entities mentioned by the user, and fabricates nothing. Against the specific criteria provided, it correctly handles all four test cases: (1) the intention to call mom is extracted with active status due to explicit commitment, (2) journaling is not extracted as it was only a soft desire, (3) the mother is extracted as a person with active status due to unresolved emotional thread, and (4) the heaviness of calls is not extracted as a separate struggle entity. The three entities extracted are appropriate, well-typed, and have accurate companion notes. The only minor issue is a pronoun inconsistency in one companion note that doesn't affect the core extraction quality.
Abstract spiritual session with no named entities — extraction returns empty array
```json
[
  {
    "name": "whether my faith is really mine or just inherited",
    "type": "struggle",
    "companion_note": "User is questioning the origins of their faith after reading different perspectives. Feels uncomfortable but describes it as something waking up.",
    "status": "active"
  },
  {
    "name": "belief in the resurrection",
    "type": "struggle",
    "companion_note": "User is uncertain whether they genuinely believe in the resurrection or have never questioned it before.",
    "status": "active"
  }
]
```
  • Output is an empty JSON array []
  • Faith, doubt, and resurrection are NOT extracted — they are abstract spiritual themes, not named entities
  • The inherited vs chosen faith question is NOT extracted as a struggle — the user never gave it a personal name or handle
  • No entities are fabricated from the emotional tone or the questions the companion asked
  • Output is valid JSON array format
  • Only extracts from user speech, not companion speech
  • Correctly identifies active status for both entities
  • Companion notes are accurate and well-sourced from transcript
  • Extracts entities that should not be extracted per criteria — the criteria explicitly states 'Faith, doubt, and resurrection are NOT extracted — they are abstract spiritual themes, not named entities'
  • Extracts the inherited vs chosen faith question as a struggle despite criteria stating 'the user never gave it a personal name or handle'
  • Misinterprets the task: the criteria indicates output should be an empty JSON array [], but extraction output contains two entities
  • Treats abstract spiritual themes and open questions as named entities when they lack concrete naming
JSON ✓
3 The extraction output violates the explicit criteria provided for this test case. The criteria clearly states that the output should be an empty JSON array because: (1) faith, doubt, and resurrection are abstract spiritual themes, not named entities; (2) the inherited vs chosen faith question was never given a personal name or handle by the user; and (3) entities should not be fabricated from emotional tone or companion questions. The output extracts exactly what the criteria says should NOT be extracted. While the JSON is valid and the companion notes are accurate reflections of the transcript, the fundamental task interpretation is incorrect. The extraction treats open-ended questions and abstract themes as extractable entities when the criteria explicitly rejects this approach.