LoCAL2 Phase 23 — Structured Context

Separates the flat Ollama message array into three conceptually distinct context tiers. MemoryAgent becomes the pre-generation relay — it receives the query, runs fast similarity search on pre-computed recall capsules, and publishes memory.context as the starting gun for GeneratorAgent. Pinned facts ("remember this") are held in GeneratorAgent's in-memory cache, kept current via push events from MemoryAgent — no file reads, no per-turn requests.

1. Motivation

Currently everything — conversation turns, tool scaffolding, retrieved memories, persona seeds — is accumulated in a single flat message array. This causes:

2. Context Tier Model

TierContentScopeOwner
Always-on System prompt (rules, identity, persona clause) + pinned facts ("remember this" items) Every generation GeneratorAgent holds pinned facts in-memory cache; MemoryAgent pushes updates via user.context.updated
Retrieved High-confidence episodic matches for current query Per query, ephemeral — not persisted to conversation history MemoryAgent pre-retrieves; delivered via memory.context message
Working context Conversation turns + tool call/result pairs (full, for transparency) Session, compacted when full GeneratorAgent + ConversationService (unchanged)
Working context stays intact including tool scaffolding — transparency on session rejoin must not be lost. Retrieved context (tier 2) is injected as a preamble into generation but is not appended to the conversation history. Visibility is handled by the context biscuit (see section 5).

3. New Event Flow

ingestion time (response.generation — off hot path):
  MemoryAgent stores engram as before
  MemoryAgent LLM produces a recall capsule — pre-distilled, ready-to-inject summary
  Capsule stored alongside raw engram in ChromaDB

startup:
  GeneratorAgent publishes user.context.request
  MemoryAgent responds with user.context { facts: [...] }
  GeneratorAgent stores facts in _user_context cache

"remember this" mid-session:
  MemoryAgent detects intent, stores pinned fact
  MemoryAgent publishes user.context.updated { fact: "..." }
  GeneratorAgent receives it, appends to _user_context cache
  Next generation already has it — no request, no file read

query time (hot path):
  query.received
    → MemoryAgent (subscribes to query.received — NEW)
        → fast ChromaDB similarity search on pre-computed capsules (no LLM)
        → if score >= threshold:  memory.context { query, session_id, query_id, attachments, retrieved: [capsule, ...] }
        → if score <  threshold:  memory.context { query, session_id, query_id, attachments, retrieved: [] }
        (always publishes — "got nothing" is a valid message)

    → GeneratorAgent (subscribes to memory.context — CHANGED from query.received)
        → injects _user_context from cache (always-on pinned facts)
        → injects retrieved capsules as preamble (if any)
        → writes context biscuit to ConversationService (query_id → { capsules, pinned_facts, persona })
        → builds messages: [system] [pinned facts?] [retrieved capsules?] [history] [user turn]
        → generates as before
MemoryAgent is now in the hot path. It must be fast — pure vector lookup, no LLM call during pre-retrieval. If MemoryAgent is slow or down, generation stalls. This is a new reliability dependency.

4. Context-Dependent Query Handling

Queries like "when was she born?" or "what did we say about that?" have no specific semantic content in isolation. The vector embedding is too vague to produce a high-confidence match — MemoryAgent publishes empty, GeneratorAgent proceeds with conversation history only.

Gemma then reads the conversation, understands the reference ("she" = person discussed in turn 2), and can call search_memory as a tool with a resolved, specific query. This is the correct layer for context-dependent retrieval — Gemma has the full conversation context that MemoryAgent lacks.

Guard: Use an aggressive confidence threshold. MemoryAgent should err toward empty rather than risk injecting plausible-but-wrong context (medium-confidence match for a different person/topic).

5. Context Biscuit — Per-Turn Transparency Record

Retrieved context (capsules) and pinned facts are injected into generation as a preamble but are NOT stored in the Ollama message array. Without a record, they're invisible on session rejoin. The context biscuit solves this — a small stored artifact per turn that captures the full context state at generation time.

ConversationService stores a parallel context_log alongside the message array, keyed by query_id:

session store (ConversationService)
  messages[]              ← LLM sees this (unchanged)
  context_log{}           ← human sees this; keyed by query_id
    <query_id>:
      capsules: [...]       ← what memory.context delivered for this turn
      pinned_facts: [...]   ← snapshot of _user_context at generation time
      persona: "..."        ← active persona at generation time
PropertyValue
Written byGeneratorAgent — just before generation, after preamble is built
Stored inConversationService context_log per session
Exposed viaGET /api/sessions/{id} — returned alongside messages
Re-injected on rejoin?No — the answer already reflects that context. Biscuit is a receipt, not a re-play.
Displayed in UIPer-turn, collapsible — similar to thinking block

The biscuit also unifies persona visibility: instead of tracking persona separately in GeneratorAgent state and forwarding it via response.generation, the biscuit captures it per-turn as a matter of record. The XAI footer reads from the biscuit rather than deriving from tool_calls.

6. Pinned Facts — Push/Cache Model

Distinct from episodic memory. Episodic = retrievable when relevant. Pinned = always injected, regardless of query.

GeneratorAgent holds pinned facts in _user_context: list[str] in memory — no file read per turn. MemoryAgent is the authoritative store (file, DB, or similar — internal detail). The two stay in sync via bus events.

EventDirectionWhen
user.context.requestGeneratorAgent → MemoryAgentGeneratorAgent startup — fetch current fact list
user.contextMemoryAgent → GeneratorAgentResponse to request — full current fact list
user.context.updatedMemoryAgent → GeneratorAgent + UIAny time a new pinned fact is stored — pushed immediately

Mid-session "remember this": MemoryAgent stores the fact and publishes user.context.updated immediately. GeneratorAgent appends to its cache. The very next generation includes it — no restart, no re-request.

"Remember this" intent detection — options (decide before 23c):

7. Retrieval Settings

All matches above the similarity threshold are injected, capped at a maximum count. Both values live in config/memory.yaml — tunable without code changes.

retrieval:
  min_similarity: 0.85   # cosine similarity; stored as distance = 1 - similarity internally
  max_results: 7         # safety cap; in practice 0.85+ threshold yields 0-3 matches

Implementation note: ChromaDB returns distance by default (lower = closer), not similarity. 0.85 cosine similarity = 0.15 distance. Convert at the query boundary — store and expose as similarity, translate to distance when calling ChromaDB.

The cap of 7 is a safety net. At 0.85+ threshold, most queries return 0–3 matches. 7 short summaries adds ~300–500 tokens in a 128k window — negligible. Watch for false positives at the margin during testing and raise the threshold if needed.

8. New Bus Subjects

SubjectPublisherSubscriberPayload
memory.context MemoryAgent GeneratorAgent query, session_id, query_id, attachments, retrieved: Capsule[]
user.context.request GeneratorAgent MemoryAgent (empty — bootstrap request on startup)
user.context MemoryAgent GeneratorAgent facts: string[]
user.context.updated MemoryAgent GeneratorAgent + UI fact: string, reason: string

9. Implementation Steps

23a — Ingestion capsule

23b — MemoryAgent relay + GeneratorAgent subscription change

23c — Pinned facts ("remember this")

23d — Transparency in UI

10. What Does NOT Change

11. Open Questions