# xMemory - Hierarchical Memory System

Implementation of "Beyond RAG for Agent Memory" (xMemory paper) for neoCoder.

## Architecture

```
L3: Themes     ─────────────────────────────────────────
    │  ThemeNode: High-level semantic clusters
    │  - Summary embedding + Centroid
    │  - kNN neighbors for graph structure
    │
L2: Semantics  ─────────────────────────────────────────
    │  SemanticUnit: Discrete knowledge facts
    │  - Factual, Procedural, Conceptual types
    │  - kNN neighbors for similarity
    │
L1: Episodes   ─────────────────────────────────────────
    │  Episode: Task execution records
    │  - (Uses existing EpisodicMemory)
    │
L0: Messages   ─────────────────────────────────────────
       Message: Raw conversation turns
       - (Uses existing WorkingMemory)
```

## Key Components

### SemanticMemory (`semantic.py`)
Stores factual knowledge extracted from episodes.
- Duplicate detection via cosine similarity
- kNN graph for retrieval
- Keyword-based search

### ThemeManager (`themes.py`)
Organizes semantics into high-level themes.
- **Assimilation**: Attach new semantics to best theme
- **Accommodation**: Split oversized/heterogeneous themes
- **Merge**: Combine similar themes

### SparsitySemanticsScorer (`scoring.py`)
Quality scoring for memory organization.
```
Score = Sparsity + Semantics

Sparsity = (N/K) * (N / Σ(n_k²))  # Balanced partition
Semantics = mean(cohesion_k * g(knn_k))  # Coherent themes
```

### HierarchicalRetriever (`retriever.py`)
Top-down retrieval with uncertainty-driven expansion.
1. Query themes first
2. If uncertain, expand to semantics
3. If still uncertain, ground in episodes

### ConsolidationEngine (`consolidation.py`)
Full pipeline: Extract → Assimilate → Accommodate

### XMemoryIntegration (`integration.py`)
Bridge with existing CognitiveMemoryManager.

## Usage

```python
from neoCoder.sdk.memory.xmemory import (
    create_xmemory_integration,
    XMemoryConfig,
)

# Create integration
xmem = create_xmemory_integration(
    episodic_memory=existing_episodic,
)

# Configure LLM functions
xmem.configure_llm(
    llm_call=my_llm_function,
    embed_fn=my_embedding_function,
)

# Process episodes
result = xmem.process_episode(episode)

# Retrieve context
context = xmem.retrieve_context("implement user auth")

# Add knowledge manually
xmem.add_knowledge(
    content="Always use parameterized queries",
    knowledge_type="procedural",
    keywords=["sql", "security"],
)

# Get quality metrics
stats = xmem.get_quality_metrics()
```

## Configuration

### SemanticMemoryConfig
- `max_units`: Maximum semantic units (default: 500)
- `embedding_dim`: Embedding dimension (default: 1536)
- `knn_k`: Neighbors per unit (default: 10)
- `similarity_threshold`: Duplicate threshold (default: 0.85)

### ThemeManagerConfig
- `attach_threshold`: Min similarity to attach (default: 0.62)
- `max_theme_size`: Split threshold (default: 12)
- `min_intra_similarity`: Cohesion threshold (default: 0.72)
- `merge_threshold`: Merge similarity (default: 0.78)

### RetrieverConfig
- `top_k_themes`: Themes to retrieve (default: 5)
- `theme_uncertainty_threshold`: Expansion trigger (default: 0.7)
- `max_context_chars`: Token budget (default: 4000)

## Benefits over Standard RAG

| Aspect | Standard RAG | xMemory |
|--------|-------------|---------|
| Retrieval | Flat top-k | Hierarchical top-down |
| Redundancy | High (similar chunks) | Low (themes dedupe) |
| Token efficiency | Fixed | Adaptive (expand on uncertainty) |
| Organization | None | Sparsity-semantics optimized |

## References

- Paper: "Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation"
- arXiv: https://arxiv.org/abs/2602.02007
- Code: https://github.com/HU-xiaobai/xMemory
