Metadata-Version: 2.4
Name: hot-context
Version: 0.1.2
Summary: Intelligent LRU context manager for LLM agents — scores chunks by relevance, recency, and access frequency. Keeps hot context, evicts cold. Drop-in for Strands, LangChain, and OpenAI.
Project-URL: Homepage, https://github.com/Ankit28P/hot-context
Project-URL: Repository, https://github.com/Ankit28P/hot-context
Project-URL: Bug-Tracker, https://github.com/Ankit28P/hot-context/issues
Project-URL: Discussion, https://github.com/Ankit28P/hot-context/discussions
Author-email: Ankitkumar Patel <ankit1990patel@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,bedrock,context-management,conversation,llm,lru,mcp,strands
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Provides-Extra: all
Requires-Dist: langchain-core>=0.1.0; extra == 'all'
Requires-Dist: numpy>=1.24; extra == 'all'
Requires-Dist: sentence-transformers>=2.0; extra == 'all'
Requires-Dist: strands-agents>=0.1.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: numpy>=1.24; extra == 'embeddings'
Requires-Dist: sentence-transformers>=2.0; extra == 'embeddings'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.1.0; extra == 'langchain'
Provides-Extra: strands
Requires-Dist: strands-agents>=0.1.0; extra == 'strands'
Description-Content-Type: text/markdown

# 🔥 hot-context

**LRU-based intelligent context manager for LLM agents.** Keeps hot context, evicts cold.

Most conversation managers use a naive sliding window — drop the oldest messages when context fills up. `hot-context` treats your context window like an intelligent cache: every chunk is scored by relevance, recency, and access frequency, and only the highest-value context survives.

[![PyPI version](https://badge.fury.io/py/hot-context.svg)](https://pypi.org/project/hot-context/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![arXiv](https://img.shields.io/badge/arXiv-2026.XXXXX-b31b1b.svg)](Coming Soon)

---

## Key Features

- **Relevance-aware trimming** — trims tool results to only the lines the LLM actually used, instead of evicting entire entries
- **Adaptive token calibration** — learns the real chars-per-token ratio from actual LLM feedback, self-corrects over time
- **Entity echo feedback loop** — uses the LLM's own response as a proxy for attention weights, zero extra API calls
- **Turn-aware eviction** — never evicts during a turn (LLM needs full context to reason), only trims between turns
- **Framework-agnostic core** with drop-in adapters for Strands, LangChain, and OpenAI

---

## How It Works

```
Turn N:
  User query → Tools return data → LLM sees FULL context → Answers
                                                              ↓
                                          Entity echo: what did the LLM use?
                                          Trimmer: keep only relevant lines from tool results
                                          Calibrator: learn real token ratio from Bedrock
                                                              ↓
Turn N+1:
  Context has trimmed tool results (5k instead of 80k) + all assistant answers preserved
```

**Scoring formula (no LLM needed):**

```
score = α·similarity(chunk, query) + β·recency(chunk) + γ·access_frequency(chunk) - δ·size_penalty(chunk)
```

- **Similarity**: Jaccard overlap on extracted entities (BM25-inspired, zero API calls)
- **Recency**: Exponential decay by turn distance
- **Access frequency**: Time-decayed count of how often the LLM actually used this chunk
- **Size penalty**: Large tool results are penalized proportionally to budget

**Topic shift detection**: When the user changes topics, similarity weight increases and frequency weight decreases — preventing stale popular chunks from dominating.

---

## Results

Tested against Strands `SlidingWindowConversationManager` on a real multi-turn agent with MCP tools (SQL queries, knowledge base lookups):

| Metric | SlidingWindow | hot-context |
|--------|--------------|-------------|
| Q3 (heavy query) | **CRASHED** — context overflow | ✅ Answered (2831 chars) |
| Peak context Q2 | Unbounded (168k+) | **105k** (trimmed) |
| Peak context Q3 | Overflow (>200k) | **113k** (controlled) |
| Queries completed | 2.5 of 3 | **3 of 3** |

Peak context went DOWN from Q1→Q2 (128k→105k) — the trimmer removed irrelevant lines from Q1's tool result.

---

## Quick Start

### Installation

```bash
# Core — no dependencies
pip install hot-context

# With AWS Strands adapter
pip install hot-context[strands]

# With LangChain adapter
pip install hot-context[langchain]

# With embedding-based similarity (recommended for production)
pip install hot-context[embeddings]

# Everything
pip install hot-context[all]
```

### Strands (drop-in replacement)

```python
from strands import Agent
from strands.models.bedrock import BedrockModel
from hot_context.adapters.strands import StrandsHotContextManager
from hot_context import ScoringWeights

agent = Agent(
    model=BedrockModel(model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0"),
    conversation_manager=StrandsHotContextManager(
        token_budget=100_000,
        weights=ScoringWeights.tool_agent(),
    ),
)
```

### LangChain (drop-in replacement)

```python
from langchain_core.runnables import RunnableWithMessageHistory
from hot_context.adapters.langchain import HotContextChatHistory
from hot_context import ScoringWeights

history = HotContextChatHistory(
    token_budget=50_000,
    weights=ScoringWeights.conversational(),
)

chain_with_history = RunnableWithMessageHistory(
    runnable=your_chain,
    get_session_history=lambda session_id: history,
)
```

### Framework-agnostic core

```python
from hot_context import HotContextManager, ScoringWeights

manager = HotContextManager(
    token_budget=100_000,
    weights=ScoringWeights.tool_agent(),
    pin_last_n=2,
)

# Before each query
manager.on_query(user_input)

# After LLM responds
manager.on_response(str(result))
```

### Adaptive calibration (Strands)

```python
# Token estimator self-calibrates from actual Bedrock token counts.
# After each LLM call, feed back the reported input_tokens:
cm = agent.conversation_manager
cm.calibrate(agent.messages, actual_input_tokens=87450)
# Future estimates are now more accurate — no manual tuning needed.
```

### Weight Presets

Use a built-in preset or define your own custom weights:

```python
ScoringWeights.tool_agent()     # high similarity — for agents with many tool calls
ScoringWeights.conversational() # high recency + frequency — for multi-turn chat
ScoringWeights.research()       # high similarity + frequency — for RAG/research agents

# Custom weights — all four components must sum to 1.0
ScoringWeights(
    similarity=0.5,    # how much query-chunk relevance matters
    recency=0.2,       # how much to favor recent turns
    access_freq=0.2,   # how much to favor chunks the LLM has used before
    size_penalty=0.1,  # how much to penalize large chunks
)
```

### Token Budget

`token_budget` is the maximum tokens reserved for conversation history. It should be sized as:

```
token_budget = model_context_limit - expected_query_tokens - system_prompt_tokens - tool_definition_tokens
```

For example, with Claude's 200k context window:

```python
HotContextManager(
    token_budget=100_000,  # leaves headroom for prompts, tools, and the response
)
```

Setting it too high risks context overflow; too low and hot-context will aggressively evict useful history.

### pin_last_n

`pin_last_n` protects the most recent `N` turns from eviction, ensuring the LLM always has immediate conversational context regardless of score:

```python
HotContextManager(
    pin_last_n=2,  # last 2 turns are never evicted (default)
)
```

Increase this if your agent relies heavily on recent tool results that shouldn't be trimmed mid-conversation.

### Custom Entity Patterns

```python
import re
from hot_context import HotContextManager

manager = HotContextManager(
    extra_patterns=[
        ("order_id", re.compile(r"\b\d{3}-\d{7}-\d{7}\b")),
        ("asin", re.compile(r"\bB[A-Z0-9]{9}\b")),
    ]
)
```

---

## Architecture

```
┌──────────────────────────────────────────────────────────────┐
│                    HotContextManager                         │
│                                                              │
│  TokenEstimator ──→ EntityExtractor ──→ ContextScorer        │
│       ↑                                      │               │
│  calibrate()                            score + rank         │
│  (from LLM)                                  │               │
│                                              ↓               │
│  Trimmer ←── FeedbackLoop ←── LLM Response ←── Evictor      │
│  (line-level    (entity echo)                (entry-level)   │
│   relevance)                                                 │
└──────────────────────────────────────────────────────────────┘

Adapters: Strands | LangChain | OpenAI
```

---

## Why Not Just Summarize?

| Approach | Token cost | Latency | Quality |
|----------|-----------|---------|---------|
| Sliding window | Free | None | Poor — drops relevant old context |
| LLM summarization | High | +500ms/turn | Good — but expensive at scale |
| **hot-context** | **Free** | **~1ms/turn** | **Good — keeps what matters, trims what doesn't** |

---

## Components

| File | Purpose |
|------|---------|
| `manager.py` | Core orchestrator: `on_query()` → score + evict, `on_response()` → feedback |
| `scorer.py` | Scoring formula with topic shift detection and pluggable similarity |
| `token_estimator.py` | Adaptive chars-per-token calibration from real LLM feedback |
| `trimmer.py` | Line-level relevance trimming of tool results |
| `chunker.py` | Smart chunking respecting tool-use/tool-result pairing |
| `entity_extractor.py` | Regex-based entity extraction (dates, IDs, numbers, custom patterns) |
| `feedback.py` | Entity echo feedback loop — updates access logs from LLM response |
| `types.py` | `ContextEntry`, `AccessRecord`, `ScoringWeights` with presets |
| `adapters/strands.py` | Drop-in `ConversationManager` for Strands agents |
| `adapters/langchain.py` | `BaseChatMessageHistory` adapter for LangChain |
| `adapters/openai.py` | Message list manager for OpenAI Chat API |

---

## Roadmap

### P2 — Benchmarks & Paper
- [ ] **Benchmark harness** — synthetic multi-turn conversations with known answers. Measure token savings + answer quality (ROUGE/BERTScore on trimmed vs full context)
- [ ] **Standard benchmarks** — LongBench, SCROLLS, QuALITY, NarrativeQA for long-context evaluation
- [ ] **Ablation studies** — trimmer-only vs scorer-only vs full pipeline vs baselines (MemGPT, LLMLingua, attention sink)
- [ ] **Token savings curves** — plot peak context vs conversation length across strategies

---

## Contributing

Issues and PRs welcome at [github.com/Ankit28P/hot-context](https://github.com/Ankit28P/hot-context).

## License

Apache 2.0 — see [LICENSE](LICENSE).
