# Context Compression Middleware — Research Deep Dive

*Compiled: March 11, 2026*

## Executive Summary

Your idea — a middleware layer that compresses massive context so cheap small-context models perform like expensive large-context ones — is real, validated by research, and already being commercialized. The bad news: **you're not first**. The good news: **the market is early, the incumbents are tiny, and the execution space is wide open.**

---

## 1. Academic Research (The Science)

### Core Papers

**LLMLingua** (Microsoft Research, EMNLP 2023)
- arxiv.org/abs/2310.05736
- Coarse-to-fine prompt compression using a small model (e.g., LLaMA-7B) to score token importance via perplexity
- **Results: 20x compression with only 1.5% performance loss**
- Uses budget controller for semantic integrity + iterative token-level compression
- Open source: github.com/microsoft/LLMLingua

**LongLLMLingua** (Microsoft, ACL 2024)
- arxiv.org/abs/2310.06839
- Extension for long-context scenarios — also fixes the "lost in the middle" problem (LLMs ignore info in the middle of long prompts)
- Question-aware compression: uses the query to decide what's important in the context

**Gist Tokens** (Stanford, NeurIPS 2023)
- arxiv.org/abs/2304.08467
- Trains an LLM to compress prompts into special "gist" embedding tokens
- **26x compression, 40% FLOPs reduction, 4.2% wall-time speedup**
- Limitation: compressed tokens only work with the fine-tuned model, not arbitrary LLMs

**In-context Autoencoder (ICAE)** (ICLR 2024)
- arxiv.org/abs/2307.06945
- Compresses long context into compact "memory slots" that the LLM conditions on
- Like an autoencoder but for natural language context within an LLM
- Similar limitation to Gist: requires model-specific training

**AutoCompressor** (Chevalier et al., 2023)
- Handles up to 30,720 tokens of prompt compression
- Similar architecture to Gist tokens — learns compressed representations

**Selective Context** (Li et al., 2023)
- github.com/liyucheng09/Selective_Context
- Simpler approach: uses self-information scores to prune low-value tokens
- **2x more content processed, ~40% token reduction**
- Available as a pip package (`selective-context`)

**CompactPrompt** (Oct 2025)
- arxiv.org/abs/2510.18043
- Unified pipeline for prompt + data compression
- **50%+ token reduction while preserving baseline accuracy** on financial QA tasks
- Tested on Claude 3.5 Sonnet — works cross-model

**CCF: Context Compression Framework** (Sep 2025)
- arxiv.org/abs/2509.09199
- Most recent academic framework for long-sequence compression

### Key Academic Findings

| Method | Compression Ratio | Quality Loss | Works Cross-Model? |
|--------|------------------|-------------|-------------------|
| LLMLingua | 20x | ~1.5% | ✅ Yes |
| Gist Tokens | 26x | Minimal | ❌ Model-specific |
| ICAE | Variable | Low | ❌ Model-specific |
| Selective Context | 2x | Negligible | ✅ Yes |
| CompactPrompt | 2x | Near-zero | ✅ Yes |

**The cross-model constraint is crucial for your idea.** Gist/ICAE achieve better compression but require model-specific training. LLMLingua and token-pruning approaches work with any model — which is what a middleware needs.

---

## 2. Existing Companies & Products

### The Token Company (YC W26) — **YOUR DIRECT COMPETITOR**
- **Founded:** 2025, San Francisco
- **Founder:** Otso Veisterä (18-year-old ML researcher, national physics champion)
- **Batch:** Y Combinator Winter 2026 (current batch)
- **Product:** "bear-1" and "bear-1.1" compression models
- **Claims:** 66% input token compression, costs reduced 3x, accuracy improved up to +1.1%
- **How it works:** Drop-in API middleware — preprocesses inputs before they hit your LLM
- **Team:** 2 people
- **Positioning:** "Optimize every LLM request in the world at the token level"
- **Key insight from founder:** "Labs are incentivized to build bigger models and sell more tokens, not reduce usage. As long as more than one lab exists, a cross-model optimization layer is structurally defensible."
- **Customer proof:** Blind LLM arena study showed compressed requests increased user preference AND lifted purchase volume 5%
- **Website:** thetokencompany.com

### TokenCrush
- **Product:** Commercial compression tool for LangChain/LangGraph RAG pipelines
- **Claims:** 80% cost reduction
- **Focus:** Production RAG pipelines specifically
- **Website:** tokencrush.ai

### Microsoft LLMLingua (Open Source)
- Not a company, but the dominant open-source option
- Integrated into Azure AI, LangChain ecosystem
- Anyone can use it — which means it's table stakes, not a moat

### CompactifAI (Multiverse Computing)
- **Funding:** €189M Series B (June 2025)
- Focus is on model compression (making models smaller), not prompt compression
- Adjacent but different problem

---

## 3. Technical Approaches (Taxonomy)

### A. Token Pruning (what LLMLingua/Selective Context do)
- Use a small model to score each token's importance
- Remove low-importance tokens
- **Pros:** Works with any LLM, no training needed, fast
- **Cons:** Compression ceiling (~20x before quality degrades), lossy

### B. Soft Prompt / Embedding Compression (Gist, ICAE, AutoCompressor)
- Train model to compress text into dense embedding vectors
- **Pros:** Much higher compression ratios (26x+)
- **Cons:** Model-specific — the compressed representation only works with the model it was trained on. Kills the middleware use case.

### C. Extractive Compression
- Select and keep only the most relevant sentences/paragraphs
- Basically smart summarization
- **Pros:** Human-readable compressed output, easy to debug
- **Cons:** Can miss distributed information

### D. Hierarchical / Layered Compression (your Layer 0/1/2 idea)
- This is what you described: progressive summarization layers
- Layer 0: raw, Layer 1: key facts, Layer 2: one-paragraph state
- **Not well-studied academically** — most papers focus on single-pass compression
- **This is an opportunity.** The multi-layer approach with drill-down is architecturally novel for production use.

### E. RAG as Implicit Compression
- Retrieval-Augmented Generation is already a form of compression — you don't send everything, you retrieve what's relevant
- The question: can you combine RAG + compression for even better results?

### F. KV-Cache Compression
- Compresses the key-value cache during inference (internal to the model)
- Not applicable to API middleware — you don't control the model's internals

---

## 4. The Hard Problems

### The Compression Paradox
You identified this: "the compression agent needs to be smart enough to judge relevance — which means you need a good model doing the compression."

**How The Token Company solves it:** They built a specialized small model (bear-1) trained specifically for compression. It's cheap to run because it's small and purpose-built. The cost of running bear-1 is a fraction of what you save on the target LLM.

**The math:** If compressing costs $0.001 and saves $0.05 on the main LLM call, that's a 50x ROI per request.

### What's Safe to Discard?
- Task-dependent: a legal document analysis can't lose any clause, but a customer support conversation can lose pleasantries
- Domain-specific compression rules outperform generic ones
- **Confidence scoring** (your idea #4) is underexplored commercially

### Latency Tradeoff
- Compression adds a round-trip to a compression model before the main LLM call
- The Token Company claims "milliseconds" — but that's for their hosted API
- For cost-sensitive batch workloads, latency doesn't matter. For real-time chat, it might.

### Cacheability
- Your "git diffs for context" idea is genuinely novel
- If context changes incrementally (conversation adds messages), you could cache the compressed state and only compress the delta
- Nobody is doing this well yet

---

## 5. Market Analysis

### TAM
- **LLM API spending in 2025:** $8.4B (Menlo Ventures, up from $3.5B in 2024)
- Growing >100% YoY
- If compression saves 30-60% of input token costs, that's a **$2.5-5B addressable market**

### Price Trends (the headwind)
- LLM inference costs are falling rapidly — Epoch AI data shows dramatic price drops
- **Counter-argument:** Usage is growing faster than prices are falling. Total spend keeps increasing.
- Also: even as frontier models get cheaper, the volume of tokens being processed is exploding (agents, RAG, multi-step reasoning)

### Who Pays?
- **Startups burning $50K+/mo on LLM APIs** — immediate ROI
- **Enterprise AI teams** with production RAG pipelines
- **Agent frameworks** — agents make many LLM calls per task, multiplying the savings
- **NOT individual developers** — too small to care about 60% savings on $5/month

---

## 6. Bottom Line: Is This Viable as a Product?

### What's Real
- The science works. 2-20x compression with minimal quality loss is proven.
- The market is massive and growing.
- There's a YC-backed competitor (The Token Company) validating the market — but they're 2 people and just launched.

### What's Differentiated About Your Vision
Your conversation described something slightly different from what exists:

1. **Agent-to-agent protocol** — not just prompt compression, but a communication standard for multi-agent systems. Nobody owns this yet.
2. **Hierarchical layers with drill-down** — existing solutions are single-pass. Your Layer 0/1/2 architecture with on-demand detail retrieval is novel.
3. **"Video codec" framing** — incremental/delta compression for conversations. Cache the compressed state, compress only new messages. This is a real technical edge.
4. **Making small models act like big ones** — The Token Company focuses on cost reduction with the SAME model. Your pitch is more radical: use compression to let GPT-4o-mini compete with Opus. That's a different (bigger) claim.

### Honest Assessment

**The opportunity is real but the window is closing.** The Token Company has YC backing and a head start. LLMLingua is open source and good enough for many use cases. The pure "save money on API calls" pitch will commoditize fast.

**Where you could win:**
- **The agent protocol layer.** Multi-agent AI is exploding (Agentforce, CrewAI, AutoGen, LangGraph). None of them have a standardized context compression protocol. You know this world from Salesforce.
- **Enterprise positioning.** An 18-year-old in SF is not selling to Fortune 500 AI teams. You are. You literally train architects on this stuff.
- **Salesforce ecosystem integration.** Agentforce + Data Cloud + compression middleware = a story that writes itself for your existing network.

**Where it's hard:**
- You'd need to build or fine-tune a compression model (bear-1 equivalent). This is ML engineering, not prompt engineering.
- The "compression paradox" is real — you need ML talent to solve it well.
- If model prices keep dropping 10x/year, the cost-savings pitch erodes (but the "fit more context in small windows" pitch doesn't).

### Recommended Next Steps

1. **Try The Token Company's API.** Sign up, compress some real prompts, see how good it actually is. Know your competition.
2. **Prototype the hierarchical approach.** Use LLMLingua (open source) to build a proof-of-concept with your Layer 0/1/2 architecture. See if it actually outperforms single-pass compression.
3. **Talk to agent framework builders.** CrewAI, LangChain, AutoGen teams. Ask them: "Would you pay for a context compression protocol?" Validate the agent-to-agent angle.
4. **Consider the Salesforce angle.** Agentforce customers are paying Salesforce per-conversation. Compression = more agent capability per dollar. That's a pitch you can make tomorrow.

---

## Appendix: Key Links

| Resource | URL |
|----------|-----|
| LLMLingua (Microsoft) | github.com/microsoft/LLMLingua |
| Gist Tokens paper | arxiv.org/abs/2304.08467 |
| ICAE paper | arxiv.org/abs/2307.06945 |
| Selective Context | github.com/liyucheng09/Selective_Context |
| The Token Company | thetokencompany.com |
| TokenCrush | tokencrush.ai |
| CompactPrompt paper | arxiv.org/abs/2510.18043 |
| Prompt Compression Survey | arxiv.org/abs/2410.12388 |
| LLM inference price trends | epoch.ai/data-insights/llm-inference-price-trends |
| LLM API market size | menlovc.com/perspective/2025-mid-year-llm-market-update |
