Metadata-Version: 2.4
Name: provenmem
Version: 0.1.2
Summary: Durable, provenanced memory for LLM apps: explicit gated writes, typed memories, and budgeted recall packets with receipts. The context window is a cache, not a database.
Author: Usman
License-Expression: MIT
Project-URL: Homepage, https://github.com/Sicatho/memvault
Project-URL: Issues, https://github.com/Sicatho/memvault/issues
Keywords: llm,memory,rag,retrieval,agents,context,provenance,bm25,embeddings,ollama
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: hypothesis>=6; extra == "dev"
Dynamic: license-file

# provenmem

[![CI](https://github.com/Sicatho/memvault/actions/workflows/ci.yml/badge.svg)](https://github.com/Sicatho/memvault/actions/workflows/ci.yml)

**Durable, provenanced memory for LLM apps. The context window is a cache — provenmem is the database.**

Most LLM apps "remember" by stuffing history back into the prompt: every turn, every standing instruction, every fact, every time. That context-window-as-database pattern gets slower, noisier, and more contradictory as the app lives — and when you finally trim it, things break quietly, because nobody knows which tokens were load-bearing.

provenmem replaces accumulation with a discipline:

```
conversation, decisions, tool results
            ↓  explicit remember() — no silent auto-ingest
durable corpus (SQLite) + provenance + supersession
            ↓  recall(query, budget=1200)
a small, bounded packet with receipts
            ↓
        your prompt
```

What makes it different from a plain RAG store is that **memory claims come with receipts**:

- **Writes are explicit and provenanced.** `remember()` requires a `source`. Nothing is stored unless your app deliberately stored it, and every memory knows where it came from.
- **Recall is budgeted and accounted.** You set a character budget for the **rendered packet** — `len(packet.text) <= budget` always, provenance framing included — and the receipt reports exactly what was returned (`rendered_chars`, plus `used_chars` for the memory text alone), what matched but was dropped (by budget or limit), and what was truncated. Each hit carries its own `why` — matched terms, cosine, or fused ranks.
- **Empty is honest.** An unrelated query returns `status: "empty"` with a reason — never the least-irrelevant memory dressed up as an answer, and never `ok` with zero hits. "Empty" distinguishes *nothing matched* from *matches exist but your filters excluded them* (`matches_withheld_by_state`), from *matches existed but none fit your budget*, and from *your query had no searchable terms*.
- **The gap is stated.** Every packet carries a `gap` line naming matching memories it did *not* include, so downstream logic can tell "complete answer" from "budget-truncated view".
- **Nothing vanishes silently.** `forget()` tombstones with a reason that can never be overwritten by a later forget (`hard=True` removes the row and overwrites its pages immediately, sidecars included, and says so on the receipt when it cannot — see the limits on what deletion can and cannot reach); `supersede()` replaces a memory while keeping the old one on the record, out of recall. Retracted and superseded history each require their own explicit flag to resurface (`include_retracted` / `include_superseded`), and every returned hit carries its `state`.
- **Memories are claims, not facts.** Packets carry provenance so *your* app decides what to trust; provenmem never asserts that a remembered thing is true. "Provenance" here means a caller-supplied label plus timestamps and lineage — it records *what your app said the origin was*, and does not authenticate that origin. Because memory text is untrusted by definition, `packet.text` and the CLI's *display* output (`recall`, `list`) escape control characters, every line terminator, and bidi overrides — a stored memory cannot forge an extra provenance line, rearrange the frame around it, or emit raw ANSI into your prompt or terminal.
- **A warning is not a failure.** Receipts carry `warning` (advisory: something worth knowing, like NUL bytes stripped from your text) separately from `complete` (whether the action actually happened). Only an incomplete action exits non-zero from the CLI.
- **Flags must be real booleans.** `hard='no'` and `include_superseded='false'` are truthy in Python, so a config or env string threaded into a flag would silently mean the opposite of what it reads. Every boolean parameter is type-checked — most sharply the one irreversible operation in the library.

Zero runtime dependencies. Embeddings, when you want them, use any Ollama or OpenAI-compatible endpoint over plain HTTP — no SDK.

## Status: designed + built, verification in process

This project uses the vocabulary of [provenmap](https://github.com/Sicatho/provenmap) on itself —
four independent axes, each false unless it holds evidence:

| axis | | what would earn it |
|---|---|---|
| **designed** | ✅ | the behavioral contract is written down and was adversarially reviewed |
| **built** | ✅ | the code exists: 232 tests, CI green on Linux + Windows × Python 3.9/3.11/3.13 |
| **connected** | — | the standalone package is not yet integrated into a production or external workflow |
| **verified** | — | no live production run has proven it |

So: **v0.1, designed and built, not yet proven in production use.** The discipline it encodes comes
from real work; this packaged implementation has not yet earned the last two axes. Treat it as a
sharp v0.1 rather than a battle-tested dependency, and please report what breaks.

## Install

```bash
pip install provenmem
```

*(The repository is still named `memvault`, the original working name. PyPI reserves that name as too close to an existing `mem-vault`, so the package, module, and command are all `provenmem`.)*

## Quickstart (library)

```python
from provenmem import Vault

v = Vault("team.vault")

v.remember("Billing uses the Postgres database; decided after the Q2 outage review",
           kind="decision", source="arch sync 2026-08-04", tags=["billing"])
v.remember("Never run migrations on Friday afternoons",
           kind="hazard", source="incident 2026-03-14 postmortem")

packet = v.recall("what database does billing use?", budget=1200)

packet.status        # "ok"
packet.text          # prompt-ready: "[mem-1 decision | arch sync 2026-08-04 | ...] Billing uses..."
packet.hits[0].why   # "matched: database, billing" — in query order, only terms that genuinely matched
packet.gap           # "" or e.g. "2 matching memories dropped by the 1200-char budget: mem-7, mem-9"
packet.receipt       # engine, query terms, matched/returned/dropped/truncated counts, budget accounting
packet.hits[0].state # "active" | "superseded" | "retracted" | "superseded+retracted"
```

Supersede instead of overwrite, forget with a reason:

```python
rec, receipt = v.remember("We deploy on Heroku", kind="fact", source="onboarding doc")
v.supersede(rec.id, "We deploy on Fly.io since May", source="platform RFC-12")
# tags/kind carry over unless you pass them; tags=[] clears them (redaction)
v.forget("mem-3", reason="belonged to the old project")     # tombstone
v.forget("mem-4", hard=True)                                # erase from the vault file (see limits)
```

## Quickstart (CLI)

```bash
provenmem remember "Billing uses Postgres" --kind decision --source "arch sync"
provenmem recall "billing database" --budget 800
provenmem show mem-1
provenmem list --all          # includes superseded/retracted, labeled
provenmem stats
provenmem reindex             # (re-)embed anything missing or stale
provenmem purge               # force residual deleted data out of the file
```

## Retrieval engines

- **BM25 (default, zero-dep):** deterministic lexical ranking. Only documents sharing a query term score at all — the source of honest emptiness. The tokenizer is unicode-aware: Latin accents fold (`café` ↔ `cafe`), Japanese voicing marks are preserved because they change the word (`バ` ≠ `パ`), and space-less scripts (Chinese, kana, Hangul, Thai, Lao, Khmer, Myanmar) are indexed by character bigrams so non-Latin memories are searchable. **Known limit:** accent folding is deliberate for Latin, so scripts where diacritics are phonemic rather than decorative — Vietnamese tone marks especially — will collide (`bạn` and `bản` both fold to `ban`). Use the embedding lane for those.
- **Embeddings (optional, no extra packages):** point provenmem at any Ollama (`/api/embed`) or OpenAI-compatible (`/v1/embeddings`) endpoint:

```python
from provenmem import Vault, Embedder

v = Vault("team.vault",
          embedder=Embedder(endpoint="http://localhost:11434", model="nomic-embed-text"))
```

```bash
export PROVENMEM_EMBED_ENDPOINT=http://localhost:11434
export PROVENMEM_EMBED_MODEL=nomic-embed-text
provenmem recall "billing database" --engine hybrid
```

With an embedder configured, `engine="auto"` fuses BM25 and cosine ranks (Reciprocal Rank Fusion, deterministic) — and the `engine_used` label always describes the scores actually returned: it says `hybrid` only when fusion really happened. The embedding model's identity is stored with every vector, so a model switch is *visible*: vectors from a different model **or different dimensions** (a re-pulled model) are skipped and reported in `receipt["degraded"]`, never silently mixed or silently dropped. If the embedder is unreachable — or answers with garbage — recall **degrades loudly**: with `engine="auto"`/`"hybrid"` it falls back to BM25 and says so in `receipt["degraded"]` (an explicit `engine="embedding"` raises instead, since silently answering lexically would misrepresent what you asked for). Writes record `embedding: "pending:<reason>"`, and `provenmem reindex` repairs everything stale — never embedded, wrong model, or wrong dimensions after a re-pull. Semantic hits at or below a similarity floor (`min_similarity`, default 0.25) are treated as non-matches, so an unrelated query stays honestly empty in every engine.

An api key is never sent over plain `http://` to a non-local host — that is a typed refusal, not a warning. Stored vectors are stamped with `style:model@host`, so pointing at a *different endpoint* serving the same model name is visible (and prompts a reindex) instead of silently mixing vector spaces.

For OpenAI-compatible endpoints from the CLI, also set `PROVENMEM_EMBED_STYLE=openai` (and `PROVENMEM_EMBED_API_KEY` if the endpoint needs one). Redirects are refused rather than followed: `urllib` would re-send your `Authorization` header to the new host, so a single 302 from a compromised or MITM'd endpoint could exfiltrate the key. Endpoint URLs are redacted (userinfo and query string) before they appear in any error, receipt, or log, so a key carried in the URL never travels with a packet. Response bodies are capped at 32 MB.

## Memory kinds

`fact` · `decision` · `preference` · `episode` · `hazard` · `note` — a closed set, because "typed memories" only means something if the types are stable. (`hazard` is the one teams forget to want: the hard-won lesson — *never run migrations on Friday* — that pays for the whole system the first time it surfaces at the right moment.)

## MCP server: coming soon

A Model Context Protocol server (`provenmem serve-mcp`) is planned for v0.2, so agent frameworks can use a vault directly as a memory tool. The library and CLI are the stable core it will wrap.

## Honest limits

- **Vault files are not encrypted.** Memories are stored as plaintext in SQLite. New files are created owner-only (`0600`) on POSIX; on Windows they inherit directory ACLs. Anyone who can read the file can read every memory — put it somewhere appropriate and use disk encryption if the contents are sensitive.
- **What `hard=True` can and cannot reach.** It deletes the row, its embedding, and overwrites the freed pages (`secure_delete`), then checkpoints and vacuums so nothing lingers in the `-wal` sidecar — and the receipt carries a warning when that could not complete. It cannot reach backups, filesystem snapshots, replicas, or blocks retained by SSD wear-levelling. Treat it as "removed from this vault", not as forensic erasure.
- **Measured scale.** Recall ranks the whole corpus in-process on every query, so latency grows linearly — measured at ~21 µs per record. On a 2026 laptop (Python 3.14, BM25, no embedder, warm process, median of 9): **1k records ≈ 20 ms, 5k ≈ 102 ms, 10k ≈ 211 ms, 25k ≈ 548 ms** per recall. An independent reviewer measured 19.7 / 103 / 209 / 542 ms on their own hardware. Comfortable to a few thousand memories, usable to ~10k, too slow for interactive use much beyond that. It is not web-scale search and has no ANN index.
- BM25 is lexical: paraphrases need the embedding lane. The budget is **characters of the rendered packet** — `len(packet.text) <= budget` always, framing included — not model tokens (≈4 chars/token for English; far fewer for CJK/emoji). A source longer than 80 chars is abbreviated on every *display* surface — `packet.text` and the CLI — while `hit.source` and `--json` keep the full value, because those are data rather than display. The unit is Python characters, so astral text (emoji, many CJK glyphs) costs several UTF-8 bytes each: budget by bytes or tokens yourself if that is your real constraint. A single over-budget memory is returned truncated — counted in `receipt["truncated_hits"]` and named in the gap; if not even the provenance frame fits, the packet is `empty` with `matches_did_not_fit_budget`.
- Packing is greedy in rank order: a lower-ranked small memory may fill space a higher-ranked large one couldn't — deliberate, and the dropped one is named in the gap.
- Concurrency: single-machine safe — threads and multiple processes coordinate through SQLite WAL plus immediate write transactions. A lost race surfaces as a typed refusal (`already_superseded`, or `vault_busy` past the lock timeout), never as two "current" versions of one memory. It is not a networked multi-writer service.
- While a vault is open, recent writes live partly in SQLite's `-wal` sidecar file; copy or back up a vault after closing it, or copy all three files. (Hard deletion is the exception: it checkpoints and vacuums immediately, so erased text does not linger in the sidecar.)
- Recall ranks the full corpus once per query, then partitions returnable hits from state-withheld ones — so the gap can never disagree with the hits, and one recall costs at most one query embedding. A consequence worth knowing: BM25 statistics are computed over the whole corpus including superseded and retracted records, so adding or retracting a memory shifts scores slightly. Scores stay deterministic; they are not comparable across vaults.
- `packet.text` is the prompt-safe rendering (escaped). `hit.text` and the `--json` output carry the memory **verbatim**, by design — if you build a prompt from those, escape them yourself.
- provenmem stores and retrieves; it does not verify. Whether a remembered claim is still *true* is your application's judgment — that's why everything carries provenance and timestamps.

## Origin

Built for my own workflow: the memory discipline in Ticos, a local-first LLM runtime, keeps its model honest by making every memory write explicit and every recall receipted — and a token census of our own runs found the context-as-database pattern silently wasting ~37k tokens across ten runs on one identical repeated context block. provenmem is that discipline, separated out as a simple tool in case it's useful elsewhere.

## License

MIT
