Metadata-Version: 2.4
Name: mdsift
Version: 0.4.0
Summary: Sift markdown workspaces (multi-repo capable): BM25F field-weighted search, structural grep, graph exploration (PPR), workspace vocabulary, and a token-budgeted gather opening move for AI agents. Live-updating MCP server, no embeddings required.
Project-URL: Homepage, https://github.com/some-where-over-the-rainbow/mdsift
Project-URL: Issues, https://github.com/some-where-over-the-rainbow/mdsift/issues
Author: mdsift contributors
License-Expression: MIT
License-File: LICENSE
Keywords: agent,bm25f,claude-code,copilot,keyphrase,knowledge-graph,markdown,mcp,multi-repo,pagerank,retrieval,search,toon,tree-sitter
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: tree-sitter-markdown>=0.3
Requires-Dist: tree-sitter>=0.21
Requires-Dist: watchdog>=3.0
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Description-Content-Type: text/markdown

# mdsift

**Sift a markdown workspace.** Field-weighted keyword search, structural grep,
and multi-layer graph exploration for AI agents — live-updating, embedding-free,
exposed as an MCP server.

Point it at a folder of markdown. Your agent gets twelve tools that stay fresh
as files change, with sub-10 ms queries and zero model downloads.

## Why not just BM25 + a wikilink graph?

Plenty of tools do flat BM25 over markdown. mdsift differs in four ways:

1. **BM25F, not BM25.** Tree-sitter parses every file into fields — title,
   headings, code identifiers, link text, body — and ranking weights them
   (a hit in a title or a function name beats a hit in prose). Code
   identifiers are subtokenized, so both `refreshAccessToken` and `token` match.
2. **A multi-layer graph with agent-controlled weights.** Three edge families —
   explicit links (markdown + wiki-links), folder hierarchy, and statistical
   TF-IDF similarity — queried together via Personalized PageRank. The agent
   dials per-edge-type multipliers per query: "related by links" vs "related
   by vocabulary" vs "related by location" are different questions.
3. **Structural grep.** Regex scoped to *where* in the document: only inside
   code blocks (optionally by language), only in headings, only in link text.
   Powered by ripgrep underneath, so it is fast and always index-fresh.
4. **No embeddings.** Pure lexical + graph statistics. Instant cold start,
   fully offline, no GPU, no model cache.

## Install

```bash
pip install mdsift          # once published; from source: pip install -e .
# ripgrep must be on PATH:  apt install ripgrep | brew install ripgrep
```

## Use

```bash
# MCP server (stdio) for Claude Desktop / Claude Code / any MCP client
mdsift-server /path/to/workspace

# multi-repo: several roots become logical workspaces; paths are "alias/rel"
mdsift-server frontend=~/code/web backend=~/code/api docs=~/code/docs

# CLI
mdsift /path/to/workspace search  '{"query": "auth retry", "top_n": 5}'
mdsift /path/to/workspace grep    '{"pattern": "retry.*backoff", "scope": "code", "lang": "python"}'
mdsift /path/to/workspace explore '{"seeds": {"docs/auth.md": 1.0}, "preset": "semantic"}'
mdsift /path/to/workspace clusters '{"min_size": 5}'
```

MCP client config:

```json
{"mcpServers": {"mdsift": {"command": "mdsift-server", "args": ["/path/to/workspace"]}}}
```

## Tools

| Tool | Use when |
|---|---|
| `gather` | **Start here.** Viewpoint-aware (same question, audience-appropriate evidence). One call: topic map + best sections (extract-compressed) + graph-related files. Replaces the search→read→search gathering loop. |
| `ontology` | Derived concept taxonomy: broader/narrower terms, full-corpus role-typed grounding with coverage stats, mined or curated definitions, bounded-context boundaries, cross-context synonym/variant alignment, non-markdown implementation grounding, glossary curation round-trip with mermaid export, and LLM-entity injection with AESOP-style reconciliation (docs/entity-injection.md). |
| `trace` | Traceability over workspace IDs (REQ_/ADR-/CHK_ + custom families): coverage report with dangling/uncovered/orphan invariants, or resolve any ID both directions — definition, references by role, co-occurring IDs. |
| `vocabulary` | The workspace's own language: shared keyphrases, code identifiers, acronyms. Also sampled into the initialize instructions (round-zero). |
| `search` | Fuzzy/conceptual queries; viewpoint lens (stakeholder/data/operational/technical/security); BM25F over weighted fields. Hits include the best-matching **section snippet**, so agents often skip the full-file read. `expand=true` merges graph-related files. |
| `read_section` | Fetch one section by heading (fuzzy-matched) — the token-efficient follow-up to search. |
| `grep` | Exact strings / regex. `scope=code\|headings\|body\|links`, `lang` filter, path glob/regex. Never stale. |
| `deps` | Direct neighborhood of one file: links in/out, folder relations, similarity edges. |
| `explore` | PPR from seed files. Presets `structural` / `semantic` / `spatial` / `balanced`, or explicit weights (`0` prunes a layer). |
| `clusters` | Topic map: graph communities + TF-IDF labels (preset `topics`). |
| `relate` | Weighted shortest connection between two files. |
| `status` | Corpus, vocabulary, and edge counts. |

The server ships MCP `instructions` carrying a live workspace-vocabulary sample (ingested at round zero, before any tool call) and steering agents to gather → read_section with session ids and budgets, and search responses embed hints pointing to graph neighbors that lexical results missed. Composition patterns agents use well: `grep` for candidates → `search` to rank
them; `search` for seeds → `explore` to widen context; `clusters` on first
contact with an unfamiliar workspace.

See [docs/token-efficiency.md](docs/token-efficiency.md) for an honest
analysis of when this saves agent tokens — the short version: ranked results
and PPR exploration replace multi-turn read loops, and section snippets attack
the file-consumption side, where most tokens actually live.

All tools accept `format` (`toon` default — 45–60% fewer tokens on tabular
results — or `json`), `max_tokens` (server-side response budget), and search
accepts `session` (repeat-snippet dedup), `granularity` (`file`/`section`),
`tags` (frontmatter filter), and `recency` (opt-in freshness boost), and `snippet_compress` (`light` default strips markdown syntax from previews; `extract` keeps only query-relevant sentences — model-free extractive compression via the index's IDF).
Frontmatter parsing covers Agent Skill (SKILL.md) files: multi-line block-scalar
descriptions and nested `metadata:` are parsed, and skill `name`/`description`
are indexed so skills are searchable by capability. Format coverage is
tested against realistic CLAUDE.md (including `@path` imports, which become
graph edges), AGENTS.md (root + nested per-directory), MADR and Nygard-style
ADRs, and SKILL.md fixtures — see tests/test_agent_formats.py — plus the
full GitHub Copilot customization surface (copilot-instructions.md,
*.instructions.md/applyTo, *.prompt.md, *.chatmode.md, *.agent.md,
.github/skills), Cursor .mdc rules (indexed by default), and Claude Rules —
see tests/test_copilot_formats.py. Hidden directories (.github, .cursor,
.claude) are scanned, grepped, and watched. Multi-repo mode gives every root
a workspace alias: one shared graph (cross-repo similarity, clusters, and
wiki-links), per-repo links stay repo-local, and `path_glob="alias/**"`
scopes any tool to one repository.

## Performance

Measured on a synthetic 2,000-file workspace (`python bench.py` reproduces):

| Path | Latency |
|---|---|
| Cold build (incl. vocabulary extraction) | ~4.4 s |
| `search` (plain / with compressed snippets) | p50 ≈ 3.5 / 5.0 ms |
| `explore` (PPR over 4 layers) | p50 ≈ 5 ms |
| `grep` scoped | p50 ≈ 40 ms (ripgrep-bound) |
| `clusters` cold / cached | ~270 ms / ~4 ms |
| `ontology` build (cached; rescan only when concepts change) | ~150–280 ms |
| viewpoint classification (cached, live-invalidated) | ~40 ms |
| **Live file update** (parse → index → graph → vocab) | **p50 ≈ 0.7 ms** |

How: columnar postings (`int32` ids + `uint16` tfs) with numpy-vectorized
scoring and a delta/tombstone tier for updates; per-edge-type scipy CSR
matrices summed at query time so agent-supplied weights cost microseconds;
similarity edges maintained via rare-term candidate shortlisting (never
all-pairs); folder nodes with capped sibling edges keep the graph linear.
One index serves both ranking and similarity vectors.

## Architecture

```
watchdog (200 ms debounce, one observer per workspace root)
  └─ tree-sitter (block + inline grammars) + frontmatter parser
       ├─ BM25F index (files)     ── search, similarity vectors
       ├─ BM25F index (sections)  ── granularity="section"
       ├─ span index              ── structural grep, read_section, snippets
       ├─ vocabulary              ── RAKE phrases, acronyms, anchor terms
       └─ graph                   ── link / sibling / parent / similar layers
                                     └─ PPR (explore, gather), clusters, relate
Output: TOON encoding, snippet compression (light/extract), token budgets,
session dedup. Multi-root: paths are alias/rel over one shared graph.
```

Update cost is O(changed file + shortlisted neighbors), never O(corpus).

## Rust migration path

The data layout (interned ids, columnar postings, CSR layers) and the MCP JSON
contract port 1:1 to Rust. First step if ever needed: move `index.search` and
`graph.ppr` into a PyO3 extension; nothing else changes.

## Hook mode

`mdsift-hook` intercepts whole-file markdown reads (Claude Code
PreToolUse; opencode plugin) and answers with the file's heading TOC plus
scoped recovery moves — enforcement instead of guidance, effective even
under skip-permissions. SessionStart injects vocabulary from the committed
ontology snapshot. See docs/hook-mode.md.

## Evaluation

`eval/` contains a four-arm design for measuring the token effect on coding
agents, including a deterministic generator for a realistic multi-repo
fullstack workspace with full agentic harnesses and planted gradeable tasks.
See eval/README.md.

## Development

```bash
pip install -e ".[dev]"
pytest tests/ -q      # 69 tests
python bench.py       # correctness + latency suite on 2,000 generated files
```

## License

MIT. Dependencies: numpy/scipy (BSD-3), tree-sitter + grammar (MIT),
watchdog (Apache-2.0); ripgrep (MIT/Unlicense) is invoked as a subprocess,
not bundled.
