Metadata-Version: 2.4
Name: pyintake
Version: 0.0.3
Summary: Knowledge-base synthesis: document intake, chunking, embeddings, vector storage, hybrid retrieval — sibling to pycharter.
Author-email: Optophi <contact@optophi.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/optophi/pyintake
Project-URL: Documentation, https://optophi.github.io/pyintake/
Project-URL: Repository, https://github.com/optophi/pyintake
Project-URL: Issues, https://github.com/optophi/pyintake/issues
Project-URL: Contributing, https://github.com/optophi/pyintake/blob/main/CONTRIBUTING.md
Keywords: knowledge-base,rag,embeddings,vector-search,hybrid-retrieval,document-intake,chunking,llm-extraction
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3,>=2.6
Requires-Dist: PyYAML<7,>=6.0
Provides-Extra: api
Requires-Dist: fastapi<1,>=0.110; extra == "api"
Requires-Dist: uvicorn[standard]<1,>=0.27; extra == "api"
Provides-Extra: ui
Requires-Dist: pyintake[api]; extra == "ui"
Requires-Dist: httpx<1,>=0.27; extra == "ui"
Requires-Dist: aiofiles>=23.0; extra == "ui"
Provides-Extra: db
Requires-Dist: sqlalchemy<3,>=2.0; extra == "db"
Requires-Dist: alembic<2,>=1.18.5; extra == "db"
Provides-Extra: postgres
Requires-Dist: pyintake[db]; extra == "postgres"
Requires-Dist: psycopg[binary]<4,>=3.1; extra == "postgres"
Provides-Extra: pdf
Requires-Dist: pypdf<7,>=6.14.2; extra == "pdf"
Provides-Extra: ai
Provides-Extra: embeddings-local
Requires-Dist: fastembed<1,>=0.4; extra == "embeddings-local"
Provides-Extra: embeddings-voyage
Requires-Dist: httpx<1,>=0.27; extra == "embeddings-voyage"
Provides-Extra: vector
Requires-Dist: sqlite-vec<0.2,>=0.1.9; extra == "vector"
Provides-Extra: local
Requires-Dist: pyintake[db,embeddings-local,vector]; extra == "local"
Provides-Extra: kb
Requires-Dist: fsspec>=2026.7.0; extra == "kb"
Requires-Dist: s3fs>=2024.2; extra == "kb"
Requires-Dist: httpx<1,>=0.27; extra == "kb"
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.2; extra == "mcp"
Provides-Extra: all
Requires-Dist: pyintake[ai,api,db,embeddings-voyage,kb,local,mcp,pdf,postgres,ui,vector]; extra == "all"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.5.3; extra == "docs"
Requires-Dist: mkdocs-material>=9.5.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
Requires-Dist: pymdown-extensions>=11.0.1; extra == "docs"
Provides-Extra: dev
Requires-Dist: pyintake[all,docs]; extra == "dev"
Requires-Dist: pytest>=9.1.1; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=7.1.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.4.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.0.0; extra == "dev"
Requires-Dist: hypothesis<7,>=6.100; extra == "dev"
Requires-Dist: ruff>=0.16.3; extra == "dev"
Requires-Dist: mypy>=2.3.1; extra == "dev"
Requires-Dist: pre-commit>=3.0.0; extra == "dev"
Requires-Dist: httpx>=0.27.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Dynamic: license-file

# PyIntake

**Knowledge-base synthesis for Python** — document intake, chunking,
embedding generation, vector + full-text storage, hybrid retrieval,
suggestion-queue governance, resource-binding lifecycle, KB
versioning, and a Next.js UI on top of it all. Sibling to
[`pycharter`](https://github.com/optophi/pycharter): pyintake owns the
**synthesis** layer; pycharter owns the **governance** layer
(contracts, concept registry, field bindings).

- **Repository:** [github.com/optophi/pyintake](https://github.com/optophi/pyintake)
- **Status:** Phases 1, 2, 4, 5 of the KB repurpose are landed — full
  pipeline, SQL-backed stores, expanded API, and a working UI scaffold.
  Phase 3 (concept-scheme registry migration) is intentionally skipped
  per the conservative resolution of pyintake's §4b open decision.
- **Roadmap:** see [`docs/proposals/`](docs/proposals/README.md) for
  the per-phase plan and the pycharter coordination items.

## Install

```bash
pip install pyintake                # core (pydantic + PyYAML only)
pip install 'pyintake[api]'         # + FastAPI / uvicorn HTTP surface
pip install 'pyintake[ui]'          # + UI server proxy (FastAPI + httpx)
pip install 'pyintake[local]'       # the offline tier: db + persistent indexes + local embeddings
pip install 'pyintake[db]'          # + SQLAlchemy 2 + Alembic (persistent stores)
pip install 'pyintake[postgres]'    # + psycopg[binary] for Postgres
pip install 'pyintake[pdf]'         # + pypdf for PDF document sources
pip install 'pyintake[all]'         # everything (api, ui, db, postgres, pdf, ai, vector)
```

``[local]`` is the recommended entry point for a self-contained
knowledge base: one install, no server, no API key, and no network at
query time. It brings a local embedding model, a durable `sqlite-vec`
vector index and a durable FTS5 full-text index. The ``ai`` extra still
reserves a namespace for the LLM adapters that land in a later round.

### Indexes that survive a restart

`vector.backend` and `fulltext.backend` default to `inmemory`, which is
right for tests, bundles and small in-process KBs — and means both
indexes are rebuilt from scratch every time the process starts. A
deployment wants the `sqlite` backends instead:

```yaml
vector:   {backend: sqlite}   # vectors in `embeddings`, vec0 index derived from it
fulltext: {backend: sqlite}   # FTS5, derived from `chunks`
```

That is what `data/seed/recipes/local-sqlite.yaml` selects. Both indexes
are **projections** of durable tables, so a damaged one is a rebuild
rather than a re-ingest:

```bash
pyintake db check   --recipe local-sqlite.yaml   # does each index agree with its source?
pyintake db reindex --recipe local-sqlite.yaml   # rebuild both; needs no embedding model
```

The vector index is a brute-force scan, so query cost is linear in
corpus size — measured at ~46ms per top-10 query over 100k x 384 vectors.
Comfortable to ~100k chunks and usable to ~500k; beyond that the answer
is the Postgres tier:

```yaml
vector:   {backend: postgres}   # pgvector, HNSW index
fulltext: {backend: postgres}   # tsvector + GIN, ranked by ts_rank_cd
```

That tier needs `pip install 'pyintake[postgres]'` and the pgvector
extension on the server (`CREATE EXTENSION vector;`). Its search is
**approximate** where the other two are exact — an HNSW graph can miss a
true neighbour, which is the trade that makes ten million vectors
searchable. See
[Retrieval backend tiers](docs/reference/retrieval-backend-tiers.md) for
what each backend is for and what is stood behind.

## CLI

The ``pyintake`` CLI follows the fleet multi-subcommand pattern (same
as ``pycharter``):

```bash
pyintake ingest notes.md paper.txt --query "arctic tern" -k 3
pyintake api                                            # start FastAPI (port 8100)
pyintake db init                                        # alembic upgrade head
pyintake db seed                                        # bundled recipes + default KBVersion
pyintake db current                                     # show active revision
pyintake db check --recipe RECIPE                       # verify derived indexes against their sources
pyintake db reindex --recipe RECIPE                     # rebuild derived indexes from stored data
pyintake ui dev                                         # Next.js dev server (port 3100)
pyintake ui build                                       # static export → ui/static/
pyintake ui serve                                       # serve built UI + proxy /api/*
pyintake worker run                                     # AutoExtractionSink loop
pyintake worker retention --dry-run                     # drop embeddings older than each contract's retention_period
pyintake mcp --scope internal                           # KB as MCP tools (stdio)
```

### The MCP server's scope is an operator decision

`pyintake mcp` binds its policy scope when it starts, and every tool
closes over it:

```bash
pyintake mcp                        # internal (default) — PII values withheld
pyintake mcp --scope public         # only public/embeddable facts
pyintake mcp --scope pii            # releases PII values; opt in deliberately
```

No tool takes a scope, an `include_pii` flag, or any other policy
argument. A tool argument is filled in by the language model, so a
policy control expressed as one is a control the model grants itself —
a prompt-injected agent would simply ask for the wider scope. Widening
means restarting the server with different flags.

`--requester NAME` labels the agent sessions the server records, so the
Trust Ledger can attribute them to the consumer that asked.

### Structured sources ground more precisely

A Python module is a document like any other — discovered by the same
`documents` globs (add `**/*.py` to `include`), loaded by a source,
chunked and indexed — read a second time for its structure. Symbols
become chunks cut at definition boundaries with exact **source spans**,
and relationships (`defines`, `imports`, `calls`, `inherits`,
`references`) become graph edges marked with how they were established:
`extracted`, `inferred`, or `ambiguous`. Uncertain targets are omitted
rather than guessed.

```bash
pyintake ingest src/shop/billing.py --query "how is an order billed" -k 3
```

The same tools answer: `kb_search` hits carry `symbol` and `span`,
`kb_entity symbol:shop.billing.bill` lists where it is defined and who
calls it, `kb_graph_neighborhood` walks callers (`direction: incoming`)
or callees (`outgoing`) with edge provenance, and `kb_agent_query`
answers "what does `bill` depend on" from the graph. No new tool, and a
public-scoped server sees neither symbols, spans nor paths. See
[Structured grounding](docs/guides/structured-grounding.md).

### Every list tool is bounded

A tool answers into the model's context window, so no tool returns the
whole knowledge base. List tools page — `kb_concepts`, `kb_search`, and
`kb_ontology` take `limit` / `offset` and return `total` plus a
`next_offset` — and every bound is a ceiling the caller cannot raise,
because a limit the model can set is a limit it can set to ten thousand.

The governed vocabulary is also exposed as MCP **resources**
(`kb://concepts`, `kb://ontology`), so a client can attach it to context
once instead of spending a tool call on it.

### Agents can contribute, under review

`kb_propose_binding` lets an agent propose that a record's field means a
governed concept. The proposal is **inert**: it lands in the existing
review queue as `pending`, attributed to the server's `--requester`, and
changes nothing until a human decides. `kb_suggestions` reads the queue
back. Neither needs a flag, because neither changes the KB.

Deciding is a separate question:

```bash
pyintake mcp --allow-review     # adds kb_decide_suggestion, kb_resolve_fuzzy
```

Off by default. An agent that can propose *and* accept is an agent
writing directly, with the review queue as decoration — so turning that
on is an operator's call, exactly like the policy scope.

Provenance is never the model's to state: the proposer recorded on a
suggestion is the server's requester, not a tool argument.

Environment variables:

| Variable | Default | Purpose |
|---|---|---|
| `PYINTAKE_API_PORT` | 8100 | API listen port |
| `PYINTAKE_UI_PORT`  | 3100 | UI dev / serve port |
| `PYINTAKE_API_URL`  | `http://127.0.0.1:8100` | UI → API base URL |
| `PYINTAKE_DATABASE_URL` | `sqlite:///pyintake.db` | SQLAlchemy URL |
| `PYINTAKE_RECIPE` | bundled `default.yaml` | Active recipe path |

## Use it from Python

```python
from pyintake import IntakeClient

client = IntakeClient.from_recipe("data/seed/recipes/default.yaml")
client.ingest("notes.md")
client.ingest("paper.txt")

for hit in client.retrieve("arctic tern migration", k=5):
    print(f"{hit.score:.3f}  {hit.document_id}  {hit.text[:80]}…")
```

## Use it from HTTP

The v1 surface mirrors the IntakeClient:

```
GET    /healthz
POST   /api/v1/ingest                  POST   /api/v1/suggestions/{id}/{accept,reject,defer}
GET    /api/v1/retrieve                GET    /api/v1/bindings
GET    /api/v1/documents               POST   /api/v1/bindings/find
GET    /api/v1/documents/{id}          POST   /api/v1/bindings/deprecate
DELETE /api/v1/documents/{id}          GET    /api/v1/kb-versions
GET    /api/v1/suggestions             POST   /api/v1/kb-versions/{id}/activate
GET    /api/v1/suggestions/{id}        POST   /api/v1/kb-versions/{id}/migrate
                                       GET    /api/v1/recipes{,/current,/{name}}
```

```bash
curl -X POST http://127.0.0.1:8100/api/v1/ingest \
     -H 'Content-Type: application/json' \
     -d '{"text": "the arctic tern migrates between the poles every year"}'

curl 'http://127.0.0.1:8100/api/v1/retrieve?q=arctic+tern&k=3'
```

### Bearer tokens are an operator decision too

By default the API authenticates nobody: every caller is anonymous and
reads at the **internal** tier, which is the same deployment you get
today. Set `PYINTAKE_API_TOKENS` and three things become true at once:

```bash
# subject:token:scope,scope — entries separated by ';'
export PYINTAKE_API_TOKENS='svc-indexer:s3cr3t-a:retrieve,ingest;web-widget:s3cr3t-b:public'
pyintake api
```

| Caller | Served as |
| --- | --- |
| `Authorization: Bearer s3cr3t-a` | `svc-indexer`, scopes `retrieve,ingest` — internal tier |
| `Authorization: Bearer s3cr3t-b` | `web-widget`, scope `public` — public tier |
| no `Authorization` header | anonymous at the **public** tier |
| an unrecognised token | `401` with a `WWW-Authenticate: Bearer` challenge |

The third row is the point. A scope that literally names a policy tier
clamps the caller to it, so an unauthenticated request now stops at
`public`: `/retrieve` answers `403`, `/search` withholds chunk snippets,
suggestions withhold the quoted record. Give a token no scopes at all
(`svc:tok:`) and it reads at the internal ceiling, like today's
anonymous caller. Comparison is constant-time, and a token is never
written to a log or echoed in an error — a rejected one is recorded by a
truncated digest.

**The token table is a secret.** Keep it in the environment or a secret
store; never in a recipe, a bundle, or anything committed. Deployments
that authenticate some other way (JWT, mTLS, a sidecar) still override
`pyintake.api.dependencies.auth.get_auth_context` directly and are
unaffected.

## Layered architecture

```
┌───────────────┐  ┌───────────────┐  ┌───────────────┐
│      UI       │  │  HTTP wrapper │  │      CLI      │
│  (Next.js 16) │  │   (FastAPI)   │  │   (argparse)  │
└───────┬───────┘  └───────┬───────┘  └───────┬───────┘
        └──────────┬───────┴──────────────────┘
                   ▼
        ┌──────────────────────────────────┐
        │      INTAKE CLIENT (façade)      │
        │  wires backends from a recipe    │
        └─────────────┬────────────────────┘
                      ▼
        ┌──────────────────────────────────┐
        │         STATELESS CORE           │
        │  pyintake.core.{pipeline,        │
        │  retrieval} — pure functions     │
        └─────────────┬────────────────────┘
                      ▼
   ┌──────────────────┼───────────────────┐
   ▼                  ▼                   ▼
 BACKENDS         GOVERNANCE +         KB-VERSIONS +
 (chunkers,        BINDINGS +          ORCHESTRATION
  embedders,       MATERIALIZE         + CACHING
  chunk/vector/    + WORKER
  fulltext stores) (Phase 2)           (Phase 4 + 5)
   │
   ▼
 SQLAlchemy 2 + Alembic  ──►  SQLite / Postgres
 (pyintake.db)
```

Two laws (inherited from the Optophi fleet):

1. **Core is stateless** — `pyintake.core.pipeline.ingest_document`
   and `pyintake.core.retrieval.retrieve` are pure functions that
   take Protocol-typed backends as keyword arguments.
2. **Domain artifacts are configuration** — chunkers, embedders,
   stores, and recipes are selected by YAML validated through
   `pyintake.config.KBRecipe`.

## Architectural relationship with pycharter

```
            pygubernator (orchestration, planned)
                 ▲       ▲
                 │       │ consumes both
                 │       │
  pyintake ──binds── pycharter
  (synthesis)        (governance)
```

pyintake produces:

- **Chunks** (text + metadata) stamped with `(model, model_version)`
  for re-embedding idempotency.
- **Embeddings** and a hybrid retrieval surface.
- **Suggestions** + **Resource-binding candidates** — pyintake
  proposes; pycharter accepts and stores the canonical record.

pyintake calls into pycharter via the
[`PycharterClient`](src/pyintake/_protocols.py) Protocol. The
concrete HTTP adapter is the next item after the pycharter Phase 0
boundary-spec conversation completes (see
[pycharter/docs/proposals/decoupling-generalization-backlog.md](https://github.com/optophi/pycharter/blob/develop/docs/proposals/decoupling-generalization-backlog.md)
CR-1..CR-5).

## Is retrieval any good?

The unit suite pins retrieval *behaviour*; `tests/eval/` measures its
*quality* against a labelled corpus, so changing an embedding provider or
a vector backend is a measurable change rather than an act of faith.

```bash
scripts/eval.sh                       # bundled default recipe
scripts/eval.sh my-recipe.yaml        # compare another recipe
```

The default recipe uses `InMemoryHashEmbedder`, a hashing vectorizer: it
matches terms, so it answers lexical queries perfectly and paraphrases
barely at all. Swapping in a real model is what the `local-embeddings`
recipe does, and the harness is how you can tell it worked:

```
                         default (hash)    local-embeddings (BGE-small)
lexical      top-1            1.000                1.000
paraphrase   top-1            0.250                0.750
distractor   top-1            0.750                0.875
overall      nDCG             0.740                0.922
```

```bash
pip install 'pyintake[embeddings-local]'
scripts/eval.sh src/pyintake/data/seed/recipes/local-embeddings.yaml
```

## Embedding providers

| provider | extra | needs | notes |
|---|---|---|---|
| `inmemory` | — | nothing | Hashing vectorizer. The CI path and the zero-dependency default. Matches terms, not meaning. |
| `fastembed` | `embeddings-local` | ~67MB model, ONNX runtime | Real semantics offline. No key, no per-call cost, reproducible by anyone with the model. |
| `voyage` | `embeddings-voyage` | `$VOYAGE_API_KEY` | Hosted. Sends an explicit `input_type` of `query` or `document`. |

The API key is read from the environment, never from a recipe — recipes
are version-controlled.

## Documentation

Build the MkDocs site locally:

```bash
pip install 'pyintake[docs]'
mkdocs serve
```

Runnable examples under [`examples/`](examples/):

| Example | Command |
|---------|---------|
| [PDF document ingest](examples/pdf_ingest/README.md) | `pip install 'pyintake[pdf]' && cd examples/pdf_ingest && python run_demo.py` |
| [Offline KB (pycharter seam)](examples/offline_kb/README.md) | `pip install 'pyintake[kb]' pycharter && cd examples/offline_kb && python run_demo.py` |

Site guides: [PDF ingest end-to-end](docs/tutorials/pdf-ingest-end-to-end.md),
[KB ingestion (Phase A)](docs/guides/kb-ingestion.md).

## Contributing

See [`CONTRIBUTING.md`](CONTRIBUTING.md), [`AGENTS.md`](AGENTS.md), and
[`ARCHITECTURE.md`](ARCHITECTURE.md). AI assistants follow the
specifications in [`.claude/CLAUDE.md`](.claude/CLAUDE.md) and the
skills under [`.claude/skills/`](.claude/skills/).
