Metadata-Version: 2.4
Name: wiki-forest
Version: 0.1.1
Summary: Forest.Wikipedia.EN: a populated English Wikipedia terrain with sealed semantic retrieval.
Author: Forest contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/schmerbert/Forest.Wikipedia.EN
Project-URL: Repository, https://github.com/schmerbert/Forest.Wikipedia.EN
Project-URL: Issues, https://github.com/schmerbert/Forest.Wikipedia.EN/issues
Project-URL: Populated release, https://huggingface.co/datasets/schmerbertt/forest-wikipedia-en
Keywords: wikipedia,offline,retrieval,semantic-search,faiss,zim
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: libzim<4,>=3.11
Requires-Dist: lxml<7,>=6
Provides-Extra: download
Requires-Dist: huggingface-hub<2,>=0.34; extra == "download"
Provides-Extra: semantic
Requires-Dist: faiss-cpu<2,>=1.11; extra == "semantic"
Requires-Dist: huggingface-hub<2,>=0.34; extra == "semantic"
Requires-Dist: numpy<3,>=2; extra == "semantic"
Requires-Dist: sentence-transformers<6,>=5; extra == "semantic"
Provides-Extra: test
Requires-Dist: pytest<10,>=8; extra == "test"
Provides-Extra: dev
Requires-Dist: build<2,>=1; extra == "dev"
Requires-Dist: pytest<10,>=8; extra == "dev"
Dynamic: license-file

# Forest.Wikipedia.EN

**The English Wikipedia forest: populated, local, and ready for people and agents.**

Forest.Wikipedia.EN turns an offline English Wikipedia canopy into stable article
bearings, authored sections, exact passages, and bounded semantic search. It is
designed to answer *where should I look?* without pretending that retrieval
alone makes something true.

The populated release contains **8,419,072 canonical article bearings** and a
sealed semantic index. Search and canopy navigation work locally. Exact article
depth is fetched from Wikimedia only when a user or agent deliberately enters an
article, then cached by revision and content hash.

> **Canopy search gives directions, not evidence.** A consuming application
> decides what was exposed, why it was selected, and what authority it carries.

## Get the populated Forest

```bash
pip install "wiki-forest[semantic]"
wiki-forest setup
wiki-forest search "milk"
```

`setup` downloads the pinned populated release from
[Hugging Face](https://huggingface.co/datasets/schmerbertt/forest-wikipedia-en),
resumes interrupted transfers, verifies every artifact, and creates a fresh local
database. Allow about **20 GB of free disk space**.

The first semantic search arms a local warm worker (often tens of seconds on a
cold machine). Later searches reuse it. Explicit hangar controls:

```bash
wiki-forest arm
wiki-forest status
wiki-forest disarm
```

From a source checkout:

```bash
git clone https://github.com/schmerbert/Forest.Wikipedia.EN.git
cd Forest.Wikipedia.EN
python -m venv .venv
# Windows: .venv\Scripts\activate
# Unix:    source .venv/bin/activate
pip install -e ".[semantic,test]"
wiki-forest --root . identity
```

## What is populated?

| Layer | Contents | Approximate size |
|---|---|---:|
| Canopy | Kiwix English Wikipedia `all_mini`, June 2026 | 11.67 GiB |
| Semantic projection | FAISS IVF-PQ index plus SQ8 reranking codes | 3.47 GiB |
| Canonical inventory | Stable mapping for 8.4 million bearings | 0.78 GiB |
| Query model | Pinned `BAAI/bge-small-en-v1.5` snapshot | 0.13 GiB |

Personal runtime state is never part of the populated release. Hydrated article
bodies live under `sources/`, while traversal metadata and revision pins live in
`forest.db`; both begin empty on every new installation.

## Try it

Resolve an exact bearing without entering the article:

```bash
wiki-forest bearing "Milk"
```

Search the unread canopy (auto-arms the warm worker on first use):

```bash
wiki-forest search "rituals around hospitality" --limit 8
```

Inspect readiness, mounted identity, and whether the semantic worker is armed:

```bash
wiki-forest status
wiki-forest identity
wiki-forest verify --quick
```

Map before you read. `map` returns locators and labels without passage bodies.
Human section titles resolve to stored MediaWiki locators:

```bash
wiki-forest map "Milk"
wiki-forest map "Milk" "Etymology and terminology"
wiki-forest map "Milk" "Etymology_and_terminology"
wiki-forest node "Milk" "Etymology_and_terminology.p1"
```

`children` remains the full-body layer for agents that already know they want
text. Both `map` and `children` hydrate on first entry unless `--offline` is set.

Full verification reads and hashes the entire populated release and can take a
while:

```bash
wiki-forest verify
```

Successful setup and full verification write a local verification receipt.
Ordinary searches validate that receipt using the trusted packaged manifest and
artifact metadata, avoiding repeated multi-gigabyte hashing. `wiki-forest verify`
always reads and hashes every declared byte again.

Remove the downloaded canopy, indexes, local database, and hydrated articles:

```bash
wiki-forest remove-data --dry-run
wiki-forest remove-data
pip uninstall wiki-forest
```

The command disarms any warm worker, reports the exact location and size,
requires confirmation, and preserves files it does not own. Use `--yes` only for
unattended removal. It refuses to operate on a source checkout or filesystem root.

Set a custom data location with `WIKI_FOREST_HOME` or pass `--root`. Existing
checkout-based integrations remain supported:

```bash
wiki-forest --root D:\data\wiki-forest identity
```

## Acceptance

Reference timings on a populated Windows checkout (2026-08-10), measured with
`python scripts/measure_acceptance.py`. Search uses a long-lived production
worker; bearing/map/node use the ordinary CLI.

| Operation | Time |
|---|---:|
| `status` | 0.28 s |
| `bearing "Milk"` | 0.35 s |
| `map "Milk"` | 0.39 s |
| `node "Milk" "History.p1"` | 0.43 s |
| Cold arm + first search | 9.4 s |
| Warm search (median of 3) | 0.14 s |

These are not SLAs. Cold arm is dominated by loading the query encoder and
mapping the sealed index; warm search should stay well under a second once the
worker is resident. Re-run the script on your machine after `wiki-forest setup`.

## Offline boundary

The bundled `all_mini` archive contains every article's title, introduction,
and infobox, without images. Consequently:

- Bearing lookup and canopy search work offline.
- The sealed semantic index works offline after installation.
- Full authored sections and passages require network access on first entry.
- A hydrated revision is pinned and reused locally; it never silently updates.
- Hydrated source custody records include the Parsoid implementation and
  MediaWiki HTML DOM specification versions embedded in the fetched document.
- `--offline` refuses first-entry hydration when an article is not already pinned.

This is deliberate. Forest.Wikipedia.EN keeps a complete shallow world available while
allowing exact depth to grow only where someone actually walks.

## Stable addresses, readable labels

Every node has a stable machine locator. Existing addresses remain valid:

```text
article              article root
article.p1           first lead passage
History              authored section
History.p1           first passage in History
Etymology_and_terminology     section whose MediaWiki id uses underscores
```

Human titles such as `Etymology and terminology` resolve to the stored locator.
APIs also return `display_title`, so interfaces can show **Lead passage 1** or
**History — passage 1** without exposing internal locator syntax.

## Architecture

```text
canopy bearing
    → article lead
        → section map
            → section
                → subsection
                    → passage
```

- `canopy/` holds the sealed Kiwix archive and checksum.
- `sources/` holds immutable hydrated revision bodies, addressed by SHA-256.
- `projections/` holds rebuildable structural and semantic indexes.
- `forest.db` holds source custody, revision pins, and navigable nodes.
- `models/` holds the pinned query encoder used by semantic retrieval.

The production retriever unions exact and lexical ZIM bearings with FAISS
candidates, reranks a bounded neighborhood from compact SQ8 codes, then applies
diversity selection. Exact title matches cannot be displaced merely because the
approximate index omitted them.

Long-lived applications can keep the verified index and model warm:

```bash
wiki-forest arm
wiki-forest-index serve-production
```

`wiki-forest search` auto-arms a localhost NDJSON worker (preferring a checkout
`.index-venv` when FAISS lives there). The stdin worker from
`wiki-forest-index serve-production` remains available for embedded hosts. See
[DESIGN.md](https://github.com/schmerbert/Forest.Wikipedia.EN/blob/main/DESIGN.md)
for retrieval design, geometry benchmarks, integrity, and rebuild notes.

## Forest ecosystem

Forest.Wikipedia.EN is usable by itself and is also a populated backend for the
wider Forest ecosystem. Its catalog identity is `forest.wikipedia.en`. It owns
source identity and navigable Wikipedia terrain. A consumer owns the reason for
retrieval: actor, query, exposure, selection, interpretation, and consequences.

### Names

| Name | Role |
|---|---|
| `Forest.Wikipedia.EN` | Product / GitHub repository title |
| `wiki-forest` / `wiki_forest` | PyPI package, CLI, and Python import |
| `forest.wikipedia.en` | Catalog identity (`catalog_id` / `forest_id` in machine output) |

The dotted catalog scales cleanly: `forest.wikipedia.fr`, `forest.arxiv.cs`, or
`forest.docs.personal` can name other populated terrains without a new metaphor
each time.

### Related — not the same

[The_Forest](https://github.com/schmerbert/The_Forest) is the resident
**custody memory** layer (signed entries, ticketed walk, optional root). This
repository is sealed **wild Wikipedia terrain** a host may search and cite into
that store. Similarity here returns directions; it never promotes anything to
ground. Crossing into a resident’s adopted truth is the consumer’s ceremony, not
this package.

That boundary lets many applications share one sealed Wikipedia substrate
without confusing a retrieved source with locally adopted truth.

## Development

```bash
pip install -e ".[semantic,test]"
pytest -q
python scripts/measure_acceptance.py
```

Dense-index construction dependencies are intentionally optional. Build and
benchmark commands live under `wiki-forest-index`. Checkout builders often keep
FAISS/PyTorch in `.index-venv`; the warm worker prefers that interpreter when
present. Ordinary lexical traversal does not require FAISS or PyTorch.

## Provenance and license

Forest.Wikipedia.EN code is MIT licensed. The populated release contains independently
licensed Wikipedia, Kiwix/openZIM, and model material. See
[THIRD_PARTY_NOTICES.md](https://github.com/schmerbert/Forest.Wikipedia.EN/blob/main/THIRD_PARTY_NOTICES.md)
before redistributing a bundle.

Forest.Wikipedia.EN is independent and is not endorsed by Wikimedia Foundation, Kiwix,
openZIM, BAAI, or Hugging Face.

## Status

`v0.1.1` polishes the hangar over the same populated Hugging Face artifact
digests as `v0.1.0` (catalog identity string updated in the bundle manifest).
Locators, core CLI commands, and machine-readable search fields remain stable
within the `v0.1` line. Additive surfaces: `map`, `arm`, `disarm`,
`hydrated_now`, `worker`, and Acceptance timings.
