Metadata-Version: 2.4
Name: cocoindex-wiki
Version: 0.0.2
Summary: Incrementally convert documents to knowledge organized in wiki
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: cocoindex,incremental-indexing,knowledge-base,llm,markdown,wiki
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Documentation
Classifier: Topic :: Software Development :: Documentation
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.11
Requires-Dist: cocoindex>=1.0.0a46
Requires-Dist: faiss-cpu>=1.7.0
Requires-Dist: instructor>=1.0.0
Requires-Dist: litellm>=1.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: sentence-transformers>=2.0.0
Requires-Dist: tenacity>=8.0.0
Requires-Dist: tree-sitter-markdown>=0.5.1
Requires-Dist: tree-sitter>=0.25.2
Requires-Dist: typer>=0.9.0
Description-Content-Type: text/markdown

# cocoindex-wiki

`ccwiki` incrementally converts a collection of markdown documents into a structured, interconnected wiki using LLMs. It plans categories, extracts and deduplicates entities, produces curated knowledge entries, and keeps the wiki in sync with the source documents as they change — powered by [CocoIndex](https://github.com/cocoindex-io/cocoindex) for incremental processing and function memoization.

Use it from your coding agent via the included [Claude Code skill](#skill-recommended), or from the CLI directly.

Inspired by [Andrej Karpathy's note on wiki-style knowledge organization](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).

## What it does

Given a directory of markdown files, `ccwiki`:

1. **Plans categories** for the wiki based on the source documents.
2. **Extracts canonical entities** (people, systems, events, concepts, ...) from each document.
3. **Resolves duplicates** across documents (e.g., "Alice Chen" and "Alice" → one canonical entry) using embedding similarity plus LLM confirmation.
4. **Writes per-entity wiki entries** by combining knowledge from every source that mentioned the entity, with markdown cross-references and source footnotes.
5. **Keeps everything incremental** — edit a source doc and only affected entries get rebuilt.

The output is a set of markdown files organized by category, suitable for browsing in any markdown renderer (Obsidian, VS Code, GitHub, etc.).

## Get Started

### Install

Using [pipx](https://pipx.pypa.io/stable/installation/):

```bash
pipx install cocoindex-wiki          # first install
pipx upgrade cocoindex-wiki          # upgrade
```

Using [uv](https://docs.astral.sh/uv/getting-started/installation/):

```bash
uv tool install --upgrade cocoindex-wiki
```

Requires Python 3.11+. After installation, the `ccwiki` command is available globally.

### Configure the LLM

Set your primary model and the matching provider API key:

```bash
export CCWIKI_LLM_MODEL="anthropic/claude-haiku-4-5-20251001"
export ANTHROPIC_API_KEY="..."

# Optional: a lighter/cheaper model for entity extraction and resolution
export CCWIKI_LLM_MODEL_LITE="anthropic/claude-haiku-4-5-20251001"
```

`CCWIKI_LLM_MODEL` accepts any [LiteLLM](https://docs.litellm.ai/)-compatible model name (OpenAI, Anthropic, Google, and others). See `tests/e2e/run_test.sh` for working examples with Gemini, OpenAI, and Anthropic.

## Coding Agent Integration

### Skill (Recommended)

Install the `ccwiki` skill so your coding agent can set up and build your wiki interactively — it handles installation, category design, config writing, and indexing on its own:

```bash
npx skills add cocoindex-io/cocoindex-wiki
```

The skill teaches the agent to:
- Install `cocoindex-wiki` and verify `CCWIKI_LLM_MODEL` + the matching provider API key are set.
- Read your source docs first and propose category sets with explicit granularity rules.
- Write `@WIKI.md` and each `@WIKI_CATEGORY.md` to disk, then iterate with you on boundaries and descriptions.
- Run `ccwiki index` to build the wiki, show the result, and re-run incrementally as you edit sources or categories.

Just ask your agent something like *"help me set up a ccwiki for this folder of notes"* or *"add a new category for research papers"*, or type `/ccwiki` to invoke the skill directly. Works with [Claude Code](https://docs.anthropic.com/en/docs/claude-code) and other skill-compatible agents.

## Manual CLI Usage

You can also use the CLI directly — useful for scripted pipelines, CI jobs, or when you want full control without an agent in the loop.

### 1. Initialize a project

In the directory containing your markdown source files:

```bash
ccwiki init
```

This creates a `@WIKI.md` at the project root with default settings. Edit it to describe your project's purpose and adjust `include_patterns` / `exclude_patterns` / `output_dir` as needed.

### 2. Plan the wiki categories

```bash
ccwiki plan
```

This reads your source documents and asks the LLM to propose a category schema, then writes a `@WIKI_CATEGORY.md` in each category subdirectory. For finer control over categories, use the [skill](#skill-recommended) instead — it keeps a human in the loop during design.

### 3. Build the wiki

```bash
ccwiki index
```

This runs the full pipeline: entity extraction → resolution → knowledge extraction → combining → file output. The resulting wiki files appear under `wiki/` (or your configured `output_dir`).

On subsequent runs, only affected entries are rebuilt (thanks to CocoIndex memoization) — edits to one source doc typically touch only the entries that mentioned the changed entities.

### Verbose logging

Add `-v` to see entity extraction and deduplication logs:

```bash
ccwiki -v index
```

## Project layout

A typical ccwiki project:

```
my-project/
├── @WIKI.md                       # Project-level config
├── docs/                          # Source markdown files
│   ├── overview.md
│   ├── team.md
│   └── ...
└── wiki/                          # Generated wiki (the output)
    ├── People/
    │   ├── @WIKI_CATEGORY.md      # Category-level config
    │   ├── Alice Chen.md
    │   └── Bob Martinez.md
    ├── Products/
    │   ├── @WIKI_CATEGORY.md
    │   └── ...
    └── ...
```

`@WIKI.md` describes the project. Each `@WIKI_CATEGORY.md` defines what kinds of entries belong in that category and how to write them. See [skills/ccwiki/references/file_formats.md](skills/ccwiki/references/file_formats.md) for the full spec.

## How it works

`ccwiki` runs a multi-phase pipeline on [CocoIndex v1](https://github.com/cocoindex-io/cocoindex):

1. **Phase 1 — Entity extraction** (per source document, parallel). An LLM reads each raw doc and extracts canonical entities, classified by category. Entity names are sanitized for filesystem safety immediately after extraction.

2. **Phase 2.1 — Entity resolution** (per category, parallel). For each category, entities are embedded with `SentenceTransformer`, similar ones are found via FAISS, and an LLM confirms or rejects matches. A stability rule prefers entities that already have an existing `.md` file, so canonical choices stay consistent across runs.

3. **Phase 2.2 — Knowledge extraction and combining** (per canonical entity). For each entity, knowledge is extracted from each contributing source doc, then combined into a single entry via a stable K-ary combining tree. The tree structure is determined by stable file-path fingerprints, so when one source doc changes only `log(N)/log(K)` combining calls re-run. Final entries include markdown cross-links between related entities and numbered `[^src1]` footnotes pointing back to source documents.

All LLM calls are memoized by CocoIndex, so re-running `ccwiki index` after small edits is fast and cheap. Wiki files are declared as CocoIndex target states, so entities that are no longer canonical (merged or removed) have their files automatically deleted.

## Testing

End-to-end test scenarios live under [tests/e2e/](tests/e2e/):

- `tests/e2e/solar_system/` — small dataset (8 docs about the Solar System)
- `tests/e2e/helios_labs/` — larger dataset (24 docs about a fictional AI startup)
- `tests/e2e/arxiv_papers/` — 50 real arXiv papers

Run a scenario against one or all LLM providers:

```bash
source ~/.env.llm  # sets OPENAI_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY
bash tests/e2e/run_test.sh tests/e2e/solar_system              # all providers
bash tests/e2e/run_test.sh tests/e2e/solar_system anthropic    # single provider
```

Results are archived under `archived_results/` in each scenario directory.

## License

Apache 2.0 — see [LICENSE](LICENSE).
