Metadata-Version: 2.4
Name: wiki-forge
Version: 1.0.0
Summary: LLM Wiki — Auto-generate knowledge base from code & docs. CGC code intelligence, Q&A chat, knowledge graph, drift detection.
Project-URL: Homepage, https://github.com/tinhthanh/llm-wiki
Project-URL: Documentation, https://github.com/tinhthanh/llm-wiki/blob/main/docs/USER_GUIDE.md
Project-URL: Issues, https://github.com/tinhthanh/llm-wiki/issues
Author-email: FINOS Team <dev@finos.asia>
License: MIT
Keywords: code-to-doc,knowledge-base,knowledge-graph,llm,mcp,rag,wiki
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Documentation
Classifier: Topic :: Software Development :: Documentation
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-frontmatter>=1.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: rank-bm25>=0.2.2
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.9
Provides-Extra: all
Requires-Dist: codegraphcontext-rust>=0.5; extra == 'all'
Requires-Dist: fastapi>=0.100; extra == 'all'
Requires-Dist: kuzu>=0.4; extra == 'all'
Requires-Dist: markitdown[docx,pptx,xlsx]>=0.1; extra == 'all'
Requires-Dist: mcp>=1.0; extra == 'all'
Requires-Dist: networkx>=3.2; extra == 'all'
Requires-Dist: pdf-oxide>=0.3; extra == 'all'
Requires-Dist: pdfplumber>=0.10; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Requires-Dist: python-multipart>=0.0.6; extra == 'all'
Requires-Dist: uvicorn>=0.24; extra == 'all'
Provides-Extra: cgc
Requires-Dist: codegraphcontext-rust>=0.5; extra == 'cgc'
Requires-Dist: kuzu>=0.4; extra == 'cgc'
Provides-Extra: converters
Requires-Dist: markitdown[docx,pptx,xlsx]>=0.1; extra == 'converters'
Requires-Dist: pdf-oxide>=0.3; extra == 'converters'
Requires-Dist: pdfplumber>=0.10; extra == 'converters'
Requires-Dist: pyarrow>=14.0; extra == 'converters'
Provides-Extra: dev
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.12; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: graph
Requires-Dist: networkx>=3.2; extra == 'graph'
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == 'mcp'
Provides-Extra: server
Requires-Dist: fastapi>=0.100; extra == 'server'
Requires-Dist: python-multipart>=0.0.6; extra == 'server'
Requires-Dist: uvicorn>=0.24; extra == 'server'
Description-Content-Type: text/markdown

# llm-wiki

Karpathy-style LLM Wiki compiler as a proper Python library.

Ingests raw markdown sources into a structured interlinked wiki with:
- **Sources / Entities / Concepts** hierarchy
- **4-tier fuzzy dedup** for concept pages
- **Async Gemini client** with retry + rate-limit handling
- **Robust JSON parser** for LLM output (3-tier recovery)
- **Post-processing** fix_links for broken wikilinks

## Install

```bash
pip install -e ".[dev]"
```

## Quick start

```bash
export GEMINI_API_KEY=your-key

# Init vault
mkdir my-wiki && cd my-wiki
mkdir raw wiki
echo "# SkyJoy is a loyalty program" > raw/intro.md

# Ingest
llm-wiki ingest raw/intro.md --vault .

# Query
llm-wiki query "What is SkyJoy?" --vault .

# Fix broken wikilinks
llm-wiki fix-links --vault . --apply
```

## Code-to-Doc (ingest-code)

Generate wiki documentation directly from source code:

```bash
# Basic: Python-only call graph
llm-wiki ingest-code ./my-project --vault .

# With CGC: 19-language call graph + operational params extraction
pip install codegraphcontext kuzu
llm-wiki ingest-code ./my-project --vault . --cgc

# Incremental update (only changed modules)
llm-wiki ingest-code ./my-project --vault . --cgc --update

# Serve wiki with Q&A chat
llm-wiki serve --vault . --port 5757
```

### CGC augmentation (`--cgc`)

When enabled, [CodeGraphContext](https://github.com/Shashankss1205/CodeGraphContext) (MIT license) provides:

| Feature | Without `--cgc` | With `--cgc` |
|---------|-----------------|--------------|
| Call graph | Python-only (~200 edges) | **19 languages, 1400+ edges** |
| Source budget | 8,000 chars/module | **16,000 chars/module** |
| Operational params | Not extracted | **Cron schedules, timeouts, constants, env vars** |
| Class hierarchy | Not available | **Parents, children, methods** |

CGC auto-indexes on first run (~2 min for 300 files). Index is cached in `.cgc_index/`.

## Programmatic API

```python
import asyncio
from pathlib import Path

from llm_wiki import GeminiClient, Settings, Vault, ingest_source


async def main():
    settings = Settings(vault_root=Path("./my-wiki"))
    vault = Vault(settings.vault_root)
    llm = GeminiClient(api_key=settings.gemini_api_key)
    try:
        result = await ingest_source(
            source=vault.raw / "intro.md",
            vault=vault,
            llm=llm,
            settings=settings,
        )
        print(f"Ingested: {result.title}")
        print(f"Tokens: {result.input_tokens} in / {result.output_tokens} out")
    finally:
        await llm.close()

asyncio.run(main())
```

## Testing

```bash
make install        # pip install -e ".[dev]"
make test           # unit tests (fast, no network)
make test-integration   # real LLM calls (requires GEMINI_API_KEY)
make lint           # ruff + mypy
```

## Architecture

```
src/llm_wiki/
├── config.py           Settings via pydantic-settings
├── models.py           Pydantic models (IngestResult, Concept, Entity)
├── exceptions.py       Custom exceptions
├── dedup.py            4-tier fuzzy matching
├── json_parser.py      3-tier robust JSON parser
├── fix_links.py        Post-process broken wikilinks
├── llm/
│   ├── base.py         LLMProvider ABC
│   └── gemini.py       Async Gemini client
├── vault/
│   ├── paths.py        Vault path resolution
│   ├── frontmatter.py  YAML frontmatter parsing
│   └── wikilinks.py    Wikilink extraction
├── pipeline/
│   ├── ingest.py       ingest_source() — 1-shot ingest
│   ├── query.py        query_wiki() — Q&A against wiki
│   └── lint.py         lint_vault() — structural + semantic
└── cli.py              Typer CLI
```

## License

MIT
