Metadata-Version: 2.4
Name: AI-Knowledge-Engineer
Version: 0.2.0
Summary: Local-first, multimodal knowledge ingestion engine that transforms heterogeneous data into structured, evidence-backed knowledge and portable second-brain vaults.
Author-email: Saoud Haidar <13765149+saoudhaidar@users.noreply.github.com>
License: MIT License
        
        Copyright (c) 2026 Saoud Haidar
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
Project-URL: Documentation, https://github.com/saoudhaidar/knowledge-engineer
Project-URL: Source, https://github.com/saoudhaidar/knowledge-engineer
Project-URL: Issues, https://github.com/saoudhaidar/knowledge-engineer/issues
Keywords: knowledge-management,ingestion,extraction,provenance,local-first,evidence
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3,>=2.9
Requires-Dist: click>=8.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: jsonschema>=4.23
Requires-Dist: markdown>=3.7
Requires-Dist: httpx>=0.27
Requires-Dist: pypdf>=5.0
Requires-Dist: python-docx>=1.1
Requires-Dist: python-pptx>=1.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: pillow>=10.0
Requires-Dist: mutagen>=1.47
Requires-Dist: fastapi>=0.115
Requires-Dist: uvicorn>=0.32
Requires-Dist: python-multipart>=0.0.12
Requires-Dist: rapidfuzz>=3.14
Requires-Dist: networkx>=3.0
Requires-Dist: python-louvain>=0.16
Provides-Extra: dev
Requires-Dist: pytest>=8.3; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: mypy>=1.13; extra == "dev"
Requires-Dist: bandit>=1.8; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Requires-Dist: types-openpyxl; extra == "dev"
Provides-Extra: providers-cloud
Requires-Dist: openai>=1.40; extra == "providers-cloud"
Requires-Dist: anthropic>=0.34; extra == "providers-cloud"
Provides-Extra: providers-graph
Requires-Dist: neo4j<6,>=5.20; extra == "providers-graph"
Dynamic: license-file

# knowledge-engineer

Local-first, multimodal knowledge ingestion and extraction framework.

The engine walks a directory of source files, extracts structured
artifacts (PDF, DOCX, PPTX, XLSX, CSV, JSON, HTML, Markdown, plain
text, images, audio, video), runs knowledge extraction with a local
LLM, deduplicates and detects contradictions, and writes the result as
a Markdown vault (Obsidian-compatible) plus a JSON dump.

Domain lives in profiles — the same engine powers a personal archive, a
software repository, or a research corpus. v0.1 ships `generic`,
`personal`, `software`, and `research` profiles.

## What it is not

- It is not an Obsidian plugin. It exports an Obsidian-compatible vault,
  but the engine itself is a standalone CLI + REST API.
- It is not a personal-life tool by default. Personal archival is one of
  four profiles; the engine is domain-independent.
- It is not a cloud service. v0.1 runs entirely on local files. Cloud
  providers are documented as a future extension point.
- It is not magic. It cannot extract knowledge from files it cannot
  read; OCR requires a real OCR provider; vision captioning requires a
  real vision provider. Providers that are not configured fail with
  `ProviderNotConfiguredError` — they never silently fabricate output.

## Installation

Requires Python 3.11+.

```bash
python -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
```

Optional OCR / vision / transcription dependencies are NOT bundled:

```bash
# OCR
brew install tesseract        # macOS
pip install pytesseract

# Vision
pip install clip-onnx         # future provider; v0.1 ships null only

# Transcription
pip install faster-whisper
```

## Quick start

```bash
# 1. Initialize a project skeleton
knowledge-engineer init ./my-project

# 2. Scan a directory into the SQLite inventory
knowledge-engineer scan ./my-project/sample-data --db ./my-project/data/knowledge.db

# 3. Run the pipeline (discovery -> classify -> extract -> knowledge -> vault)
knowledge-engineer process \
    --db ./my-project/data/knowledge.db \
    --profile generic \
    --vault ./my-project/vault \
    --output-format markdown

# 4. Inspect
knowledge-engineer status --db ./my-project/data/knowledge.db
knowledge-engineer review  --db ./my-project/data/knowledge.db

# 5. Export
knowledge-engineer export \
    --db ./my-project/data/knowledge.db \
    --vault ./my-project/vault \
    --format obsidian   # or markdown | json
```

## What it does on a corpus

The pipeline runs these stages per source file:

1. **DISCOVER** — recursive walk, sha256 + MIME detection, symlink
   guards, archive-bomb limits.
2. **CLASSIFY** — MIME → PipelineKind (document / image / audio / video
   / structured / text / archive / skip).
3. **EXTRACT** — type-specific extractors produce a normalised Artifact
   (DocumentArtifact, ImageArtifact, etc.).
4. **NORMALIZE** — text encoding, length caps, language hints.
5. **ENTITY_EXTRACTION** — LLM extracts typed entities via
   temperature=0 structured output, schema-validated.
6. **CLAIM_EXTRACTION** — same, claims (subject, predicate, object,
   evidence).
7. **RELATIONSHIP_EXTRACTION** — typed relationships between entities.
8. **EVENT_EXTRACTION** — dated occurrences.
9. **VALIDATION** — claims without evidence are queued for review.
10. **PROVENANCE** — every record tracks the source + artifact + the
    extractor version that produced it.
11. **STORE** — SQLite persistence (WAL, foreign keys, busy timeout).
12. **EXPORT** — Markdown, Obsidian-compatible, or JSON.

Each stage records its outcome in `processing(source_id, stage)` so the
pipeline is resumable and auditable.

## Provider model

Each capability has an interface:

| Capability     | Interface                              | v0.1 default        | To enable                                  |
| -------------- | -------------------------------------- | ------------------- | ------------------------------------------ |
| LLM            | `providers.llm.LLMProvider`            | Ollama (real)       | `pip install ollama` + start daemon        |
| OCR            | `providers.ocr.OCRProvider`            | NullProvider        | `pip install pytesseract` + `brew install tesseract` |
| Vision         | `providers.vision.VisionProvider`      | NullProvider        | future: `clip-onnx`, `ollama-vision`       |
| Transcription  | `providers.transcription.Transcription`| NullProvider        | `pip install faster-whisper` + model       |
| Embeddings     | `providers.embeddings.Embeddings`      | NullProvider        | future: `sentence-transformers`            |

Null providers raise `ProviderNotConfiguredError` when asked to do work.
This is the framework's honesty contract.

## Documentation

- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)
- [docs/DATA_MODEL.md](docs/DATA_MODEL.md)
- [docs/CONFIGURATION.md](docs/CONFIGURATION.md)
- [docs/PROVIDERS.md](docs/PROVIDERS.md)
- [docs/EXTENDING.md](docs/EXTENDING.md)
- [docs/VAULT.md](docs/VAULT.md)
- [docs/SECURITY.md](docs/SECURITY.md)
- [docs/PRIVACY.md](docs/PRIVACY.md)
- [docs/TESTING.md](docs/TESTING.md)
- [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md)
- [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md)
- [docs/ROADMAP.md](docs/ROADMAP.md)

## License

MIT. See [LICENSE](LICENSE).
