Metadata-Version: 2.4
Name: athenaeum-kb
Version: 0.1.0
Summary: Tools for intelligent interaction with knowledge bases
License-Expression: MIT
Requires-Python: >=3.11
Requires-Dist: langchain-chroma>=0.2
Requires-Dist: langchain-core>=0.3
Requires-Dist: langchain-openai>=0.3
Requires-Dist: markitdown>=0.1
Requires-Dist: pydantic>=2.0
Requires-Dist: rank-bm25>=0.2.2
Provides-Extra: all-ocr
Requires-Dist: docling>=2.0; extra == 'all-ocr'
Requires-Dist: mistralai>=1.0; extra == 'all-ocr'
Requires-Dist: pillow>=10.0; extra == 'all-ocr'
Requires-Dist: torch>=2.0; extra == 'all-ocr'
Requires-Dist: transformers>=4.40; extra == 'all-ocr'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: docling
Requires-Dist: docling>=2.0; extra == 'docling'
Provides-Extra: lighton
Requires-Dist: pillow>=10.0; extra == 'lighton'
Requires-Dist: torch>=2.0; extra == 'lighton'
Requires-Dist: transformers>=4.40; extra == 'lighton'
Provides-Extra: mistral
Requires-Dist: mistralai>=1.0; extra == 'mistral'
Description-Content-Type: text/markdown

# Athenaeum 

## Project Scope and Goals

The goal of this project is to build a Python library that equips AI agents with a robust set of tools for intelligent interaction with knowledge bases. The library focuses on document ingestion, semantic search, and structured access to content, making it suitable for agent-based systems, RAG pipelines, and automation workflows.

Once the module is fully tested and validated, the intended outcome is to package and publish it as a reusable Python library on PyPI.

## Tools

### `load_doc`
Load a document into the knowledge base, automatically extracting content, metadata, and embeddings.

```python
load_doc(path: str) -> str
```

**Parameters:**
- `path`: Path to the document file

**Supported formats:** PDF, PPTX, DOCX, XLSX, JSON, CSV, TXT, MD, HTML, XML, RTF, EPUB

**Returns**: A document identifier (doc_id) that can be used for subsequent operations.

### `list_docs`
List all documents currently stored in the knowledge base.

```python
list_docs() -> list[DocSearchHit]
```

**Returns:** A list of documents, including metadata (id, name, format, etc.) and, when available, a table of contents.

### `search_docs`

Search across all documents in the knowledge base.

```python
search_docs(
    query: str,
    top_k: int = 10,
    scope: Literal["names", "contents"] = "contents",
    strategy: Literal["hybrid", "bm25", "vector"] = "hybrid"
) -> list[DocSearchHit]
```

**Parameters:**
- `query`: Search query text
- `top_k`: Maximum number of results (default: 10)
- `scope`: Where to search
  - `"contents"`: Search within document contents (default)
  - `"names"`: Search only document names
- `strategy`: Search strategy (only applies when scope is "contents")
  - `"hybrid"`: Combines vector and BM25 search (default)
  - `"bm25"`: Keyword-based search only
  - `"vector"`: Semantic similarity search only

**Returns**: A ranked list of documents matching the query.

### `search_doc_contents`

Search within a specific document.

```python
search_doc_contents(
    doc_id: str,
    query: str,
    top_k: int = 5,
    strategy: Literal["hybrid", "bm25", "vector"] = "hybrid"
) -> list[ContentSearchHit]
```

**Parameters:**
- `doc_id`: Document identifier
- `query`: Search query text
- `top_k`: Maximum number of results (default: 5)
- `strategy`: Search strategy
  - `"hybrid"`: Combines vector and BM25 search (default)
  - `"bm25"`: Keyword-based search only
  - `"vector"`: Semantic similarity search only

**Returns**: A list of matching content fragments with relevance scores.

### `read_doc`

Read a specific range of lines from a document.

```python
read_doc(
    doc_id: str,
    start_line: int = 1,
    end_line: int = 100
) -> Excerpt
```

**Parameters:**
- `doc_id`: Document identifier
- `start_line`: Starting line number (1-indexed, default: 1)
- `end_line`: Ending line number (1-indexed, inclusive, default: 100)

**Returns**: A document excerpt containing the requested lines.

## Search Strategies

- **Hybrid Search** (Default): Combines vector similarity search with BM25 keyword search using Reciprocal Rank Fusion (RRF).
- **Vector Search**: Uses embedding models for semantic similarity search.
- **BM25 Search**: Traditional keyword-based search using the BM25 algorithm.

## Document Ingestion Workflow

The ingestion pipeline is triggered by the load_doc(path) function and follows these steps:
1. Validation
    - Verify that the file exists.
    - Confirm that the file format is supported.
2. Content Extraction
    - Run an OCR or parsing pipeline to convert the raw file into Markdown.
    - If the document contains images:
        - Replace them with placeholders in the Markdown.
        - Store image references for later retrieval.
3. Pre-processing
    - Generate document metadata (e.g., id, name, format).
    - Build a table of contents (TOC) from Markdown headings when possible.
4. Indexing
    - Generate vector embeddings using the configured embedding model.
    - Store embeddings in the vector database for semantic retrieval.

## Data Models
A preliminary set of domain models is defined in `models.py`. These classes are exploratory and serve as a conceptual starting point for the project.

They are not considered final and may be refactored, renamed, or removed as the overall architecture of Athenaeum evolves and solidifies.


