Metadata-Version: 2.4
Name: embedding-ingestion
Version: 1.0.0
Author-email: jalal <jalalkhaldi3@gmail.com>
Requires-Python: <3.13,>=3.11
Requires-Dist: minio<8.0.0,>=7.2.19
Requires-Dist: polars<2.0.0,>=1.36.1
Requires-Dist: pydantic-settings<3.0.0,>=2.12.0
Requires-Dist: pydantic<3.0.0,>=2.12.4
Requires-Dist: qdrant-client<2.0.0,>=1.16.1
Requires-Dist: retrievalbase<2.0.0,>=1.0.0
Description-Content-Type: text/markdown

# embedding-ingestion

`embedding-ingestion` is a packaged ingestion pipeline that loads a text dataset from MinIO, generates embeddings through an OpenAI-compatible embedding endpoint, and stores vectors in Qdrant.

The project is intentionally config-driven. The runner, loader, dataset, embedder, and processor classes are all resolved from `module_path` values, which makes the package reusable across datasets and model backends without changing application code.

## What it does

Current built-in pipeline:

1. Load a `TextDataset` subclass from MinIO.
2. Convert dataset rows into LangChain `Document` objects.
3. Generate embeddings asynchronously.
4. Recreate a Qdrant collection.
5. Upsert document vectors and metadata into Qdrant.
6. Verify that points were written successfully.

Core components:

- `MinioLoader` in [`src/embedding_ingestion/loaders.py`](src/embedding_ingestion/loaders.py)
- `QdrantStore` in [`src/embedding_ingestion/store.py`](src/embedding_ingestion/store.py)
- `MinioVLLMQdrantRunner` in [`src/embedding_ingestion/runners.py`](src/embedding_ingestion/runners.py)
- CLI entrypoint in [`src/embedding_ingestion/main.py`](src/embedding_ingestion/main.py)

## Architecture

The package defines three extension points:

- `DocumentLoader`: responsible for fetching and optionally filtering documents.
- `Store`: responsible for embedding, persistence, and post-write verification.
- `Runner`: orchestrates the end-to-end ingestion flow.

At runtime, the CLI resolves the runner from the YAML config file at `/config/config.yaml`, then instantiates nested loader/store settings through Pydantic models.

## Requirements

- Python `>=3.11,<3.13`
- A reachable MinIO-compatible object store
- A reachable Qdrant instance
- An OpenAI-compatible embeddings endpoint
- A dataset class that subclasses `retrievalbase.dataset.TextDataset`

## Installation

### Local

Production dependencies:

```bash
make install
```

Development environment:

```bash
make dev-install
```

If you prefer using `uv` directly:

```bash
uv sync --group dev --all-extras
```

## Configuration

The application expects a YAML config at `/config/config.yaml` by default.

Example:

```yaml
module_path: embedding_ingestion.runners.MinioVLLMQdrantRunner

loader:
  module_path: embedding_ingestion.loaders.MinioLoader
  endpoint: minio:9000
  access_key: ${MINIO_ACCESS_KEY}
  secret_key: ${MINIO_SECRET_KEY}
  bucket: datasets
  dataset_module_path: your_project.datasets.MyTextDataset
  dataset_minio_path: corpora/my-dataset.parquet

store:
  module_path: embedding_ingestion.store.QdrantStore
  url: http://qdrant:6333
  collection_name: my_embeddings
  distance: cosine
  embedder:
    module_path: retrievalbase.evaluation.openai_compatible.OpenAICompatibleEmbedder
    model_name: text-embedding-3-large
    base_url: http://vllm:8000/v1
  processor:
    module_path: retrievalbase.evaluation.nomic.NomicProcessor
```

## Running

### CLI

After the config file is mounted or created at `/config/config.yaml`:

```bash
embedding-ingestion
```

Equivalent:

```bash
python -m embedding_ingestion.main
```

### Docker

Build:

```bash
docker build -t embedding-ingestion .
```

Run:

```bash
docker run --rm \
  -v /absolute/path/to/config:/config \
  embedding-ingestion
```

The image entrypoint runs:

```bash
python -m embedding_ingestion.main
```

## Best practices

### Treat ingestion as destructive by default

`QdrantStore.create()` deletes and recreates the target collection before writing. Use a dedicated collection per run or environment, and do not point this job at a production collection unless full replacement is intended.

### Use stable dataset classes

`dataset_module_path` must point to a concrete `TextDataset` subclass with a working `from_minio(...)` implementation. Keep that class in a versioned package so ingest behavior remains reproducible.

### Validate non-app dependencies before running

Before executing the pipeline, confirm:

- the MinIO bucket and object key exist
- the Qdrant URL is reachable
- the embedding endpoint is healthy
- the selected embedding model returns the expected vector dimension

### Watch for deduplication behavior

The built-in loader groups rows by `page_content` and keeps the first metadata entry. If duplicate text with different metadata matters in your use case, change that behavior before relying on the default loader.

### Make runs idempotent where possible

Point IDs are generated deterministically from `page_content` and `metadata`. That is good for stable re-ingestion semantics, but only if your upstream dataset normalization is also stable.

## Extending the package

Add a custom implementation when you need a different source or vector store:

- subclass `DocumentLoader` for a new ingestion source
- subclass `Store` for a new destination
- subclass `Runner` if orchestration needs to change

Then point the YAML `module_path` values at your custom classes.


## Repository layout

```text
src/embedding_ingestion/
  __init__.py      # abstract loader/store/runner contracts
  loaders.py       # MinIO dataset loader
  store.py         # Qdrant embedding store
  runners.py       # concrete pipeline runner
  settings.py      # config models
  utils.py         # runner loading and deterministic document IDs
  main.py          # CLI entrypoint
```
